I'm trying to set up an existing SQS Queue as a subscriber to an SNS topic. In the AWS console in the permissions tab, I can set the policy document to
{
"Version": "2012-10-17",
"Id": "arn:aws:sqs:us-east-1:7670234568007:stdsourcequeue/SQSDefaultPolicy",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "sqs:SendMessage",
"Resource": "arn:aws:sqs:us-east-1:7670234568007:stdsourcequeue",
"Condition": {
"ArnEquals": {
"aws:SourceArn": "arn:aws:sns:us-east-1:7670234568007:new_posts"
}
}
}
]
}
How can I do this using the aws-cli
This is an practical example of how to do it using set-queue-attributes:
cat >/tmp/sqs_polcy << EOL
{
"Policy": "{\"Version\":\"2012-10-17\",\"Id\":\"arn:aws:sqs:us-east-1:7670234568007:stdsourcequeue\/SQSDefaultPolicy\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":\"*\",\"Action\":\"sqs:SendMessage\",\"Resource\":\"arn:aws:sqs:us-east-1:7670234568007:stdsourcequeue\",\"Condition\":{\"ArnEquals\":{\"aws:SourceArn\":\"arn:aws:sns:us-east-1:7670234568007:new_posts\"}}}]}"
}
EOL
aws sqs set-queue-attributes \
--queue-url https://<your-queue-url> \
--attributes file:///tmp/sqs_polcy
Above I create /tmp/sqs_polcy file with the policy, which is required for the set-queue-attributes command.
Your policy must be stringified from json, before you can use in the CLI command.
These are generally managed via the CLI command of add-permission for SQS.
However as you're using your own custom policy the AWS documentation states the following
AddPermission generates a policy for you. You can use SetQueueAttributes to upload your policy.
This would be accessible via the set-queue-attributes function.
Your policy will need to be be converted into a JSON file, against the key value of Policy.
As a word of caution by doing this it will replace the policy attached to your SQS queue, so make sure to validate it before hand.
Related
I'm getting this error message when trying to see the log file in AWS CloudWatch for my AWS Lambda function.
An error occurred while describing log streams.
The specified log group does not exist.
Log group does not exist
The specific log group: /aws/lambda/xxxxx does not exist in this account or region.
By the way, I'm using the Singapore region.
Make sure that your Lambda function's execution role has sufficient permissions to write logs to CloudWatch, and that the log group resource in the IAM policy includes your function's name.
In the IAM console, review and edit the IAM policy for the execution role to make sure that:
The write actions CreateLogGroup and CreateLogStream are allowed. You should attach these policies in the IAM roles of the Lambda function
Note: If you don't need custom permissions for your function, you can attach the managed policy AWSLambdaBasicExecutionRole, which allows Lambda to write logs to CloudWatch.
The AWS Region specified in the Amazon Resource Name (ARN) is the
same as your Lambda function's Region.
The log-group resource includes your Lambda function name. For
example, if your function is named myLambdaFunction, the log-group is
/aws/lambda/myLambdaFunction.
Here is an example of the permissions in the JSON format
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "logs:CreateLogGroup",
"Resource": "arn:aws:logs:region:accountId:*"
},
{
"Effect": "Allow",
"Action": [
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": [
" arn:aws:logs:region:accountId:log-group:/aws/lambda/functionName:*"
]
}
]
}
In an attempt to further tighten the security of our solution we are now looking at the used SNS topics and SQS queues. All our components live in the same AWS account.
For starters we want to restrict the access to the SQS queues based on IP. So only requests coming from our NAT Gateway IP will be allowed. We don't allow anonymous access to our SQS queues.
But there seems no way to achieve this as the creator of the SQS queues - the AWS account id - has access per default. So you can't create an effective permission for another user in the same AWS account id. As this newly created user, user2, will fall under the same AWS account id, with the same set of permissions.
Am I correct in my understanding that all users in the same AWS account id have access per default to all created SQS queues as long as their IAM policy permits it? And is my assumption right that the same behavior goes for the SNS topics?
Below is the policy I would like the implement. Beside this policy I have no other policies active for this SQS q. But it is not honoring the source IP condition. I still can connect from everywhere when I use a correct AWS access key/secret combination. Only when I set the AWS principal to * - everyone - the policy seems effective.
{
"Version": "2012-10-17",
"Id": "arn:aws:sqs:eu-west-1:4564645646464564:madcowtestqueue/SQSDefaultPolicy",
"Statement": [
{
"Sid": "Sid1589365989662",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::4564645646464564:user/user2"
},
"Action": [
"SQS:DeleteMessage",
"SQS:SendMessage",
"SQS:ReceiveMessage"
],
"Resource": "arn:aws:sqs:eu-west-1:143631359317:madcowtestqueue",
"Condition": {
"IpAddress": {
"aws:SourceIp": "1.1.1.1"
}
}
}
]
}
Reference:
Using identity-based policies with Amazon SQS - Amazon Simple Queue Service
Using identity-based policies with Amazon SNS - Amazon Simple Notification Service
Amazon SQS
Amazon SQS has the ability to define Amazon SQS policies. These policies can be used in addition to IAM policies to grant access to a queue.
For example, a policy can be added that permits anonymous access to a queue, which is useful for external applications to send messages to the queue.
Interestingly, these policies can also be used to control access to the queue by IP address.
To test this, I did the following:
Created an Amazon SQS queue
Used an Amazon EC2 instance to send a message to the queue -- Successful
Added the following policy to the SQS queue:
{
"Version": "2012-10-17",
"Id": "Queue1_Policy_UUID",
"Statement": [
{
"Sid": "Queue1_AnonymousAccess_AllActions_IPLimit_Deny",
"Effect": "Deny",
"Principal": "*",
"Action": "SQS:SendMessage",
"Resource": "arn:aws:sqs:ap-southeast-2:xxx:queue",
"Condition": {
"NotIpAddress": {
"aws:SourceIp": "54.1.2.3/32"
}
}
}
]
}
The IP address is that of my Amazon EC2 instance.
I then tried send a message to the queue again from the EC2 instance -- Successful
I then ran the identical command from my own computer -- Not successful
Therefore, it would appear that the SQS policy can override the permissions granted via IAM.
(Be careful... I added a policy that Denied sqs:* on the queue, and I wasn't able to edit the policy or delete the queue! I had to use the root account to delete it.)
Amazon SNS
I managed to achieve the same result with Amazon SNS using this access policy:
{
"Version": "2008-10-17",
"Id": "__default_policy_ID",
"Statement": [
{
"Sid": "__default_statement_ID",
"Effect": "Deny",
"Principal": {
"AWS": "*"
},
"Action": "SNS:Publish",
"Resource": "arn:aws:sns:ap-southeast-2:xxx:topic",
"Condition": {
"NotIpAddress": {
"aws:SourceIp": "54.1.2.3/32"
}
}
}
]
}
I am struggling with what I would think would be a simple task.
I want to configure my SQS queue to allow S3 buckets in my account to send messages, but disallow outsiders to send messages. (Outsiders means any principal that is not a member of my AWS account)
The only SQS permission configuration that I can get to work is Effect=Allow, Principals=*, Actions=SQS:SendMessage, Conditions=None
Any other permission causes me to see this error when I create the [S3 event -> SQS]: Unable to validate the following destination configurations. Permissions on the destination queue do not allow S3 to publish notifications from this bucket.
Principals=* concerns me. From the documentation I can find, this means that the SQS queue is accessible from anyone in the world. Is this true? This is obviously very bad.
How can I allow my S3 buckets to SendMessage to my SQS queue, and not anonymous users to push messages?
It is acceptable to me to allow any resource in my AWS account to SendMessage to SQS. I just need to block access to anonymous AWS users. This is a very basic requirement and I'm very surprised that I can't find a simple way to do this.
You can find the secure configuration right in the document
"Condition": {
"ArnLike": { "aws:SourceArn": "arn:aws:s3:*:*:bucket-name" }
}
Note that for both the Amazon SNS and Amazon SQS IAM policies, you can specify the StringLike condition in the policy, instead of the ArnLike condition.
"Condition": {
"StringLike": { "aws:SourceArn": "arn:aws:s3:*:*:bucket-name" }
}
Full example pulled from the doc
{
"Sid": "example-statement-ID",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": [
"SQS:SendMessage"
],
"Resource": "SQS-ARN",
"Condition": {
"ArnLike": { "aws:SourceArn": "arn:aws:s3:*:*:bucket-name" }
}
}
I’ve tested a variation of wide policy access , and got to the same point – the log groups is created, but the log stream isn’t.
Followed https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/amazon-mq-configuring-cloudwatch-logs.html and the expected result is getting those messages in CloudWatch, but nothing's coming in.
The goal is to have audit and general MQ logs in CloudWatch.
Has anyone managed to stream MQ logs in CloudWatch? How could I further debug this?
I managed to create the Amazon MQ Broker with logging enabled, and publishing log messaged to Cloudwatch using terraform's provider 1.43.2 -- my project has a lock-down on an older tf provider version, so if you're using a newer one you should be fine
https://github.com/terraform-providers/terraform-provider-aws/blob/master/CHANGELOG.md#1430-november-07-2018
This was the policy that I didn't get right the first time, and needed for MQ to post to Cloudwatch:
data "aws_iam_policy_document" "mq-log-publishing-policy" {
statement {
actions = [
"logs:CreateLogStream",
"logs:PutLogEvents",
]
resources = ["arn:aws:logs:*:*:log-group:/aws/amazonmq/*"]
principals {
identifiers = ["mq.amazonaws.com"]
type = "Service"
}
}
}
resource "aws_cloudwatch_log_resource_policy" "mq-log-publishing-policy" {
policy_document = "${data.aws_iam_policy_document.mq-log-publishing-policy.json}"
policy_name = "mq-log-publishing-policy"
}
Make sure this policy has been correctly applied, otherwise nothing will come up on Cloudwatch. I did so using aws cli:
aws --profile my-testing-profile-name --region my-profile-region logs describe-resource-policies
and you should see the policy in the output.
Or if you're using aws cli you can try
aws --region [your-region] logs put-resource-policy --policy-name AmazonMQ-logs \
--policy-document '{
"Statement": [
{
"Action": [
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Effect": "Allow",
"Principal": {
"Service": "mq.amazonaws.com"
},
"Resource": "arn:aws:logs:*:*:log-group:/aws/amazonmq/*"
}
],
"Version": "2012-10-17"
}'
Install the AWS CLI agent for Windows and configure your credentials https://docs.aws.amazon.com/cli/latest/userguide/install-windows.html
Create a JSON file in "C:\Users\YOUR-USER\" containing your policy. For example: C:\Users\YOUR-USER\policy.json. You can simply copy this one here and paste into your .json file:
{"Version": "2012-10-17","Statement": [{"Effect": "Allow","Principal": {"Service": "mq.amazonaws.com"},"Action":["logs:CreateLogStream","logs:PutLogEvents"],"Resource" : "arn:aws:logs:*:*:log-group:/aws/amazonmq/*"}]}
Open your CMD and simply type:
aws --region eu-central-1 logs put-resource-policy --policy-name amazonmq_to_cloudwatch --policy-document file://policy.json
Well Done ! This will create a AWS RESOURCE POLICY, which sometimes is not possible to create in the IAM console.
On AWS we've implemented functionality that AWS lambda pushes message to AWS queue;
However during this implementation I had to manuall grant permissions to AWS lambda to add message to particular queue. And this apporach with manual clicks not so good for prod deployment.
Any suggestions how to automate process of adding permissions between AWS services (mainly lambda and SQS) and cretate "good" deployment package for prod env ?
Each Lambda function has an attached role, which you can specify permissions for in the IAM dashboard. If you give the Lambda functions' role the permission to push to an SQS queue, you're good to go. For example, attach this JSON as a custom role (see http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/SQSExamples.html):
{
"Version": "2012-10-17",
"Id": "Queue1_Policy_UUID",
"Statement":
{
"Sid":"Queue1_SendMessage",
"Effect": "Allow",
"Principal": {
"AWS": "111122223333"
},
"Action": "sqs:SendMessage",
"Resource": "arn:aws:sqs:us-east-1:444455556666:queue1"
}
}
You can use asterisks to give permission to multiple queues, like:
"Resource": "arn:aws:sqs:us-east-1:444455556666:production-*"
To give sendMessage permission to all queues that start with production-.