Depending on which page I create a subscription for a SNS topic to a SQS queue (both belonging to the same account) from within the AWS console, I notice a difference in how SQS policy for the queue is updated.
When I initially created the subscription from the SNS topic page in
the console, the queue's access policy did not get updated.
When I created the policy from SQS queue's page, I see the queue policy is
updated automatically allowing "SQS:SendMessage" from the SNS topic
ARN.
Is the difference in behavior between these two use cases intentional? Is there a reason behind why creating the subscription from SNS topic page does not update the permission automatically?
I don't think there is any specially reason for that. Its just inconsistencies in how AWS Console works. In fact, there is plenty of such inconsistencies in AWS Console.
The most basic example of that is when you delete a resource. Some resources will just delete without asking anything, others will force you to write "delete", others "delete me", or "permanently delete", or write "resource-name" or confirm something else.
Related
We're working on creating a CloudFormation stack which will be published to clients to install an "on-prem" agent of our product on a client's AWS environment.
We want to initiate the connection from our end and add some configurations based on the outputs of the CloudFormation.
Is there a way to automatically transmit the output of CloudFormation back to the publisher of the template? I've been looking for a solution online but with no avail. Not sure this is possible - but I'm hoping someone here has maybe done something similar.
Couldn't find any solution. From what I'm seeing the "only" way to do so is to ask the client to manually send back the output, but I'm glad to be proven wrong :)
So I happen to be working on something similar, and this is what I've found that works. There are caveats which I will enumerate.
TLDR;
To have a CloudFormation stack "phone home" you can specify a SNS topic that the stack will post changes to when it is created, updates, and/or deleted. The CloudFormation docs (see 'Notification options') and CLI reference (see '--notification-arns') mention this. You can then have a Lambda respond to the SNS messages to enable your desired behavior. Unfortunately you cannot specify this topic in the stack template and can only provide is when making the call to create the stack. This can be worked around by using a nested CloudFormation stack.
My (current) approach:
Use a SNS topic that has a resource policy which allows the installing user's AWS account to publish messages to your topic. AWS documents how to do this, but be warned that using a AWS Account Principal in the SNS topic policy (which so far is the only thing that has worked for me) allows anything in the customer account to publish to the SNS topic. This security issue is one part I have yet to work around and ideally a combination of a Service Principal and restrictive IAM policy conditions could be used instead.
Create a template that you provide to your user that acts as the top-level stack, and contains a nested stack which you will use to create resources in your user's AWS account. This also gives you an opportunity to do things like use a mapping that specifies per-region SNS topics for your CloudFormation notifications.
When specifying your nested stack in your top-level stack template you use the NotificationARNs property to provide the SNS topic ARN so that the nested stack will report back to you.
The Lambda (or other mechanism) that subscribes to the SNS topic will need to parse and filter the messages that CloudFormation posts to react appropriately. When needing to acquire the ARN of something like an IAM role in the user's AWS account my recommendation would be to give the role a static name such that you can know the role's ARN by using the user's AWS account ID and the role name. Ideally the role will have permission to retrieve information about the CloudFormation stack so that you can get ARNs and other data about any other resources your stack created.
We have a Snowflake pipe on a S3 integration which is setup to auto-ingest using a SNS topic dedicated to object creation events on the S3 bucket, following Snowflake documentation
While loading data by refreshing the pipe works perfectly, it seems that SNS subscription triggered by creation of the pipe setting aws_sns_topic to the topic ARN and auto_ingest = true does not work since after creation of the pipe, we cannot see the SNS->SQS subscription.
We have used this pattern many times and all the existing ones work perfectly except this one. Also, I should mention that the infrastructure in AWS is created with Terraform. Finally, the pipe is created (or replaced) after any changes to the SNS topic. I am happy to post my IAM policy on the role assigned to SFK for a second look but I have made sure it does have sns:Subscribe on the SNS topic.
Note: We have tried direct direct S3 event notification bucket to SQS subscription as proposed by Snowflake docs and that did not help either.
AWS has redesigned the SQS console experience few days or a week before. I am not able to find the Permissions tab in the new version. Could anyone please share the screenshots how to navigate?
To access the Access Policy:
Click on the queue
Go to the Access policy tab
Click Edit
You will need to edit the policy in JSON. I think there was previously a simpler way to add permissions, but that method simply modified the policy. Now you need to edit the policy manually.
You can use the AWS Policy Generator to help construct a policy.
Looks like when first creating the queue in the new console, it has a wizard that lets you add other AWS accounts, IAM roles/users - but after the initial queue creation, if you go to edit the queue, that wizard isn't there.
I'm working in an environment where anyone with the necessary access is allowed to create an S3 bucket; however, it's getting to a point where we have a lot of buckets and it is hard to keep track of who created the bucket. I know it is possible to tag the buckets with the owner name, but I am looking for a more automated solution.
Is it possible to invoke a lambda function every time a bucket is created? Or is it possible to track bucket creation with cloudtrail where system administrators would get an sns notification when an s3 bucket is created?
I know it is possible to configure s3 event notification inside a bucket to trigger lambda functions/cloudwatch metrics, but I need a trigger for the entire s3 application.
Cloudtrail tracks all API Actions occurring within an account. What you want to do is create a cloudwatch event rule that triggers off the CreateBucket action then have it invoke Lambda or trigger a sns notification.
See: Creating a CloudWatch Events Rule That Triggers on an AWS API Call Using AWS CloudTrail
You can use EventBridge to get at these events, via CloudTrail. The example on here is of a CreateBucket request.
I am trying to figure out a way to trigger a Lambda on the creation or update or a Role in AWS.
The use case is that when a Role is created, we need to update our Identity Server with the new or changed Role.
I'm looking at cloud trail, and having mixed results. I could schedule a lambda to run, but I'd prefer to make it more real time.
Any ideas?
Sounds like going CloudTrail's way is exactly what AWS suggested.
What issue you got into?
AWS CloudTrail saves logs to an S3 bucket (object-created event).
Amazon S3 detects the object-created event.
Amazon S3 publishes the s3:ObjectCreated:* event to AWS Lambda by
invoking the Lambda function, as specified in the bucket notification
configuration. Because the Lambda function's access permissions policy
includes permissions for Amazon S3 to invoke the function, Amazon S3
can invoke the function.
AWS Lambda executes the Lambda function by assuming the execution role
that you specified at the time you created the Lambda function.
The Lambda function reads the Amazon S3 event it receives as a
parameter, determines where the CloudTrail object is, reads the
CloudTrail object, and then it processes the log records in the
CloudTrail object.
You can Ensure a log metric filter and alarm exist for IAM policy changes and similar.
Alarm can put message to SNS for example.
Lambda can be triggered by that SNS message.
Real-time monitoring of API calls can be achieved by directing CloudTrail Logs to CloudWatch Logs and establishing corresponding metric filters and alarms. It is recommended that a metric filter and alarm be established changes made to Identity and Access Management (IAM) policies.