We have a Snowflake pipe on a S3 integration which is setup to auto-ingest using a SNS topic dedicated to object creation events on the S3 bucket, following Snowflake documentation
While loading data by refreshing the pipe works perfectly, it seems that SNS subscription triggered by creation of the pipe setting aws_sns_topic to the topic ARN and auto_ingest = true does not work since after creation of the pipe, we cannot see the SNS->SQS subscription.
We have used this pattern many times and all the existing ones work perfectly except this one. Also, I should mention that the infrastructure in AWS is created with Terraform. Finally, the pipe is created (or replaced) after any changes to the SNS topic. I am happy to post my IAM policy on the role assigned to SFK for a second look but I have made sure it does have sns:Subscribe on the SNS topic.
Note: We have tried direct direct S3 event notification bucket to SQS subscription as proposed by Snowflake docs and that did not help either.
Related
Depending on which page I create a subscription for a SNS topic to a SQS queue (both belonging to the same account) from within the AWS console, I notice a difference in how SQS policy for the queue is updated.
When I initially created the subscription from the SNS topic page in
the console, the queue's access policy did not get updated.
When I created the policy from SQS queue's page, I see the queue policy is
updated automatically allowing "SQS:SendMessage" from the SNS topic
ARN.
Is the difference in behavior between these two use cases intentional? Is there a reason behind why creating the subscription from SNS topic page does not update the permission automatically?
I don't think there is any specially reason for that. Its just inconsistencies in how AWS Console works. In fact, there is plenty of such inconsistencies in AWS Console.
The most basic example of that is when you delete a resource. Some resources will just delete without asking anything, others will force you to write "delete", others "delete me", or "permanently delete", or write "resource-name" or confirm something else.
I have a usecase where I want to put data into an S3 bucket, for it to read later, by another account. I only want the other account to be able to read the file in S3, and once they have read it, I will then delete the file myself.
I have been reading the S3 documentation, and cannot see they cover this usecase: of sending a notification when a file in an S3 bucket is read ?
Can anyone help, or suggest an alternative workflow ? I have been looking at AWS SNS and was wondering if that would be a better solution ?
You could use CloudTrail and CloudWatch Events to enable this workflow.
By default S3 API calls are not logged so you'd want to enable that following the instructions here.
Then enable a CloudWatch event rule for the Simple Storage Service where the "GetObject" operation occurs.
Have this event invoke a Lambda function that will remove the object.
More information available here.
how is the best solution to create a alert with athena data?
the loadbalance logs is saved in S3 and acessed with Athena.
I want alert me based on a athena query. I thinking about populate Cloudwatch metrics with athena result, and create a alert in cloudwatch. Can I create a metric direct with athena result?
I can resolve with lambda, but I'm search a better solution.
Your question is a little bit ambiguous.
Do you want to create an alert when Athena writes to an S3 bucket?
You can achieve this through SNS and the S3 bucket config. Check this tutorial:
Example Walkthrough 1: Configure a Bucket for Notifications (Message Destination: SNS Topic and SQS Queue)
Then you can launch a different AWS service (like a Lambda function) through SNS to react to this event.
Keep in mind that you will have to wait for Athena to finish the query as it will run asynchronously. Here an example in Java to achieve this:
Athena Query execution Handling
I want to tag AWS resources like dynamodb tables or EC2 objects right at the time of creation.
I will be using id or name fields of the objects to tag the resources.
Is there any 'post-create' trigger available?
--
The current problem is that even if I run my script to tag AWS resources, I can't run it immediately after every resource creation, I end up seeing a lot of billing untagged.
You can do this through AWS Service Catalog service. this has a capability of auto tagging of provisioned resources. AWS reference link. AutoTags are tags that identify the portfolio, product, and user that launched a product, and are automatically applied by AWS Service Catalog to provisioned resources
You can configure a lambda function to write a tag based on the CloudTrail event that is generated whenever a resource is created.
To get Lambda to run against a CloudTrail event you need to setup your CloudTrail to write events to a S3 bucket, then trigger the Lambda on the object creation event in the bucket.
The lambda uses the bucket key in its context object to read the event and determine if a tag needs to be applied.
Check the AWS Documentation for further detail about triggering Lambda from CloudTrail.
Also GorillaStack has published an example on Github of using lambda to auto-tag newly created resources. You could use this as a basis for your solution.
I'm trying to achieve the ask from the title. For example, my architecture design involves trigger a Lambda function whenever a new data land on the open data s3 bucket (say this one: https://registry.opendata.aws/sentinel-2/).
I read https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html and Amazon S3 triggering another a Lambda function in another account but I non of those really help me so far. Had anyone done similar task before? Thanks in advance!
You can configure Amazon S3 events to send a message to an Amazon SNS Topic OR have it trigger an AWS Lambda function.
If you wish to do any further logic (eg checking specific permissions), you would need to do that within the Lambda function.
See: Configuring Amazon S3 Event Notifications - Amazon Simple Storage Service