I am trying to refresh external table using SNS in snowflake.
I have followed this tutorial to refresh.
https://www.youtube.com/watch?v=PCNa3d6rMO0
it is working as expected.but when I use same topic to trigger another table in another S3 bucket. Can't I use the same topic and create event notifications in Bucket2?
Here is my Access Policy :
{
"Version": "2008-10-17",
"Id": "__default_policy_ID",
"Statement": [
{
"Sid": "__default_statement_ID",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": [
"SNS:Publish",
"SNS:RemovePermission",
"SNS:SetTopicAttributes",
"SNS:DeleteTopic",
"SNS:ListSubscriptionsByTopic",
"SNS:GetTopicAttributes",
"SNS:AddPermission",
"SNS:Subscribe"
],
"Resource": "arn:aws:sns:us-west-1:58:snowflake-dev-SNS",
"Condition": {
"StringEquals": {
"AWS:SourceOwner": "55"
}
}
},
{
"Sid": "__console_pub_0",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "SNS:Publish",
"Resource": "arn:aws:sns:us-west-1:55:snowflake-dev-SNS"
},
{
"Sid": "1",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::2:user/b6m8-s-p2s9"
},
"Action": "sns:Subscribe",
"Resource": "arn:aws:sns:us-west-1:55:snowflake-dev-SNS"
}
]
}
Thanks,
Xi
I'm not a Snowflake person, but it appears that the linked demonstration is Configuring Amazon SNS to Automate Snowpipe Using SQS Notifications
.
According to that documentation, the following code is used to create the pipe:
create pipe snowpipe_db.public.mypipe
auto_ingest=true
aws_sns_topic='<sns_topic_arn>'
as
copy into snowpipe_db.public.mytable
from #snowpipe_db.public.mystage
file_format = (type = 'JSON');
The copy into snowpipe_db.public.mytable seems to be hard-coded for a destination table. It seems that each Snowpipe can only be used to load data into a single table.
Therefore, you would likely need to use a different Snowpipe, and therefore a different SNS Topic and SQS queue, if you wish to load data into a different table.
Related
I'm trying to get CloudWatch to send to my SNS topic. The topic is encrypted. However I get this error
Received error: "CloudWatch Alarms does not have authorization to access the SNS topic encryption key."
I have added KMS permissions to the access policy associated with the SNS topic but I get this error when I try to save the policy
Error code: InvalidParameter - Error message: An error occurred while setting the attribute access policy. Invalid parameter: Policy statement action out of service scope!
Is there something in the policy I am misisng?
{
"Version": "2012-10-17",
"Id": "SNS",
"Statement": [
{
"Sid": "SNSPolicy",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": [
"SNS:GetTopicAttributes",
"SNS:SetTopicAttributes",
"SNS:AddPermission",
"SNS:RemovePermission",
"SNS:DeleteTopic",
"SNS:Subscribe",
"SNS:ListSubscriptionsByTopic",
"SNS:Publish"
],
"Resource": "arn:aws:sns:*:my-account-name:my-topic-name"
},
{
"Sid": "CloudWatch",
"Effect": "Allow",
"Principal": {
"Service": "cloudwatch.amazonaws.com"
},
"Action": [
"kms:Decrypt",
"kms:GenerateDataKey*",
"kms:DescribeKey"
],
"Resource": [
"arn:aws:kms:*:my-account-name:key/*"
],
"Condition": {
"StringEquals": {
"kms:RequestAlias": "alias/my-alias-name"
}
}
}
]
}
Cloudwatch is not having necessary permission to access the KMS key which is being used by the SNS topic.
Below SNS topic access policy will give access to the Amazon cloudwatch events.
{
"Version": "2012-10-17",
"Id": "__default_policy_ID",
"Statement": [
{
"Sid": "__default_statement_ID",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": [
"SNS:GetTopicAttributes",
"SNS:SetTopicAttributes",
"SNS:AddPermission",
"SNS:RemovePermission",
"SNS:DeleteTopic",
"SNS:Subscribe",
"SNS:ListSubscriptionsByTopic",
"SNS:Publish"
],
"Resource": "arn:aws:sns:<region>:<account>:<snstopic>",
"Condition": {
"StringEquals": {
"AWS:SourceOwner": "<account>"
}
}
},
{
"Sid": "AWSEvents_test_Id10633056916634",
"Effect": "Allow",
"Principal": {
"Service": "events.amazonaws.com"
},
"Action": "sns:Publish",
"Resource": "arn:aws:sns:<region>:<account>:<snstopic>"
}
]
}
Below customer managed KMS key policy, it will give access to Amazon cloudwatch event source.
{
"Version": "2012-10-17",
"Id": "key-consolepolicy-3",
"Statement": [
{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<account>:<iamuser>",
"Service": "cloudwatch.amazonaws.com"
},
"Action": "kms:*",
"Resource": "*"
}
]
}
Several AWS services publish events to Amazon SNS topics. To know more, Kindly refer here
I have a AWS S3 bucket in account A, This bucket was created by AWS Control Tower.
And used for collecting logs from all other account in my org,
I was trying to understand the bucket policy which is something like this
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowSSLRequestsOnly",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::aws-controltower-logs-12345656-us-east-1",
"arn:aws:s3:::aws-controltower-logs-12345656-us-east-1/*"
],
"Condition": {
"Bool": {
"aws:SecureTransport": "false"
}
}
},
{
"Sid": "AWSBucketPermissionsCheck",
"Effect": "Allow",
"Principal": {
"Service": [
"config.amazonaws.com",
"cloudtrail.amazonaws.com"
]
},
"Action": "s3:GetBucketAcl",
"Resource": "arn:aws:s3:::aws-controltower-logs-12345656-us-east-1"
},
{
"Sid": "AWSConfigBucketExistenceCheck",
"Effect": "Allow",
"Principal": {
"Service": [
"config.amazonaws.com",
"cloudtrail.amazonaws.com"
]
},
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::aws-controltower-logs-12345656-us-east-1"
},
{
"Sid": "AWSBucketDelivery",
"Effect": "Allow",
"Principal": {
"Service": [
"config.amazonaws.com",
"cloudtrail.amazonaws.com"
]
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::aws-controltower-logs-12345656-us-east-1/o-1234/AWSLogs/*/*"
}
]
}
Now all other account in my org are able to dump there cloudtrail logs within this S3.
But i dont get one thing, i did not specified any particular or individual account number, but still other accounts are able to write content in this bucket,
Although i do see the principal which mentions relevant service name that can dump, but should,nt it only for this account itself ?
Let's analyze the rules one by one:
The first rule only says that no access without SSL is possible, it does nothing if SSL layer is present:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowSSLRequestsOnly",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::aws-controltower-logs-12345656-us-east-1",
"arn:aws:s3:::aws-controltower-logs-12345656-us-east-1/*"
],
"Condition": {
"Bool": {
"aws:SecureTransport": "false"
}
}
},
The next two actions allow only read:
{
"Sid": "AWSBucketPermissionsCheck",
"Effect": "Allow",
"Principal": {
"Service": [
"config.amazonaws.com",
"cloudtrail.amazonaws.com"
]
},
"Action": "s3:GetBucketAcl",
"Resource": "arn:aws:s3:::aws-controltower-logs-12345656-us-east-1"
},
{
"Sid": "AWSConfigBucketExistenceCheck",
"Effect": "Allow",
"Principal": {
"Service": [
"config.amazonaws.com",
"cloudtrail.amazonaws.com"
]
},
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::aws-controltower-logs-12345656-us-east-1"
},
So the only action which allows any writing is this one:
{
"Sid": "AWSBucketDelivery",
"Effect": "Allow",
"Principal": {
"Service": [
"config.amazonaws.com",
"cloudtrail.amazonaws.com"
]
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::aws-controltower-logs-12345656-us-east-1/o-1234/AWSLogs/*/*"
}
]
}
And it says the following: You can put object under /o-1234/AWSLogs as long as you are one of the following two AWS services: Config or Cloudtrail.
Clearly, if knowing the bucket name and the org ID allows me to persuade Config or Cloudtrail to use that bucket I cannot see anything what would stop me from doing that except from some internal protection inside those services.
Based on this document:
https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-set-bucket-policy-for-multiple-accounts.html
It seems that for allowing an account 111111111111 to write to that bucket you should use the following ARN pattern: "arn:aws:s3:::myBucketName/optionalLogFilePrefix/AWSLogs/111111111111/*",
So while the answer provided by #izayoi does not provide any explanation, it is still correct. Cloudtrail service should guarantee you that it will always use that account id in the log, so you can narrow down the access by listing all your account numbers. Of course, it must be updated with every each new account.
Conclusion: Yes, knowing the bucket name and your organization ID should allow every AWS account in the world to use your bucket for Cloudtrail logging with the current policy...interesting. I would probably go with listing your account numbers.
"Resource": "arn:aws:s3:::aws-controltower-logs-12345656-us-east-1/o-1234/AWSLogs/*/*"
The 1st "*" enables all account numbers.
I've been through the AWS SNS/SQS subscription instructions multiple times, and have gone through a few different blogs and StackOverflow posts trying various things. However, no matter how many times I try to publish a message to SNS and poll/receive it from SQS, it never pops out the other side. I'm hoping it's something super obvious that I'm missing, that someone with more experience/fresher eyes than me can see.
For some background, I created both SQS and SNS in the same account. I create the subscription last after I have set the access policies for both of them. And I make sure that when I do create the subscription that I get the little green checkmark that says the subscription has been confirmed. Nothing fancy, no FIFO queue, I've set the visibility timeout to 1 minute, and receive message wait time relatively low for the queue as well. I only have in the account one SQS and one SNS, so it's not like I can mess up subscribing or giving access to the wrong ones.
For my SQS access policy this is what it looks like:
{
"Version": "2012-10-17",
"Id": "allow_sns_access_policy",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "sns.amazonaws.com"
},
"Action": "sqs:SendMessage",
"Resource": "arn:aws:sqs:us-east-1:<my_unique_id>:<name_of_my_queue>",
"Condition": {
"ArnEquals": {
"aws:SourceArn": "arn:aws:sns:us-east-1:<my_unique_id>:<name_of_my_topic>"
}
}
}
]
}
My SNS access policy is as follows:
{
"Version": "2012-10-17",
"Id": "allow_sqs_access_policy",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "sqs.amazonaws.com"
},
"Action": "sns:Subscribe",
"Resource": "arn:aws:sqs:us-east-1:<my_unique_id>:<name_of_my_topic>",
"Condition": {
"ArnEquals": {
"aws:SourceArn": "arn:aws:sns:us-east-1:<my_unique_id>:<name_of_my_queue>"
}
}
}
]
}
I make sure to copy and paste arn resources to limit spelling mistakes. I've experimented with other things, but all of them also didn't work. This to me should work but doesn't.
I guess the policies can be finicky sometimes. I attempted to remove the policies, but that didn't work even though they're in the same region. So here is what I wound up setting for the two policies to get them to correctly communicate. Maybe someone has suggestions for how to make this better, but as of right now this is what works.
SNS policy:
{
"Version": "2008-10-17",
"Id": "__default_policy_ID",
"Statement": [
{
"Sid": "__default_statement_ID",
"Effect": "Allow",
"Principal": {
"Service": "sqs.amazonaws.com"
},
"Action": [
"SNS:GetTopicAttributes",
"SNS:SetTopicAttributes",
"SNS:AddPermission",
"SNS:RemovePermission",
"SNS:DeleteTopic",
"SNS:Subscribe",
"SNS:ListSubscriptionsByTopic",
"SNS:Publish",
"SNS:Receive"
],
"Resource": "arn:aws:sns:<region>:<topic_owner>:<topic>",
"Condition": {
"StringEquals": {
"AWS:SourceOwner": "<topic_owner>"
}
}
},
{
"Sid": "s3-publish",
"Effect": "Allow",
"Principal": "*",
"Action": "sns:Publish",
"Resource": "arn:aws:sns:<region>:<topic_owner>:<topic>",
"Condition": {
"StringEquals": {
"AWS:SourceAccount": "<topic_owner>"
},
"ArnLike": {
"AWS:SourceArn": "arn:aws:s3:*:*:*"
}
}
}
]
}
SQS Policy
{
"Version": "2008-10-17",
"Id": "__default_policy_ID",
"Statement": [
{
"Sid": "__owner_statement",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<topic_owner>:root"
},
"Action": "SQS:*",
"Resource": "arn:aws:sqs:<region>:<topic_owner>:<queue_name>"
},
{
"Sid": "topic-subscription-arn:aws:sns:<region>:<topic_owner>:<topic_name>",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "SQS:SendMessage",
"Resource": "arn:aws:sqs:<region>:<topic_owner>:<queue_name>",
"Condition": {
"ArnLike": {
"aws:SourceArn": "arn:aws:sns:<region>:<topic_owner>:<topic_name>"
}
}
}
]
}
I am trying to set up a notification for the code pipeline using its notification rule which supporters SNS.
As you can see in the picture the status is "unreachable"
If I look at the link here aws troubleshoot
I have followed all the step even the step of adding of codestar-notifications in Acces policy of SNS topic.
{
"Sid": "AWSCodeStarNotifications_publish",
"Effect": "Allow",
"Principal": {
"Service": "codestar-notifications.amazonaws.com"
},
"Action": "SNS:Publish",
"Resource": "arn:aws:codestar-notifications:us-east-1:272075499248:notificationrule/50d629524d433dceeafdb6c5fe136e404f29e9e5"
}
But still, the status remains the same also tried with manually starting the pipeline but still not working.
Am I missing something? could anyone help me out of this?
EDIT:
{
"Version": "2008-10-17",
"Id": "__default_policy_ID",
"Statement": [
{
"Sid": "__default_statement_ID",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": [
"SNS:GetTopicAttributes",
"SNS:SetTopicAttributes",
"SNS:AddPermission",
"SNS:RemovePermission",
"SNS:DeleteTopic",
"SNS:Subscribe",
"SNS:ListSubscriptionsByTopic",
"SNS:Publish",
"SNS:Receive"
],
"Resource": "arn:aws:sns:us-east-1:272075499248:develop",
"Condition": {
"StringEquals": {
"AWS:SourceOwner": "272075499248"
}
}
},
{
"Sid": "AWSCodeStarNotifications_publish",
"Effect": "Allow",
"Principal": {
"Service": "codestar-notifications.amazonaws.com"
},
"Action": "SNS:Publish",
"Resource": "arn:aws:sns:us-east-1:272075499248:develop"
}
]
}
The previous answers here were too confusing and some of them were incomplete. Here is the complete solution with the ins and outs.
1. Simple Notification Service Access Policy Confusion
Whenever you create an SNS topic by itself, the default access policy will look something like this:
{
"Version": "2008-10-17",
"Id": "__default_policy_ID",
"Statement": [
{
"Sid": "__default_statement_ID",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": [
"SNS:Publish",
"SNS:RemovePermission",
"SNS:SetTopicAttributes",
"SNS:DeleteTopic",
"SNS:ListSubscriptionsByTopic",
"SNS:GetTopicAttributes",
"SNS:AddPermission",
"SNS:Subscribe"
],
"Resource": "arn:aws:sns:us-east-2:123456789012:my-sns-topic",
"Condition": {
"StringEquals": {
"AWS:SourceOwner": "123456789012"
}
}
}
]
}
The above is wrong and will not let your CodePipeline access the SNS topic (make it reachable/"Active")! Change the Access policy for your SNS topic to the following instead:
{
"Version": "2008-10-17",
"Statement": [
{
"Sid": "CodeNotification_publish",
"Effect": "Allow",
"Principal": {
"Service": "codestar-notifications.amazonaws.com"
},
"Action": "SNS:Publish",
"Resource": "arn:aws:sns:us-east-2:123456789012:my-sns-topic"
}
]
}
NOTE 1: Change 123456789012 to your AWS account ID, and my-sns-topic to the name of your SNS topic.
NOTE 2: If your region is different than us-east-2, then change that too in the above snippet.
NOTE 3: Both the SNS topic and the CodePipeline Notification rule should be in the same region, otherwise this won't work.
2. Notification Rule and Notification Rule Target Issue
Whenever you create a Notification Rule and then a Notification Rule Target,
the only way possible for AWS to refresh the Notification target status is for you to delete the Notification rule target from CodePipeline -> Settings (on the left side bar) -> Notification rules -> Notification rule targets (this is extremely important!).
NOTE: Deleting the Notification rule target from the notification rule itself won't do anything; because of that, when you re-add it in that page, the Notification rule target will still be the old one and thus the Notification target status will remain "Unreachable".
If after everything it still says it's unreachable, repeat exactly steps #1 and #2 again, you may have missed something.
One way to solve this is to use the CodePipeline user interface to create the Topic. This will set all of the required permissions for you. When creating the Notification Rule, under "Targets", select "Create Target" and enter the name of the Topic you wish to create. The topic will be created with permissions already set. You will just need to subscribe to the topic to receive the notifications.
The JSON file is correct, but you should delete and re-create the target rule
It could be that your pipeline’s IAM execution role doesn’t have the required permissions to publish messages to the topic. Make sure your pipeline can publish messages in both the IAM role and the SNS policy and give it another go. A telltale sign of this is the CodePipeline notification console showing “Unreachable” next to the SNS topic.
The SNS access policy will look like the following:
{
"Version": "2008-10-17",
"Id": "__default_policy_ID",
"Statement": [
{
"Sid": "StatusNotificationsPolicy",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789123:root",
"Service": "codestar-notifications.amazonaws.com"
},
"Action": "sns:Publish",
"Resource": "arn:aws:sns:ap-southeast-2:123456789123:gimme-alerts"
},
{
"Sid": "__default_statement_ID",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": [
"SNS:GetTopicAttributes",
"SNS:SetTopicAttributes",
"SNS:AddPermission",
"SNS:RemovePermission",
"SNS:DeleteTopic",
"SNS:Subscribe",
"SNS:ListSubscriptionsByTopic",
"SNS:Publish",
"SNS:Receive"
],
"Resource": "arn:aws:sns:ap-southeast-2:123456789123:gimme-alerts",
"Condition": {
"StringEquals": {
"AWS:SourceOwner": "123456789123"
}
}
}
]
}
https://www.stephengream.com/codepipeline-notifications
The following did work for me.
I followed the suggestion by Phil Gilligan in the other answer. It automatically created the access policy in sns topic when its created from CodeCommit itself.
Change the account id and repo name according to your own case.
There is no other rule just this one rule. It seems like the rules are evaluated and one rule overridden the other. I think if one rule is more restrictive it takes precedence over the other.
{
"Version": "2008-10-17",
"Statement": [
{
"Sid": "CodeNotification_publish",
"Effect": "Allow",
"Principal": {
"Service": "codestar-notifications.amazonaws.com"
},
"Action": "SNS:Publish",
"Resource": "arn:aws:sns:us-east-1:ACCOUNT_ID:REPO_NAME"
}
]
}
Answers about Access Policy are right. But the change is not applied immediately. Very annoying.
Just create new topic from CodePipeline Settings. The access policy will be auto-adjusted.
Is it possible to create an S3 bucket policy that allows read and write access from aws machine learning ? I tried below bucket policy, but not work.
bucket policy:
{
"Version": "2008-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "machinelearning.amazonaws.com"
},
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Resource": "arn:aws:s3:::cxra/*"
},
{
"Effect": "Allow",
"Principal": {
"Service": "machinelearning.amazonaws.com"
},
"Action": "s3:PutObjectAcl",
"Resource": "arn:aws:s3:::cxra/*",
"Condition": {
"StringEquals": {
"s3:x-amz-acl": "bucket-owner-full-control"
}
}
},
{
"Effect": "Allow",
"Principal": {
"Service": "machinelearning.amazonaws.com"
},
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::cxra"
},
{
"Effect": "Allow",
"Principal": {
"Service": "machinelearning.amazonaws.com"
},
"Action": [
"s3:GetBucketLocation"
],
"Resource": "arn:aws:s3:::cxra"
}
]
}
Error screen
Double check your policy against Granting Amazon ML Permissions to Output Predictions to Amazon S3 and Granting Amazon ML Permissions to Read Your Data from Amazon S3
As they are documented as two separate policies, I suggest you follow that, rather than joining the two policies in one single document. It will be easier to troubleshoot.