I have created a CF script that creates an EC2 instance that contains a web service. It also creates an SNS Topic and a Subscription that uses this web service as it's http endpoint.
The script successfully creates the stack; the Topic and the Subscription exist. However, the Subscription remains in the PendingConfirmation state.
What must I do to get my script to confirm this Subscription upon creation?
I had a similar issue and my problem ended up being a misconfigured CloudFomation template. An AWS::SQS::QueuePolicy is required to give your SNS topic permission to send messages to the queue.
"SQSQueuePolicy": {
"Properties": {
"PolicyDocument": {
"Id": "usecase1",
"Statement": [
{
"Action": "SQS:SendMessage",
"Condition": {
"ArnEquals": {
"aws:SourceArn": {
"Ref": "SnsTopic"
}
}
},
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Resource": {
"Fn::GetAtt": [
"SqsQueue",
"Arn"
]
},
"Sid": "1"
}
],
"Version": "2012-10-17"
},
"Queues": [
{
"Ref": "SqsQueue"
}
]
},
"Type": "AWS::SQS::QueuePolicy"
}
You need to Subscribe to endpoint for this to work.
Read the value for SubscribeURL and visit that URL. To confirm the subscription and start receiving notifications at the endpoint, you must visit the SubscribeURLURL (for example, by sending an HTTP GET request to the URL)
When you visit the URL, you will get back a response that looks like the following XML document. The document returns the subscription ARN for the endpoint within the ConfirmSubscriptionResult element.
<ConfirmSubscriptionResponse xmlns="http://sns.amazonaws.com/doc/2010-03-31/">
<ConfirmSubscriptionResult>
<SubscriptionArn>arn:aws:sns:us-west-2:123456789012:MyTopic:2bcfbf39-05c3-41de-beaa-fcfcc21c8f55</SubscriptionArn>
</ConfirmSubscriptionResult>
<ResponseMetadata>
<RequestId>075ecce8-8dac-11e1-bf80-f781d96e9307</RequestId>
</ResponseMetadata>
</ConfirmSubscriptionResponse>
As an alternative to visiting the SubscribeURL, you can confirm the subscription using the ConfirmSubscription action with the Token set to its corresponding value in the SubscriptionConfirmation message. If you want to allow only the topic owner and subscription owner to be able to unsubscribe the endpoint, you call the ConfirmSubscription action with an AWS signature.
You can Refer to this AWS Documentation
Hope this Helps!
Related
Steps to reproduce:
Create a new DynamoDB table Events with partition key id
Create a new item in Events with an id of 123456. All other fields are blank.
Create an API-Gateway. Create path of events and a resource of remove-event under it.
Create a method of POST. AWS Service, DynamoDB, , and an action of DeleteItem. Add an IAM role that has access to the Events table too.
Open postman. Fill out the authorization and add the following parameters:
POST https://<DB GOES HERE>.execute-api.us-east-2.amazonaws.com/prod/events/remove-event
In the body, put (This is following the provided sample request here):
{
"TableName": "Events",
"Key": {
"id": {
"S": "123456"
}
},
"ReturnValues": "NONE"
}
Finally, hit send.
{
"message": "Internal server error"
}
I have no idea what is causing this. Create, Update, and Read work with issue. It is ONLY Delete that fails.
EDIT: In case anyone is curious about my policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "execute-api:Invoke",
"Resource": "arn:aws:execute-api:*:<sensitive infomation>:*/*/*/*"
}
]
}
Removing this changes the 500 response to a 403, so this is not a policy issue.
I have a CloudFormation template that includes an API Gateway configured using an OpenAPI 3.0 body. I would like to make the OpenAPI specification available to the users of my API. Ideally within a nice GUI but this answer suggests that's not possible. I do not want to set up a developer portal.
My CFT contains a DocumentationVersion element that creates documentation for the API.
According to this and this I should be able to download my documentation from a URL like
https://apigateway.[my_aws_region].amazonaws.com/restapis/[my_api_id]/stages/[my_api_stage]/exports/oas30
Indeed when I go to this URL I get something like
{"logref":"56f5173b-a329-11e9-a8d5-e97c525eb634","message":"Missing Authentication Token"}
Which suggests that this will work with the correct token.
This page shows that you can control access to API documentation with a policy. (Although oddly it says that the account_id to use is the one of the users you want to grant access to - is that correct?)
So I tried adding the following resource to my CFT:
"ApiDocumentationAccessPolicy": {
"Type": "AWS::IAM::ManagedPolicy",
"Properties": {
"Description": "Read access to API documentation restricted by IP",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"apigateway:GET"
],
"Resource": {
"Fn::Join": [
"",
[
"arn:aws:apigateway::",
{ "Ref": "AWS::AccountId" },
":/restapis/",
"*/documentation/*"
]
]
},
"Condition" : {
"IpAddress": {
"aws:SourceIp": ["xxx.xxx.xxx.xxx" ]
}
}
}
]
}
}
},
However, I still get "Missing Authentication Token". I am confident that the "condition" part is correct as it is used elsewhere in CFT.
Is what I want to do possible, and if so where am I going wrong?
EDIT Adding "Principal": "*" to the policy statement above and also moving it directly to the APIGateway's policy don't appear to help either.
I need to programmatically disable a lambda's SNS trigger, however, I seem to be unable to do so. I want this to show "Disabled" in the AWS Lambda console for the function:
Here's the code I've tried:
function updateEndpoints(endpoints, enable) {
const promises = [];
endpoints.forEach((endpoint) => {
console.log(`${enable ? 'Enabling' : 'Disabling'} Endpoint: ${endpoint}`);
promises.push(
SNS.setEndpointAttributes({
EndpointArn: endpoint,
Attributes: {
Enabled: enable ? 'True' : 'False',
},
}).promise()
.catch((e) => {
console.error(`Error ${enable ? 'Enabling' : 'Disabling'} Endpoint: ${endpoint}`);
console.error(e);
}));
});
return Promise.all(promises);
}
The endpoint ARN is passed in correctly with a string like (with correct values in place of the <> below):
-
arn:aws:lambda:<region>:<accountId>:function:<functionName>
-
This produces an error from AWS for each endpoint I try to enable or disable:
-
InvalidParameter: Invalid parameter: EndpointArn Reason: Vendor lambda is not of SNS
-
Is it not possible to disable the trigger/endpoint for a lambda via SNS? How would one go about doing this? I would prefer not to have to unsubscribe/subscribe as this would take the subscription objects out of CloudFormation's scope (correct?). I looked at updateEventSourceMappings, however, per the documentation, that only works with DynamoDB streams, Kinesis Streams, and SQS -- not SNS.
I found the (100%) correct way to do this. While the answer from #John Rotenstein could be used, it's not quite right, but should still work.
I found when you click the toggle, the lambda's policy is actually updated:
Enabled:
{
"Version": "2012-10-17",
"Id": "default",
"Statement": [
{
"Sid": "my-lambda-1552674933742",
"Effect": "Allow",
"Principal": {
"Service": "sns.amazonaws.com"
},
"Action": "lambda:InvokeFunction",
"Resource": "arn:aws:lambda:us-west-2:1234567890:function:my-lambda",
"Condition": {
"ArnLike": {
"AWS:SourceArn": "arn:aws:sns:us-west-2:1234567890:my-lambda"
}
}
}
]
}
Disabled:
{
"Version": "2012-10-17",
"Id": "default",
"Statement": [
{
"Sid": "my-lambda-1552674933742",
"Effect": "Allow",
"Principal": {
"Service": "sns.amazonaws.com"
},
"Action": "lambda:DisableInvokeFunction",
"Resource": "arn:aws:lambda:us-west-2:1234567890:function:my-lambda",
"Condition": {
"ArnLike": {
"AWS:SourceArn": "arn:aws:sns:us-west-2:1234567890:my-lambda"
}
}
}
]
}
Notice Action is lambda:InvokeFunction vs. lambda:DisableInvokeFunction.
My process to do this is as follows:
- Lambda.listFunctions
- for each function, Lambda.removePermission
- for each function, Lambda.addPermission
Notes:
the Lambda api has a default safety throttle of 100 concurrent executions per account per region.
You can only update resource-based policies for Lambda resources within the scope of the AddPermission and AddLayerVersionPermission API actions. You can't author policies for your Lambda resources in JSON, or use conditions that don't map to parameters for those actions. See docs here
Also, you can use Lambda.getPolicy to see the policy of the lambda to ensure it is updated.
It appears that there is no capability to "disable" a Lambda subscription to an SNS topic.
I base my reasoning on the follow steps I took:
Created an AWS Lambda function
Created an Amazon SNS topic
Subscribed the Lambda function to the SNS topic (done via the SNS console)
Confirmed in the Lambda console that the function subscription to SNS is "enabled"
Ran aws sns list-subscriptions-by-topic --topic-arn arn:aws:sns:ap-southeast-2:123456789012:my-topic
Saw that the Lambda function was subscribed
The response was:
{
"Subscriptions": [
{
"SubscriptionArn": "arn:aws:sns:ap-southeast-2:123456789012:stack:...",
"Owner": "123456789012",
"Protocol": "lambda",
"Endpoint": "arn:aws:lambda:ap-southeast-2:743112987576:function:my-function",
"TopicArn": "arn:aws:sns:ap-southeast-2:123456789012:stack"
}
]
}
I then disabled the trigger in the Lambda console and saved the Lambda function. When I re-ran the above command, the results were empty:
{
"Subscriptions": []
}
When I enabled it again, the subscription returned.
So, my assumption is that, since the "disable/enable" button actually adds and removes a subscription, there does not appear to be any capability to 'disable' a subscription.
I am trying to build a Kibana dashboard fed with twitter data collected via AWS Kinesis firehose where data passes into an S3 bucket which triggers a Lambda function which passes the data to AWS Elastic Search and then to Kibana. I am following this blog https://aws.amazon.com/blogs/big-data/building-a-near-real-time-discovery-platform-with-aws/
The data is loading into the S3 bucket correctly but it never arrives in Kibana, I believe this is because the Lambda function is not being triggered by events in S3 as I would have hoped (there are no invocations or logs). I think this is because I have not set permissions properly. The Lambda function can be invoked manually by the test event.
On the Lambda function page I chose an existing role which I called lambda_s3_exec_role which has the AWSLambdaExecute policy attached to it but I feel I'm missing something else more specific to S3. I have been unable to follow this line in the blog in the create lambda function section because I do not recognise those options:
"10. Choose lambda_s3_exec_role (if this value does not exist, choose Create new role S3 execution role)."
Can anyone help me create the appropriate role/policy for the Lambda function, or spot what the problem may be?
From view permissions on the Lambda function I currently have:
FUNCTION POLICY
{
"Version": "2012-10-17",
"Id": "default",
"Statement": [
{
"Sid": "****",
"Effect": "Allow",
"Principal": {
"Service": "s3.amazonaws.com"
},
"Action": "lambda:InvokeFunction",
"Resource": "****",
"Condition": {
"ArnLike": {
"AWS:SourceArn": "arn:aws:s3:::****"
}
}
}
]
}
EXECUTION ROLE
{
"roleName": "lambda_s3_exec_role",
"policies": [
{
"document": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:*"
],
"Resource": "arn:aws:logs:*:*:*"
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Resource": "arn:aws:s3:::*"
}
]
},
"name": "AWSLambdaExecute",
"id": "****",
"type": "managed",
"arn": "arn:aws:iam::aws:policy/AWSLambdaExecute"
}
]
}
The permissions you have listed look OK so I am going to try provide some steps that might help find the issue as it is difficult to understand specifically where your issue might be.
Does the execution role have the trust relationship with a trusted entity of lambda.amazonaws.com
Does your event prefix match the prefix in firehose. In the tutorial they are both twitter/raw-data/. If firehose is writing to a path that isn't the event prefix then the event won't be invoked.
Does the lambda trigger any errors when you manually invoke it
Does the lambda write to the logs when you manually invoke it
Test the lambda using dummy data (example data below)
CLI
aws lambda invoke \
--invocation-type RequestResponse \
--function-name helloworld \
--region region \
--log-type Tail \
--payload file://dummy_event.json \
--profile adminuser \
outputfile.txt
Example data
source
dummy_event.json
{
"Records":[
{
"eventVersion":"2.0",
"eventSource":"aws:s3",
"awsRegion":"us-west-2",
"eventTime":"1970-01-01T00:00:00.000Z",
"eventName":"ObjectCreated:Put",
"userIdentity":{
"principalId":"AIDAJDPLRKLG7UEXAMPLE"
},
"requestParameters":{
"sourceIPAddress":"127.0.0.1"
},
"responseElements":{
"x-amz-request-id":"C3D13FE58DE4C810",
"x-amz-id-2":"FMyUVURIY8/IgAtTv8xRjskZQpcIZ9KG4V5Wp6S7S/JRWeUWerMUE5JgHvANOjpD"
},
"s3":{
"s3SchemaVersion":"1.0",
"configurationId":"testConfigRule",
"bucket":{
"name":"sourcebucket",
"ownerIdentity":{
"principalId":"A3NL1KOZZKExample"
},
"arn":"arn:aws:s3:::sourcebucket"
},
"object":{
"key":"HappyFace.jpg",
"size":1024,
"eTag":"d41d8cd98f00b204e9800998ecf8427e",
"versionId":"096fKKXTRTtl3on89fVO.nfljtsv6qko"
}
}
}
]
}
Struggled with this for a long time and eventually realized that your rule that triggers the lambda cannot have exactly the same name as the lambda itself or it won't work.
I am trying to prototype a distributed application using SNS and SQS.I have this topic:
arn:aws:sns:us-east-1:574008783416:us-east-1-live-auction
and this queue:
arn:aws:sqs:us-east-1:574008783416:queue4
I created the queue using the JS Scratchpad. I added the subscription using the Console. I AddPermission to the queue using the scratchpad. The queue policy is now:
{
"Version":"2008-10-17",
"Id":"arn:aws:sqs:us-east-1:574008783416:queue4/SQSDefaultPolicy",
"Statement":[
{
"Sid":"RootPerms",
"Effect":"Allow",
"Principal":{
"AWS":"574008783416"
},
"Action":"SQS:*",
"Resource":"arn:aws:sqs:us-east-1:574008783416:queue4"
}
]
}
I have an email subscription on the same topic and the emails arrive fine but the messages never arrive on the queue. I've tried SendMessage directly to the queue - rather than via SNS - using Scratchpad and it works fine. Any ideas why it won't send to the queue?
This was posted a while back on the AWS forums: https://forums.aws.amazon.com/thread.jspa?messageID=202798
Then I gave the SNS topic the permission to send messages to the SQS queue. The trick here is to allow all principals. SNS doesn't send from your account ID -- it has its own account ID that it sends from.
Adding to Skyler's answer, if like me you cringe at the idea of allowing any principal (Principal: '*'), you can restrict the principal to SNS:
Principal:
Service: sns.amazonaws.com
Although this behavior is undocumented, it works.
Most of the answers (beside #spg answer) propose usage of principal: * - this is very dangerous practice and it will expose your SQS to whole world.
From AWS docs
For resource-based policies, such as Amazon S3 bucket policies, a wildcard (*) in the principal element specifies all users or public access.
We strongly recommend that you do not use a wildcard in the Principal element in a role's trust policy unless you otherwise restrict access through a Condition element in the policy. Otherwise, any IAM user in any account in your partition can access the role.
Therefore it is strongly not recommended to use this principal.
Instead you need to specify sns service as your principal:
"Principal": {
"Service": "sns.amazonaws.com"
},
Example policy:
{
"Version": "2012-10-17",
"Id": "Policy1596186813341",
"Statement": [
{
"Sid": "Stmt1596186812579",
"Effect": "Allow",
"Principal": {
"Service": "sns.amazonaws.com"
},
"Action": [
"sqs:SendMessage",
"sqs:SendMessageBatch"
],
"Resource": "Your-SQS-Arn"
}
]
}
With this policy sns will be able to send messages to your SQSs.
There are more permissions for SQS but from what I see SendMessage and SendMessageBatch should be enough for SNS->SQS subscribtion.
Here's a full CloudFormation example of Skyler's answer
{
"Resources": {
"MyTopic": {
"Type": "AWS::SNS::Topic"
},
"MyQueue": {
"Type": "AWS::SQS::Queue"
},
"Subscription": {
"Type" : "AWS::SNS::Subscription",
"Properties" : {
"Protocol" : "sqs",
"TopicArn" : {"Ref": "MyTopic"},
"Endpoint": {"Fn::GetAtt": ["MyQueue", "Arn"]}
}
},
"QueuePolicy": {
"Type": "AWS::SQS::QueuePolicy",
"Properties": {
"Queues": [
{"Ref": "MyQueue"}
],
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Sid": "allow-sns-messages",
"Effect": "Allow",
"Principal": {"Service": "sns.amazonaws.com"},
"Action": "sqs:SendMessage",
"Resource": {"Fn::GetAtt": ["MyQueue", "Arn"]},
"Condition": {
"ArnEquals": {
"aws:SourceArn": {"Ref": "MyTopic"}
}
}
}
]
}
}
}
}
}
Amazon has more options in their Sending Amazon SNS Messages to Amazon SQS Queues document.
I just experienced this and took me a while to figure out why:
If I create a SQS subscription from the SNS console, it does not add necessary permissions to the SQS access policy.
If I create the subscription in the SQS console to the same SNS, it does.
Old question but using an AWS SDK version > 1.10
Check out the docs SQS-SNS sendMessage Permission
private static void updateQueuePolicy(AmazonSQS sqs, String queueURL, String topicARN) {
Map<String, String> attributes = new HashMap<String, String>(1);
Action actions = new Action() {
#Override
public String getActionName() {
return "sqs:SendMessage"; // Action name
}
};
Statement mainQueueStatements = new Statement(Statement.Effect.Allow)
.withActions(actions)
.withPrincipals(new Principal("Service", "sns.amazonaws.com"))
.withConditions(
new Condition()
.withType("ArnEquals")
.withConditionKey("aws:SourceArn")
.withValues(topicARN)
);
final Policy mainQueuePolicy = new Policy()
.withId("MainQueuePolicy")
.withStatements(mainQueueStatements);
attributes.put("Policy", mainQueuePolicy.toJson());
updateQueueAttributes(sqs, queueURL, attributes);
}
Which outputs a policy similar to
{
Policy={
"Version":"2012-10-17",
"Id":"MainQueuePolicy",
"Statement":
[
{
"Sid":"1",
"Effect":"Allow",
"Principal": {
"Service": "sns.amazonaws.com"
},
"Action":["sqs:SendMessage"],
"Condition":
{"ArnEquals":
{"aws:SourceArn":["arn:aws:sns:us-east-1:3232:testSubscription"]}
}
}
]
}
}
Like the other answers mentioned, you must opt in and grant permission to this SNS topic to publish to your SQS queue.
If you use terraform, you can use the sqs_queue_policy resource.
Here is an example:
resource "aws_sqs_queue_policy" "your_queue_policy" {
queue_url = "${aws_sqs_queue.your_queue.id}"
policy = <<POLICY
{
"Version": "2012-10-17",
"Id": "sqspolicy",
"Statement": [
{
"Sid": "First",
"Effect": "Allow",
"Principal": "*",
"Action": "sqs:SendMessage",
"Resource": "${aws_sqs_queue.your_queue.arn}",
"Condition": {
"ArnEquals": {
"aws:SourceArn": "${aws_sns_topic.your_topic.arn}"
}
}
}
]
}
POLICY
}
If you have enabled encryption on your queue, can also be a reason for SNS not being able to put message on subscriber queue. You need to give access to SNS to that KMS key.
This article explains how to solve this problem:
only you need subscribe the queue to the topic from the queue console.
step one: select the queue
step two: queue Actions
step three: Subscribe Queue to SNS Topic
step choose: the topic
end.
It's by AWS::SQS::QueuePolicy.
You need define a this kind of policy to allow specific SNS perform actions to specific SQS.
QueuePolicy:
Type: AWS::SQS::QueuePolicy
Properties:
Queues:
- Ref: MyQueue1
- Fn::Sub: arn:aws:sqs:us-east-1:${AWS::AccountId}:my-queue-*
PolicyDocument:
Version: '2012-10-17'
Statement:
- Sid: allow-sns-messages
Effect: Allow
Principal:
Service: sns.amazonaws.com
Action: sqs:SendMessage
Resource:
- Fn::Sub: arn:aws:sqs:us-east-1:${AWS::AccountId}:my-queue-*
Condition:
ArnEquals:
aws:SourceArn:
- Fn::Sub: arn:aws:sns:us-east-1:${AWS::AccountId}:source-sns-*
With Lambda this kind of policies dont apply. Please if anyone know why please share it.