Aim : Use terraform to create an SNS topic in multiple regions.
Right now it lets me create a single SNS in the default region or one specific region i input.
Hurdles :
providers.tf doesnt support iterating the region via a list
sns topic creation in terraform doesnt take allow region arguments to be passed as well.
The only workaround i could do is do something found here -
Using Terraform to manage multiple AWS regions
But this involves a lot of code duplication and is not efficient or scalable.
Is there any efficient iterative approach to solve this problem ?
Related
If given the correct permissions, Lambda functions will automatically create CloudWatch log groups to hold log output from the Lambda. Same for LambdaEdge functions with the addition that the log groups are created in each region in which the LambdaEdge function has run and the name of the log group includes the name of the region. The problem is that the retention time is set to forever and there is no way to change that unless you wait for the log group to be created and then change the retention config after the fact.
To address this, I would like to create those log groups preemptively in Terraform. The problem is that the region would need to be set in the provider meta argument or passed in the providers argument to a module. I had originally thought that I could get the set of all AWS regions using the aws_regions data source and then dynamically create a provider for each region. However, there is currently no way to dynamically generate providers (see https://github.com/hashicorp/terraform/issues/24476).
Has anyone solved this or a similar problem in some other way? Yes, I could create a script using the AWS CLI to do this, but I'd really like to keep everything in Terraform. Using Terragrunt is also an option, but I wanted to see if there were any solutions using pure Terraform before I go that route.
I want to do the first approach listed here, where in order to trigger various lambda functions to run when a single S3 bucket is updated, I would put an SNS queue in between.
Currently I have each Lambda as a stack, and I would prefer to keep it that way, especially since I have separate pipeline stages I need to separate anyways. However, I want to be able to make them all share the same SNS queue. What would be the best way to do this?
From my thinking, the best way to approach would be to create an "sns queue stack" that creates the topic, then pass that topic into each lambda stack and subscribe the lambda functions that way, but I'm still unsure of the best way to deploy this sns queue stack.
This is most confusing to me in regards to using a deployment pipeline in the CDK. I have multiple pipeline stages, each with multiple deployment groups and multiple lambda stacks in said deployment groups. How should I add in this stack to ensure it is deployed properly?
One guess I have would be to add it into the very first stage in the very first deployment group before all other stacks and then it should work for every other stage, but I'm not sure if this is a way that would work.
We can use two approaches.
Multiple stacks in single CDK Project:
We have single CDK project with multiple stacks with in the same project. For example we have 1 stack with SNS topic and 1 stack for each lambda and its SNS subscription. We can use sns topic name accross stacks like documented here
const snsStack = new MySnsStack(app, 'my-sns-stack');
// each stack takes property topic as input, which behind the scenes perform cloudformation export and import.
new MyLambdaOne(app, 'Stack2', {
topic: snsStack.topic
});
new MyLambdaTwo(app, 'Stack2', {
topic: snsStack.topic
});
All we need to do is cdk deploy and stacks are arranged and deployed in proper sequence. i.e. sns stack first and rest of the lambda stacks next based on references.
Multiple CDK Projects :
We have 1 stack per cdk project. so, we have multiple CDK projects to maintain. We then have to manually export topic Arn using cfnOutput from the first stack and import topic arn in other stacks using Fn.ImportValue.
Then we need to run multiple deployes cdk deploy MySnsStack, cdk deploy MyLambdaStack , etc. separately. First the sns stack, rest of them in parallel.
I have one DynamoDB table, and there is a secondary index on the table.
But I have a faced duplication problem when I was query something.
I don't want my Lambda function don't trace secondary index...
I saw IAM policy but there is no relational policy.
How can I solve this problem? This is my lambda function: aws-dynamodb-to-elasticsearch/dynamodb-to-es.py at master ยท vladhoncharenko/aws-dynamodb-to-elasticsearch
This is probably because you have many Lambda functions or many Lambda Function versions in your account for that region.
[Total size of all the deployment packages that can be uploaded per region | 75 GB][1]
Looks like this is a pretty common problem for serverless and someone has developed a plugin to help alleviate this issue: https://github.com/claygregory/serverless-prune-plugin
If you want to deal with this manually you'll need to use either the console, or an sdk/cli to delete old lambda versions. https://docs.aws.amazon.com/cli/latest/reference/lambda/delete-function.html
In our "standard" AWS account, I have a system that does something like this:
CloudWatch Rule (Scheduled Event) -> Lambda function (accesses DynamoDB table, makes computations, writes metrics) -> CloudWatch Alarm (consume metrics, etc.)
However, in our separate CN account, we need to do a similar thing, but in CN, there's no Lambda...
Is there any way we can do something similar to what was done above using the systems available to CN? For example, is it possible to create a rule and have it trigger a lambda function in our "standard/nonCN" AWS account that access the other account's DynamoDB table?
I ultimately accomplished this by having the Lambda and the CloudWatch alarm live in the non-CN account, and then having the Lambda access the dynamoDB table across accounts and across regions.
This actually ended up working, though it did involve me using user credentials instead of a role like I would have been able to had it not been CN.
If anyone is interested in more details on this solution, feel free to comment and I can add more.
You can mix and match between AWS resources between regions. When you do your code, you need to make sure you have the regions correctly configured to those resources.
With respect to trigger, Have the trigger where ever you have your lambda. That will ease your process.
Hope it helps.
I have a large template for CloudFormation that has hundreds of resources. All are successfully updated, during an update, except one: an SNS alarm topic.
When deploying the stack, I get no errors, but even if the topic is non-existent the topic is never created.
I'm not expecting anyone to be able to provide me with a solution, but I would simply like to know how to troubleshoot the problem. It would be helpful to get output from the deployment, but the events are so few and really don't reflect the amount of resources being updated/created that they rarely help finding out what goes wrong.
Validation of the template is also successful, but that's almost a given since deploying also succeeds.
Regarding troubleshooting live CloudFormation stacks in general, CloudFormation just recently added support for Change Sets, which should help you preview changes and troubleshoot potential issues with updates before you attempt to apply them to your running stack.
Regarding SNS topics specifically, creating an SNS topic from scratch using the AWS::SNS::Topic resource works correctly. However, if you are using a TopicName property in your SNS resource, make sure that the name is unique across your entire AWS Account, as noted in the documentation:
Resource names must be unique across all of your active stacks. If you reuse templates to create multiple stacks, you must change or remove custom names from your template.
So reusing a constant TopicName in a stack deployed multiple times could cause the issue you're describing.
Also, if you're attempting to update existing CloudFormation-created topics with added/removed subscriptions, note the following important notice in the documentation:
Important
After you create an Amazon SNS topic, you cannot update its properties by using AWS CloudFormation. You can modify an Amazon SNS topic by using the AWS Management Console.
As a potential workaround for adding/removing subscriptions to existing SNS topics via CloudFormation, there is a cloudformation-helpers library containing a Custom::SnsSubscription resource (example) that might help.