I am trying to move a Dynamo table from one stack to another while keeping data.
I found this documentation for CloudFormation. However the issue is that once I run serverless deploy it does not recognise the link and creates a new table with the same name.
I have an option to run a script and move data manually, but it would be nice to have a mechanism to move/refactor resources around for non-Dynamo resources as well.
Does anyone have experience with these kind of issues?
Thanks.
Related
We are successfully spinning up an AWS CloudFormation stack that includes a serverless RDS PostgreSQL instance. Once the PostgreSQL instance is in place, we're automatically restoring a PostgreSQL database dump (in binary format) that was created using pg_dump on a local development machine on the instance PostgreSQL instance just created by CloudFormation.
We're using a Lambda function (instantiated by the CloudFormation process) that includes a build of the pg_restore executable within a Lambda layer and we've also packaged our database dump file within the Lambda.
The above approached seems complicated for something that presumably has been solved many times... but Google searches have revealed almost nothing that corresponds to our scenario. We may be thinking about our situation in the wrong way, so please feel free to offer a different approach (e.g., is there a CodePipeline/CodeBuild approach that would automate everything). Our preference would be to stick with the AWS toolset as much as possible.
This process will be run every time we deploy to a new environment (e.g., development, test, stage, pre-production, demonstration, alpha, beta, production, troubleshooting) potentially per release as part of our CI/CD practices.
Does anyone have advice or a blog post that illustrates another way to achieve our goal?
If you have provisioned everything via IaC (Infrastructure as Code) anyway that is most of the time-saving done and you should already be able to replicate your infrastructure to other accounts/ stacks via different roles in your AWS credentials and config files by passing the —profile some-profile flag. I would recommend AWS SAM (Serverless application model) over Cloudformation though as I find I only need to write ~1/2 the code (roles & policies are created for you mostly) and much better & faster feedback via the console. I would also recommend sam sync (it is in beta currently but worth using) so you don’t need to create a ‘change set’ on code updates, purely the code is updated so deploys take 3-4 secs. Some AWS SAM examples here, check both videos and patterns: https://serverlessland.com (official AWS site)
In terms of the restore of RDS, I’d probably create a base RDS then take a snapshot of that and restore all other RDS instances from snapshots rather than manually creating it all the time you are able to copy snapshots and in fact automate backups to snapshots cross-account (and cross-region if required) which should be a little cleaner
There is also a clean solution for replicating data instantaneously across AWS Accounts using AWS DMS (Data Migration Services) and CDC (Change Data Capture). So, process is you have a source DB and a target DB and also a 'Replication Instance' (eg EC2 Micro) that monitors your source DB and can replicate that across to different instances (so you are always in sync on data, for example if you have several devs working on separate 'stacks' to not pollute each others' logs, you can replicate data and changes out seamlessly if required from one DB) - so this works with Source DB and Destination DB both already in AWS however you can also use AWS DMS of course for local DB migration over to AWS, some more info here: https://docs.aws.amazon.com/dms/latest/sbs/chap-manageddatabases.html
I'm trying to understand how to implement Pulumi in our AWS environment.
I understand that a stack can be used to have the same resource structure for production and development, but that results in independant instances, which is great.
However we also have a shared management VPC where certain tools reside such as pgadmin, gitlab, monitoring tools,...
So what would be the best approach for that?
Force a stack for that management project (if that is possible)?
Put a constraint in code so that this management stuff is only deployed in production stack?
Or am I missing the approach of stacks here?
Your first thought is the right way to go.
Deploy shared resources in their own stack(s) and use Stack References to share information across stacks.
These links discuss these concepts:
https://www.pulumi.com/docs/intro/concepts/organizing-stacks-projects/
https://www.pulumi.com/docs/intro/concepts/programming-model/#stack-references
I'm fairly new to AWS CDK, and I just created an architecture with an Aurora cluster inside of it. Is there a way to initialize a database schema inside of my CDK files? It seems a bit annoying to create the instance and then have to connect through other means in order to set up a schema. My impression of CDK is that you can do everything relating to setting up the services, so I was curious if this is a possibility.
If this isn't possible, can I use a lambda that fires off after the CDK deployment? Or maybe have a snapshot inside of s3? I'm just trying to find the best solution!
You can do this, although it requires you to execute it as a custom resource within the CDK.
You would need to create a Lambda function the performs the SQL DDL tasks for you. This would then return a successful flag back to the execution runtime to say that this "resource" was successfully created.
CloudFormation amateur here. Been looking online and can't find any references as to how I would go about creating my tables after my RDS instance is stood up through CloudFormation. Is it possible to specify a Lambda to launch and create all the tables, or maybe specify a SQL file to be applied? What's the standard pattern on this?
There aren't any CloudFormation resources which deal with the 'internals' of an RDS instance once it's been created; it's a black box which you're expected to configure (with users, tables and data) outside of CloudFormation. This is a bit sad, as versioning, managing and deploying database schemas is a classic deployment problem on the borderline between infrastructure and application, which is also exactly where CloudFormation sits (as a tool for versioning, managing and deploying other infrastructure).
What you could do, although it's not a beginner-level feature, is use a custom resource to connect to the RDS instance once it's created, and run appropriate SQL commands to create the users and tables. You define the schema in your CloudFormation template (either as SQL or a more structured description similar to DynamoDB AtributeDefinitions), and that schema is passed to your custom resource lambda function, which then runs the queries in the database.
If you go down this route, you'll probably do a lot of wheel-inventing of how to translate differences in the 'before' and 'after' schemas into ALTER TABLE SQL statements to fire at the database. Your custom resource lambda code needs to be very robust and always send a response back to CloudFormation, even in the case of an error. And don't forget that if your stack update fails for any reason, CloudFormation will call your custom resource again 'in reverse' (ie asking you to alter the database from the post-update schema back to the pre-update schema) as part of the rollback process.
If you do go down this road and come up with something at all robust, do publish it, because I'm sure a lot of people would be interested in it! I had a quick look online but I couldn't find anything obvious pre-existing.
Unfortunately, CF template cannot run the setup script directly. You can try, like what you have mentioned, using Lambda to run the table set up once the RDS has been created successfully by the CF template. DependsOn attribute helps. You should pass the credentials to Lambda to access the RDS.
I have a web site hosted in AWS that makes use of a number of AWS services. The environment was created manually using a combination of the web console and the AWS CLI. I'd like to start managing it using CloudFormation. I've used the CouldFormer tool to create a template of the stack but I can't find a way to use it to manage the existing environment. It will allow me to create a duplicate environment without too many problems but I don't really want to delete the entire production environment so I can recreate it using CloudFormation.
Is there a way to create a template of an existing environment and start updating it with CloudFormation?
#Sailor is right... unfortunately, I can't quote a credible source either - it's just a combination of working with Cloud Formation for an extended period of time and knowing enough about it. (Maybe I'm the credible source)
But what you could do is use your Cloud Former stack, and roll your existing production infrastructure into it.
For example, if you've got some EC2 images and a scaling group - roll that out, and then start terminating the others. How you'd do it would depend on your environment, but if it's architected for the cloud, it shouldn't be too difficult.
Currently there is not way to do that.