I'm fairly new to AWS CDK, and I just created an architecture with an Aurora cluster inside of it. Is there a way to initialize a database schema inside of my CDK files? It seems a bit annoying to create the instance and then have to connect through other means in order to set up a schema. My impression of CDK is that you can do everything relating to setting up the services, so I was curious if this is a possibility.
If this isn't possible, can I use a lambda that fires off after the CDK deployment? Or maybe have a snapshot inside of s3? I'm just trying to find the best solution!
You can do this, although it requires you to execute it as a custom resource within the CDK.
You would need to create a Lambda function the performs the SQL DDL tasks for you. This would then return a successful flag back to the execution runtime to say that this "resource" was successfully created.
Related
I'm new to Cloud Formation. I want to update the settings of already created a lot of RDS instances using Cloud Formation. I don't have the info about either those resources were created through CF or manually. Is it possible to update such resources with CF?
I can think of another way like I can use AWS SDK (boto3) but doing it with CF is perefrable.
The only way to do this from CloudFormation (CF) is to develop your own CF custom resource. This will be a lambda function which will use AWS SDK to query the state of your RDS databases, and perform any actions you want.
Since its fully custom, you can program any logic which satisfies your requirements.
If the resources were created manually, you can also import them to CF, and then update using CF.
We are successfully spinning up an AWS CloudFormation stack that includes a serverless RDS PostgreSQL instance. Once the PostgreSQL instance is in place, we're automatically restoring a PostgreSQL database dump (in binary format) that was created using pg_dump on a local development machine on the instance PostgreSQL instance just created by CloudFormation.
We're using a Lambda function (instantiated by the CloudFormation process) that includes a build of the pg_restore executable within a Lambda layer and we've also packaged our database dump file within the Lambda.
The above approached seems complicated for something that presumably has been solved many times... but Google searches have revealed almost nothing that corresponds to our scenario. We may be thinking about our situation in the wrong way, so please feel free to offer a different approach (e.g., is there a CodePipeline/CodeBuild approach that would automate everything). Our preference would be to stick with the AWS toolset as much as possible.
This process will be run every time we deploy to a new environment (e.g., development, test, stage, pre-production, demonstration, alpha, beta, production, troubleshooting) potentially per release as part of our CI/CD practices.
Does anyone have advice or a blog post that illustrates another way to achieve our goal?
If you have provisioned everything via IaC (Infrastructure as Code) anyway that is most of the time-saving done and you should already be able to replicate your infrastructure to other accounts/ stacks via different roles in your AWS credentials and config files by passing the —profile some-profile flag. I would recommend AWS SAM (Serverless application model) over Cloudformation though as I find I only need to write ~1/2 the code (roles & policies are created for you mostly) and much better & faster feedback via the console. I would also recommend sam sync (it is in beta currently but worth using) so you don’t need to create a ‘change set’ on code updates, purely the code is updated so deploys take 3-4 secs. Some AWS SAM examples here, check both videos and patterns: https://serverlessland.com (official AWS site)
In terms of the restore of RDS, I’d probably create a base RDS then take a snapshot of that and restore all other RDS instances from snapshots rather than manually creating it all the time you are able to copy snapshots and in fact automate backups to snapshots cross-account (and cross-region if required) which should be a little cleaner
There is also a clean solution for replicating data instantaneously across AWS Accounts using AWS DMS (Data Migration Services) and CDC (Change Data Capture). So, process is you have a source DB and a target DB and also a 'Replication Instance' (eg EC2 Micro) that monitors your source DB and can replicate that across to different instances (so you are always in sync on data, for example if you have several devs working on separate 'stacks' to not pollute each others' logs, you can replicate data and changes out seamlessly if required from one DB) - so this works with Source DB and Destination DB both already in AWS however you can also use AWS DMS of course for local DB migration over to AWS, some more info here: https://docs.aws.amazon.com/dms/latest/sbs/chap-manageddatabases.html
Using the CDK is it possible to get an existing CfnDBCluster to make modifications to?
I have an AWS::RDS::DBCluster in CloudFormation whose TimeoutAction I want to change (CloudFormation doesn't support it and I don't want to use the AWS cli).
CDK doesn't natively support importing existing resources for modification.
https://medium.com/#visya/how-to-import-existing-aws-resources-into-cdk-stack-f1cea491e9
This article describes using CDK to generate the template, then use the AWS Management Console to import the resource into the stack.
Here is an issue to track the support within CDK itself: https://github.com/aws/aws-cdk-rfcs/issues/52
In your case, specifically since DB Clusters support this, you could create snapshot of the database then delete it. Then reference the snapshot id when creating recreating the cluster with CDK. Obviously it would require downtime though.
https://docs.aws.amazon.com/cdk/api/latest/docs/#aws-cdk_aws-rds.DatabaseClusterFromSnapshot.html
I have an existing VPC with a few EC2 instances already created and running with security groups, route tables, NACLs all applied where required. Is there a way to convert a setup like this into a cloudformation stack or a terraform equivalent of configuration files?
Update: A modern substitute for CloudFormer is Former2.
You can use CloudFormer, but it doesn't seem to be maintained much anymore.
See: Using CloudFormer (Beta) to Create AWS CloudFormation Templates from Existing AWS Resources - AWS CloudFormation
While that type of capability might sound like a good idea, it is quite difficult to make a template from running infrastructure. It runs into questions, such as:
Should the definition for an Amazon EC2 instance reference the specific Security Group that currently exists, or should it create another one and reference that new one?
Should it create a new VPC, or launch resources in the existing VPC?
Was an instance created directly, or was it launched by Auto Scaling?
In the end, it's probably easier to just write the CloudFormation template yourself, rather than spend time trying to 'fix' one that is automatically generated.
Some tips for writing CloudFormation templates:
Never write them by hand. Always copy the templates or examples from the documentation.
Copy snippets from your existing templates rather than doing them all from scratch again.
Use YAML rather than JSON (less errors due to unbalanced braces)
CloudFormation amateur here. Been looking online and can't find any references as to how I would go about creating my tables after my RDS instance is stood up through CloudFormation. Is it possible to specify a Lambda to launch and create all the tables, or maybe specify a SQL file to be applied? What's the standard pattern on this?
There aren't any CloudFormation resources which deal with the 'internals' of an RDS instance once it's been created; it's a black box which you're expected to configure (with users, tables and data) outside of CloudFormation. This is a bit sad, as versioning, managing and deploying database schemas is a classic deployment problem on the borderline between infrastructure and application, which is also exactly where CloudFormation sits (as a tool for versioning, managing and deploying other infrastructure).
What you could do, although it's not a beginner-level feature, is use a custom resource to connect to the RDS instance once it's created, and run appropriate SQL commands to create the users and tables. You define the schema in your CloudFormation template (either as SQL or a more structured description similar to DynamoDB AtributeDefinitions), and that schema is passed to your custom resource lambda function, which then runs the queries in the database.
If you go down this route, you'll probably do a lot of wheel-inventing of how to translate differences in the 'before' and 'after' schemas into ALTER TABLE SQL statements to fire at the database. Your custom resource lambda code needs to be very robust and always send a response back to CloudFormation, even in the case of an error. And don't forget that if your stack update fails for any reason, CloudFormation will call your custom resource again 'in reverse' (ie asking you to alter the database from the post-update schema back to the pre-update schema) as part of the rollback process.
If you do go down this road and come up with something at all robust, do publish it, because I'm sure a lot of people would be interested in it! I had a quick look online but I couldn't find anything obvious pre-existing.
Unfortunately, CF template cannot run the setup script directly. You can try, like what you have mentioned, using Lambda to run the table set up once the RDS has been created successfully by the CF template. DependsOn attribute helps. You should pass the credentials to Lambda to access the RDS.