I'm trying to understand how to implement Pulumi in our AWS environment.
I understand that a stack can be used to have the same resource structure for production and development, but that results in independant instances, which is great.
However we also have a shared management VPC where certain tools reside such as pgadmin, gitlab, monitoring tools,...
So what would be the best approach for that?
Force a stack for that management project (if that is possible)?
Put a constraint in code so that this management stuff is only deployed in production stack?
Or am I missing the approach of stacks here?
Your first thought is the right way to go.
Deploy shared resources in their own stack(s) and use Stack References to share information across stacks.
These links discuss these concepts:
https://www.pulumi.com/docs/intro/concepts/organizing-stacks-projects/
https://www.pulumi.com/docs/intro/concepts/programming-model/#stack-references
Related
I am trying to move a Dynamo table from one stack to another while keeping data.
I found this documentation for CloudFormation. However the issue is that once I run serverless deploy it does not recognise the link and creates a new table with the same name.
I have an option to run a script and move data manually, but it would be nice to have a mechanism to move/refactor resources around for non-Dynamo resources as well.
Does anyone have experience with these kind of issues?
Thanks.
It's more of an open question and I'm just hoping for any opinions and suggestions. I have AWS in mind but it probably can relate also to other cloud providers.
I'd like to provision IaaC solution that will be easily maintainable and cover all the requirements of modern serverless architecture. Terraform is a great tool for defining the infrastructure, has many official resources and stable support from the community. I really like its syntax and the whole concept of modules. However, it's quite bad for working with Lambdas. It also raises another question: should code change be deployed using the same flow as infrastructure change? Where to draw the line between code and infrastructure?
On the other hand, Serverless Framework allows for super easy development and deployment of Lambdas. It's strongly opinionated when it comes to the usage of resources but it comes with some many out-of-the-box features that it's worth it. It shouldn't really be used for defining the whole infrastructure.
My current approach is to define any shared resources using Terraform and any domain-related resources using Serverless. Here I have another issue that is related to my previous questions: deployment dependency. The simple scenario: Lambda.1 adds users to Cognito (shared resource) which has Lambda.2 as a trigger. I have to create a custom solution for managing the deployment order (Lambda.2 has to be deployed first, etc.). It's possible to hook up the Serverless Framework deployment into Terraform but then again: should the code deployment be mixed with infrastructure deployment?
It is totally possible to mix the two and I have had to do so a few times. How this looks actually ends up being simpler than it seems.
First off, if you think about whatever you do with the Serverless Framework as developing microservices (without the associated infrastructure management burden), that takes it one step in the right direction. Then, what you can do is decide that everything that is required to make that microservice work internally is defined within that microservice as a part of the services configuration in the serverless.yml, whether that be DynamoDB tables, Auth0 integrations, Kinesis streams, SQS, SNS, IAM permissions allocated to functions, etc. Keep that all defined as a part of that microservice. Terraform not required.
Now think about what that and other microservices might need to interact with more broadly. They aren't critical for that services internal operation but are critical for integration into the rest of the organisations infrastructure. This includes things like deployment IAM roles used by the Serverless Framework services to deploy into CloudFormation, Relational Databases that have to be shared amongst multiple services and resources, networking elements (VPC's, Security Groups, etc), monolithic clusters like ElasticSearch and Redis ... all of these elements are great candidates for definition outside of the Serverless Framework and work really well with Terraform.
Any resource would be able to connect to these Terraform defined resource as needed, unlike that hard association such as Lambda functions triggered off of an API Gateway endpoint.
Hope that helps
I have created an infrastructure with terraform and now I need to setup a CI... I'm thinking of using terraform also. Is it possible to extract certain part of tf code to place to Pipeline in order of updating ECS tasks ignoring the rest of infrastructure?
As ydaetskcoR suggested in a comment, if you really want to run parts of your Terraform configuration independently of the rest, you're better off splitting it up.
What I'd suggest is several terraform projects grouped in the way you'd organize responsibility and releases, (e.g. projects might have their own, VPC might be on its own, some shared infrastructure in its own), and use Terraform remote state to connect them all.
In an architecture where you have all of your components using aws-specific products, like web servers in EC2 instances, your CDN using CloudFront, microservices in ECS, why would someone choose to use Terraform?
It's usually very individual choice. There are some aspects (not really pros and cons, but just points to which you need to answer):
Are you going to be fully AWS-oriented? If yes, of course you can use both, but if not, you probably anyway (if not now, then in future) will need some additional tools, so in such case Terraform can cover both cases.
Do you already know Terraform? If not, then learning either is probably the same challenge, but if you know Terraform from other projects, you can just start using it. That also apply to your hiring strategy - are you going to look for people who know just AWS (then most likely CloudFormation as well), or in general with some Sys/Dev-Ops experience, then Terraform is a common tool.
There is one another important thing - CloudFormation can't handle existing resources (Terraform usually can, with import, however there are exceptions). That means you can migrate from CloudFormation to Terraform (done it few times, not a big deal in fact), but not the oposite.
I don't want to recommend a tool on itself, but more as an approach: start with CloudFormation and then see if it fits you or not. As mentioned in last point, you can always move to Terraform, but if you start witout CloudFormation, then the doors are closed for you.
Also if you are going to be fully AWS-oriented, CloudFormation can play nicely with more of the services: CodePipeline integrates with it, in case of CI/CD pipeline CloudFormation is a fully managed service (it keeps the states, exposes export values etc.), while in case of Terraform you need to provide own infrastructure for states storage and exposing outputs (you can use for example S3 bucket in AWS for that, but still you need to configure Terraform to use it - CloudFormation is a zero-config in such case).
I am new to AWS (Amazon Web Services) as well as our own custom boto based python deployment scripts, but wanted to ask for advice or best practices for a simple configuration management task. We have a simple web application with configuration data for several different backend environments controlled by a command line -D defined java environment variable. Sometimes, the requirement comes up that we need to switch from one backend environment to another due to maintenance or deployment schedules of our backend services.
The current procedure requires python scripts to completely destroy and rebuild all the virtual infrastructure (load balancers, auto scale groups, etc.) to redeploy the application with a change to the command line parameter. On a traditional server infrastructure, we would log in to the management console of the container, change the variable, bounce the container, and we're done.
Is there a best practice for this operation on AWS environments, or is the complete destruction and rebuilding of all the pieces the only way to accomplish this task in an AWS environment?
It depends on what resources you have to change. AWS is evolving everyday in a fast paced manner. I would suggest you to take a look at the AWS API for the resources you need to deal with and check if you can change a resource without destroying it.
Ex: today you cannot change a Launch Group once it is created. you must delete it and create it again with the new configurations. but if you have one auto scaling group attached to that launch group you will have to delete the auto scaling group and so on.
IMHO a see no problems with your approach, but as I believe that there is always room for improvement, I think you can refactor it with the help of AWS API documentation.
HTH
I think I found the answer to my own question. I know the interface to AWS is constantly changing, and I don't think this functionality is available yet in the Python boto library, but the ability I was looking for is best described as "Modifying Attributes of a Stopped Instance" with --user-data as being the attribute in question. Documentation for performing this action using HTTP requests and the command line interface to AWS can be found here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_ChangingAttributesWhileInstanceStopped.html