We are increasingly using AWS Parameter Store for managing configuration.
One issue we have is managing which variables need to be set when releases occur to different environments (staging, dev, prod etc). There is always a lot of work to configure the different environments, and it is easy to overlook required settings when we release microservices.
It seems what is needed is a database migration similar to Flyway or Liquibase, but I haven't found any products available, and it is unclear to me how secrets would be managed with this system.
What are people doing to manage pushing new configuration into Parameter Store when new application code is deployed?
Do you know AWS AppConfig? It’s a different way of managing configuration and I’m not sure if this fits your requirements but it might be worth a look.
Related
I am building a lab utility to deploy my teams development environments (testing / stress etc).
At present, the pipeline is as follows:
Trigger pipeline via HTTP request, args contain the distribution, web server and web server version using ARGs that are passed too multi stage dockerfiles.
Dockerx builds the container (if it doesn't exist in ECR)
Pipeline job pushes that container to ECR (if it doesn't already exist).
Terraform deploys the container using Fargate, sets up VPCs and a ALB to handle ingress externally.
FQDN / TLS is then provisioned on ...com
Previously when I've made tools like this that create environments, environments were managed and deleted solely at project level, given each environment had it's own project, given this is best practice for isolation and billing tracking purposes, however given the organisation security constraints of my company, I am limited to only 1 project wherein I can create all the resources.
This means I have to find a way to manage/deploy 30 (the max) environments in one project without it being a bit of a clustered duck.
More or less, I am looking for a way that allows me to keep track, and tear down environments (autonomously) and their associated resources relevant to a certain identifier, most likely these environments can be separated by resource tags/groups.
It appears the CDKTF/Pulumi looks like a neat way of achieving some form of "high-level" structure, but I am struggling to find ways to use them to do what I want. If anyone can recommend an approach, it'd be appreciated.
I have not tried anything yet, mainly because this is something that requires planning before I start work on it (don't like reaching deadends ha).
I'm presently looking into GCP's Deployment Manager to deploy new projects, VMs and Cloud Storage buckets.
We need a web front end that authenticated users can connect to in order to deploy the required infrastructure, though I'm not sure what Dev Ops tools are recommended to work with this system. We have an instance of Jenkins and Octopus Deploy, though I see on Google's Configuration Management page (https://cloud.google.com/solutions/configuration-management) they suggest other tools like Ansible, Chef, Puppet and Saltstack.
I'm supposing that through one of these I can update something simple like a name variable in the config.yaml file and deploy a project.
Could I also ensure a chosen name for a project, VM or Cloud Storage bucket fits with a specific naming convention with one of these systems?
Which system do others use and why?
I use Deployment Manager, as all 3rd party tools are reliant upon the presence of GCP APIs, as well as trusting that those APIs are in line with the actual functionality of the underlying GCP tech.
GCP is decidedly behind the curve on API development, which means that even if you wanted to use TF or whatever, at some point you're going to be stuck inside the SDK, anyway. So that's why I went with Deployment Manager, as much as I wanted to have my whole infra/app deployment use other tools that I was more comfortable with.
To specifically answer your question about validating naming schema, what you would probably want to do is write a wrapper script that uses the gcloud deployment-manager subcommand. Do your validation in the wrapper script, then run the gcloud deployment-manager stuff.
Word of warning about Deployment Manager: it makes troubleshooting very difficult. Very often it will obscure the error that can help you actually establish the root cause of a problem. I can't tell you how many times somebody in my office has shouted "UGGH! Shut UP with your Error 400!" I hope that Google takes note from my pointed survey feedback and refactors DM to pass the original error through.
Anyway, hope this helps. GCP has come a long way, but they've still got work to do.
I have an EBS backed EC2 web server. I am trying to get to the point where I can automate launching an identical server (identical in terms of software and config).
From my understanding there are two ways to achieve this:
Install all required software, make config changes and then create a custom AMI from the EBS volume. Use this AMI to launch any future instances.
Use the original AMI and use a Bootstrap Script install all the required software and make necessary config changes.
To me, creating a custom AMI and not having to write a script seems a lot easier initially. But Im aware that any modifications to the server will mean creating a new AMI. I can also see how a bootstrap script is much more transparent and allows one to easily inspect what is going to be installed/configured.
I wondered if anyone could point out anymore advantages/disadvantages between the two approaches. Is there one which is considered best practice?
If you create custom AMI:
It'll be faster to provision new servers.
It'll be easy to maintain the unification of the environment.
It'll be difficult to update the AMI in-case of any modifications.
If you use bootstrapping script:
It'll be easier to modify in case of any changes.
It'll be hard to maintain the unification of the environment (e.g. can face versioning issues).
Slow to provision new servers.
BTW, how about using AMI for things that are less likely to change and then using bootstrap scripts e.g. user data to do things that'll change over time.
I am trying to build out our AWS environments using Terraform but am hitting some issues scaling. I have a repository of just modules that I want to use repeatedly when building my environments and a second repository just to handle the actual implementations of those modules.
I am aware of HashiCorp's Github page that has an example but there, each environment is one state file. I want to split environments out but then have multiple state files within each environment. When the state files get big, applying small updates takes way too long.
Every example I've seen where multiple state files are used, the Terraform files are extremely un-DRY and not ideal.
I would prefer to be able to have different variable values between environments but have the same configuration.
Has anyone ever done anything like this? Am I missing something? I'm a bit frustrated because every Terraform example is never at scale and it makes it hard for n00b such as myself to start down the right path. Any help or suggestions is very much appreciated!
The idea of environment unfortunately tends to mean different things to different people and organizations.
To some, it's simply creating multiple copies of some infrastructure -- possibly only temporary, or possibly long-lived -- to allow for testing and experimentation in one without affecting another (probably production) environment.
For others, it's a first-class construct in a deployment architecture, with the environment serving as a container into which other applications and infrastructure are deployed. In this case, there are often multiple separate Terraform configurations that each have a set of resources in each environment, sharing data to create a larger system from smaller parts.
Terraform has a feature called State Environments that serves the first of these use-cases by allowing multiple named states to exist concurrently for a given configuration, and allowing the user to switch between them using the terraform env commands to focus change operations on a particular state.
The State Environments feature alone is not sufficient for the second use-case, since it only deals with multiple states in a single configuration. However, it can be used in conjunction with other Terraform features, making use of the ${terraform.env} interpolation value to deal with differences, to allow multiple state environments within a single configuration to interact with a corresponding set of state environments within another configuration.
One "at scale" approach (relatively-speaking) is described in my series of articles Terraform Environment+Application Pattern, which describes a generalization of a successful deployment architecture with many separate applications deployed together to form an environment.
In that pattern, the environments themselves (which serve as the "container" for applications, as described above) are each created with a separate Terraform configuration, allowing each to differ in the details of how it is configured, but they each expose data in a standard way to allow multiple applications -- each using the State Environments feature -- to be deployed once for each environment using the same configuration.
This compromise leads to some duplication between the environment configurations -- which can be mitgated by using Terraform modules to share patterns between them -- but these then serve as a foundation to allow other configurations to be generalized and deployed multiple times without such duplication.
Amazon Web Services offer a number of continuous deployment and management tools such as Elastic Beanstalk, OpsWorks, Cloud Formation and Code Deploy depending on your needs. The basic idea being to facilitate code deployment and upgrade with zero downtime. They also help manage best architectural practice using AWS resources.
For simplicity lets assuming a basic architecture where you have a 2 tear structure; a collection of application servers behind a load balancer and then a persistence layer using a multi-zone RDS DB.
The actual code upgrade across a fleet of instances (app servers) is easy to understand. For a very simplistic overview the AWS service upgrades each node in turn handing connections off so the instance in question is not being used.
However, I can't understand how DB upgrades are managed. Assume that we are going from version 1.0.0 to 2.0.0 of an application and that there is a requirement to change the DB structure. Normally you would use a script or a library like Flyway to perform the upgrade. However, if there is a fleet of servers to upgrade there is a point where both 1.0.0 and 2.0.0 applications exist across the fleet each requiring a different DB structure.
I need to understand how this is actually achieved (high level) to know what the best way/time of performing the DB migration is. I guess there are a couple of ways they could be achieving this but I am struggling to see how they can do it and allow both 1.0.0 and 2.0.0 to persist data without loss.
If they migrate the DB structure with the first app node upgrade and at the same time create a cached version of the 1.0.0. Users connected to the 1.0.0 app persist using the cached version of the DB and users connected to the 2.0.0 app persist to the new migrated DB. Once all the app nodes are migrated, the cached data is merged into the DB.
It seems unlikely they can do this as the merge would be pretty complex but I can't see another way. Any pointers/help would be appreciated.
This is a common problem to encounter once your application infrastructure gets into multiple application nodes. In the olden days, you could take your application offline for "maintenance windows" during which you could:
Replace application with a "System Maintenance, back soon" page.
Perform database migrations (schema and/or data)
Deploy new application code
Put application back online
In 2015, and really for several years this approach is not acceptable. Your users expect 24/7 operation, so there must be a better way. Of course there is, the answer is a series of patterns for Database Refactorings.
The basic concept to always keep in mind is to assume you have to maintain two concurrent versions of your application, and there can be no breaking changes between these two versions. This means that you have a current application (v1.0.0) currently in production and (v2.0.0) that is scheduled to be deployed. Both these versions must work on the same schema. Once v2.0.0 is fully deployed across all application servers, you can then develop v3.0.0 that allows you to complete any final database changes.