We use Pivotal Cloud Foundry's YML file to set up the environment. Everything is fine. According to DEVOPS if we have to modify/create an environment variable, we have to modify YML and push the app again. I am wondering if it is possible to modify/create an environment variable while the PCF app is running. It will be really cool if it can be done without having to redeploy the app. If it can't be done, is it because of Java's way of handling the environment?
Thanks
Can we change/modify the environment in PCF (Pivotal Cloud Foundy) at Runtime?
Yes and no.
You can modify environment variables associated with an application while the application is running using the cf set-env (to set or update) and cf unset-env (to delete).
This will update the environment variable in Cloud Controller at the time you run the command. However, this will not update the environment variable inside of a running application container. In order for your application to see the change that you made, you must cf restart, cf restage or cf push.
This has nothing to do with language specifics (i.e. it doesn't matter what language you're using). It is a requirement because the container where your application is running gets created with a fixed set of environment variables. When those change, the container must be recreated. That said, even if the container could be changed at runtime, in Linux a process' environment variables cannot be updated externally at runtime either (there are technically some ways to do this, but it's really unlikely you'd do this in practice). The process itself should be restarted for environment variables to change.
If you want to have your configuration update at runtime, you can look at something like Spring Cloud Config server & its refresh capabilities. That said, it turns out that most applications and frameworks assume configuration is read once while the app is starting up, so your application would need to support changing the configuration you want to change at runtime as well.
I use this instruction:
http://docs.spring.io/spring-boot/docs/current/reference/html/deployment-install.html
to deploy Spring Boot application on EC2 Ubuntu server.
I'm going to move configuration from application.properties file to environment variables using:
spring.datasource.password=${MYAPP_DATASOURCE_PASSWORD}
If I'll make many copies of this configuration, how to manage it? For example, change datasource password on all instances?
I'll have to do it manually ? It'll going to be tedious, and error prone work...
"I think you'll have to invest some time in automation. Two things that come to mind can solve your problems i.e. Ansible and Docker. Or even plain AMIs with MYAPP_DATASOURCE_PASSWORD set in environment variables"
#anand1st answer solves the problem.
I'm new to the configuration management and deployment tools. I have to implement a Continuous Delivery/Continuous Deployment tool for one of the most interesting projects I've ever put my hands on.
First of all, individually, I'm comfortable with AWS, I know what Ansible is, the logic behind it and its purpose. I do not have same level of understanding of Docker but I got the idea. I went through a lot of Internet resources, but I can't get the the big picture.
What I've been struggling is how they fit together. Using Ansible, I can manage my Infrastructure as Code; building EC2 instances, installing packages... I can even deploy a full application by pulling its code, modify config files and start web server. Docker is, itself, a tool that packages an application and ensures that it can be run wherever you deploy it.
My problems are:
How does Docker (or Ansible and Docker) extend the Continuous Integration process!?
Suppose we have a source code repository, the team members finish working on a feature and they push their work. Jenkins detects this, runs all the acceptance/unit/integration test suites and if they all passed, it declares it as a stable build. How Docker fits here? I mean when the team pushes their work, does Jenkins have to pull the Docker file source coded within the app, build the image of the application, start the container and run all the tests against it or it runs the tests the classic way and if all is good then it builds the Docker image from the Docker file and saves it in a private place?
Should Jenkins tag the final image using x.y.z for example!?
Docker containers configuration :
Suppose we have an image built by Jenkins stored somewhere, how to handle deploying the same image into different environments, and even, different configurations parameters ( Vhosts config, DB hosts, Queues URLs, S3 endpoints, etc...) What is the most flexible way to deal with this issue without breaking Docker principles? Are these configurations backed in the image when it gets build or when the container based on it is started, if so how are they injected?
Ansible and Docker:
Ansible provides a Docker module to manage Docker containers. Assuming I solved the problems mentioned above, when I want to deploy a new version x.t.z of my app, I tell Ansible to pull that image from where it was stored on, start the app container, so how to inject the configuration settings!? Does Ansible have to log in the Docker image, before it's running ( this sounds insane to me ) and use its Jinja2 templates the same way with a classic host!? If not, how is this handled?!
Excuse me if it was a long question or if I misspelled something, but this is my thinking out loud. I'm blocked for the past two weeks and I can't figure out the correct workflow. I want this to be a reference for future readers.
Please, it would very helpful to read your experiences and solutions because this looks like a common workflow.
I would like to answer in parts
How does Docker (or Ansible and Docker) extend the Continuous Integration process!?
Since docker images same everywhere, you use your docker images as if they are production images. Therefore, when somebody committed a code, you build your docker image. You run tests against it. When all tests pass, you tag that image accordingly. Since docker is fast, this is a feasible workflow.
Also docker changes are incremental; therefore, your images will have minimal impact on storage. Also when your tests fail, you may also choose to save that image too. In this way, developer will pull that image and investigate easily why your tests failed. Developer may choose to run tests in their machine too since docker images in jenkins and their machine are not different.
What this brings that all developers will have same environment, same version of all software since you decide which one will be used in docker images. I have come across to bugs that are due to differences between developer machines. For example in the same operating system, unicode settings may affect your code. But in docker images all developers will test against same settings, same version software.
Docker containers configuration :
If you are using a private repository, and you should use one, then configuration changes will not affect hard disk space much. Therefore except security configurations, such as db passwords, you can apply configuration changes to docker images(Baking the Configuration into the Container). Then you can use ansible to apply not-stored configurations to deployed images before/after startup using environment variables or Docker Volumes.
https://dantehranian.wordpress.com/2015/03/25/how-should-i-get-application-configuration-into-my-docker-containers/
Does Ansible have to log in the Docker image, before it's running (
this sounds insane to me ) and use its Jinja2 templates the same way
with a classic host!? If not, how is this handled?!
No, ansible will not log in the Docker image, but ansible with Jinja2 templates can be used to change dockerfile. You can change dockerfile with templates and can inject your configuration to different files. Tag your files accordingly and you have configured images to spin up.
Regarding your question about handling multiple environment configurations using the same Docker image, I have been planning on using a Service Discovery tool like Consul as a centralized config/property management tool. So, when you start your container up, you set an ENV var that tells it what application it is (appID), and what environment config it should use (ex: MyApplication:Dev) and it will pull its config from Consul at startup. I still have to investigate the security around Consul (as if we are storing DB connection credentials in there for example, how do we restrict who can query/update those values). I don't want to just use this for containers, but all apps in general. Another cool capability is to change the config value in Consul and have a hook back into your app to apply the changes immediately (maybe like a REST endpoint on your app to push changes down to and dynamically apply it). Of course your app has to be written to support this!
You might be interested in checking out Martin Fowler's blog articles on immutable infrastructure and on Phoenix servers.
Although not a complete solution, I have suggestions for two of your issues. Although they might not be perfect, these are the practices we are using in our workflow, and prove themselves so far.
Defining different environments - supposing you've written a different Ansible role for each environment you launch, we define an environment variable setting the environment we wish the container to belong to. We then download the suitable configuration file from an S3 bucket using the env variable set before into the container (which should be possible if you supply AWS creds or give your server an IAM role) and inject these parameters into the code when building it.
Ansible doesn't need to log into the docker app, but the solution is a bit tricky. I've tried two ways of tackling this problem, and both aren't ideal. The first one is to download the configuration file as part of the docker image command line, and build the app on container startup. While this solution works - it breaches the Docker philosophy and makes the image highly prone to build errors.
Another solution is pushing several images to your docker hub repo, and then pulling the appropriate image according to the environment at hand.
In a broader stroke, I've tried launching our app completely with Ansible and it was hell, many configuration steps are tricky and get trickier when you try to implement them as a playbook. When I switched to maintaining the severs alone with Ansible, and deploying the app itself with Docker things got a lot easier.
I'm trying to automate the process of deploying code using github and jenkins job to deploy my Springboot Application on AWS .
I want to know where should I place the application.properties file in case I m deploying a war file on Tomcat and don't want this file to be pushed onto github as it may contain some database credentials , not to be exposed.
Should I put separate application-prod.properties file in Tomcat (AWS) so that my war file will be independent of these properties ?
See my answer here.
In a nutshell, you externalise the properties and then pass one or more profiles that will activate one or more Spring Configuration classes. Each Configuration class will load one or more property file. In your case, if you only have one environment, you can just create a configuration file for one profile.
Then, on your AWS instance, you will deploy the configuration file separately. At runtime, you will need to pass to your Spring Boot application the active profile(s). You can do this by passing the VM argument: -Dspring.profiles.active=[your-profile]
I'm completing the final lectures on an online course that shows how to create from scratch a Spring Boot website with Thymeleaf, Spring Security, Email and Data JPA, how to process credit card payments with Stripe and how to deploy to AWS. You can register your interest here.
how about using spring-cloud-starter-config instead of local properties ?
If using spring-cloud-start-config, all configurations should be loaded from your config-center instead of reading them locally.
Even if you have multiple different environments, spring-cloud-starter-config could handle it with different profiles.
What's more, spring-cloud-starter-config could use local environment variables too.
By the way, the only local resource could be bootstrap.yml if you are using spring-cloud-starter-config.
Wish I can help you!
I am currently using AWS ElasticBeanStalk and I was curious as to how (as in internally) it knows that when you fire up an instance (or it automatically does with scaling), to unpack the zip I uploaded as a version? Is there some enviroment setting that looks up my zip in my S3 bucket and then unpacks automatically for every instance running in that environment?
If so, could this be used to automate a task such as run an SQL query on boot-up (instance deployment) too? Are these automated tasks changeable or viewable at all?
Thanks
I don't know how beanstalk knows which version to download and unpack, but running a task on start-up is trivial. Check out cloud-init, a tool written by Ubuntu that's now packaged in Amazon Linux. It allows you to pass arbitrary shell scripts into the UserData section of the instance configuration, and those shell scripts will run on startup.
It's a great way to bootstrap instances on startup, which avoids the soul-sucking misery of managing AMIs.
A quick (possibly non-applicable) warning: If you're running a SQL query on a database that lives on the beanstalk AMI, you're pretty much guaranteed to lose your database at some point. Those machines are designed to be entirely transient. Do not put databases on them. See this answer for more details.
Since your goal seems to be to run custom configuration tasks, the answer is yes, there is a way to do that. You can define custom actions in an .ebextensions file packaged with your app. For example, you can configure a command to run every time a new machine is deployed:
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html#linux-commands