We use Pivotal Cloud Foundry's YML file to set up the environment. Everything is fine. According to DEVOPS if we have to modify/create an environment variable, we have to modify YML and push the app again. I am wondering if it is possible to modify/create an environment variable while the PCF app is running. It will be really cool if it can be done without having to redeploy the app. If it can't be done, is it because of Java's way of handling the environment?
Thanks
Can we change/modify the environment in PCF (Pivotal Cloud Foundy) at Runtime?
Yes and no.
You can modify environment variables associated with an application while the application is running using the cf set-env (to set or update) and cf unset-env (to delete).
This will update the environment variable in Cloud Controller at the time you run the command. However, this will not update the environment variable inside of a running application container. In order for your application to see the change that you made, you must cf restart, cf restage or cf push.
This has nothing to do with language specifics (i.e. it doesn't matter what language you're using). It is a requirement because the container where your application is running gets created with a fixed set of environment variables. When those change, the container must be recreated. That said, even if the container could be changed at runtime, in Linux a process' environment variables cannot be updated externally at runtime either (there are technically some ways to do this, but it's really unlikely you'd do this in practice). The process itself should be restarted for environment variables to change.
If you want to have your configuration update at runtime, you can look at something like Spring Cloud Config server & its refresh capabilities. That said, it turns out that most applications and frameworks assume configuration is read once while the app is starting up, so your application would need to support changing the configuration you want to change at runtime as well.
Related
I have a app which is pushed to cf using cf push. Now I have changed the one of the environment variable and then restage the app using cf restage. What I understand is when we do a restage it will again compile the droplet and build it same can be done with cf push again for the app. So what I want to know is the difference between the 2 commands and how internally cf handles this?
The difference is that one uploads files and one does not.
When you run cf push, the cli is going to take bits off your local file system, upload them, stage your app and if that succeeds, run your app. This is what you want if you have made changes to files in your app and want to deploy them.
When you cf restage, the cli is not going to upload anything. It will just restage your app, and if that succeeds, run the app. This is what you want if there are no app change or you do not have the app source code, yet you want to force the build packs to run again & restart your app using the new droplet.
When you cf restart, the cli won't upload or restage, it will just stop and start the app. This is the fastest option, but only works if you just need to pick up environment changes like a changed service, memory limit or environment variables. It's also good, if you just want to try and have your app placed onto a different Diego Cell.
If you are just making changes to environment variables, you can probably get away with a cf restart, unless those environment variables are being used by one of your buildpacks, like JBP_CONFIG_*.
Hope that helps!
Regarding
So what I want to know is the difference between the 2 commands
cf push internally does the following sequence
uploading : creating a package from your app
staging : creating a container image using the package
starting : starting the container as a long running process, based on the container image
Some use cases where we need to restage
To introduce buildpack changes : CF operators update the buildpacks for various reasons (one example: security patches) effectively updating the runtime dependencies which our apps use.
Introduction of new software components : It might be possible that we need to introduce new / additional components which were not foreseen at the time of initial push. An example can be : introduction of monitoring agent for your app
When the root file system is changed.
In all the 3 scenarios above, our application code is not changing but we need to re-create the container image. In these cases we need to restage. If we compare with cf push , in case of restage we skip the package creation step.
My group is using Jenkins to build and deploy a Spring Boot application to AWS (Elastic Beanstalk).
For development I was selecting the desired profile for the environment I was in (DEV, QA) by setting the spring.profiles.active property in application.properties.
But dev-ops would like to set the property externally. Since the app is being deployed to a SpotInst managed EC2, I have no control over the environment.
I've been doing a lot of searching, but haven't as yet come across anything that fits this situation.
I'm using Gradle for building. The build generates a WAR file and is deployed to Tomcat.
Is there anyway to accomplish this with some sort of parameter substitution?
Thanks.
Our devOps person discovered that we can set environment properties through AWS Beanstalk. By going to Configuration on the beanstalk instance and clicking on the Modify button under Software, scroll down and there is a section for environment variables. He put in spring.profiles.active=qa and that took care of it.
Since we have one beanstalk for each separate environment, it only has to be set once for each.
I am not sure if this will fit into your devOps process but there is very easy and clean way to achieve this.
Configure your application.yaml file like below:
spring:
profiles:
active: ${SPRING_PROFILE:"dev"}
And while starting the application pass this parameter SPRING_PROFILE as command line argument. Note that dev is default value which will be used if the program doesn't find any matching command line argument or environment variable.
You can set the default value to the one you want to use for devOps while for all other purpose where you can control you can pass as argument.
In Google shell which is a part of Google cloud, I set environment variable GOOGLE_APPLICATION_CREDENTIALS because It is need it for PHP NLP project [info: https://cloud.google.com/natural-language/docs/quickstart-client-libraries#client-libraries-install-php]. My project worked fine, but I notice that variable GOOGLE_APPLICATION_CREDENTIALS lasts on my sistem only one day. This is my third time that I am setting it. My project doesn't work when I am missing required variable. Am I doing something wrong?
EDIT:
It is default OS (Debian) when you create new App on Google App engine.
When I type help in Google shell I get info with:
Your 5GB home directory will persist across sessions, but the VM is ephemeral and will be reset
approximately 20 minutes after your session ends. No system-wide change will persist beyond that.
You are completely right, Cloud Shell is running on an ephemeral instance that resets some minutes after the session has ended, reason why you are losing the content of the environment variable you mentioned.
The documentation about limitations in Cloud Shell clearly states that it is intended for interactive use only, and any non-interactive session or intensive usage can be automatically terminated with (or without) a warning.
Therefore, and understanding from your question that you have a background script that is working with Cloud Natural Language, I would strongly advise you to move to a "real" instance of Compute Engine, in which you will have much more control about what is happening. This will allow more flexibility and you will be able to use a bigger machine type, given that Cloud Shell runs on a g1-small GCE instance which, in general, is not enough to run an application. Also, depending on your use case, you may even consider App Engine.
That being said, I have found that when constructing the LanguageClient instance, you may also not use Application Default Credentials and, instead, use the keyFile or keyFilePath variables (explained in the PHP Client Library reference) to pass the path to the JSON key directly to your code, instead of reading it from the environment variable.
Lets assume you are using Linux, make sure that:
The system is not being restarted, and if it is, make sure to set the environment variables accordingly (see how to set permantent environment variables)
Let's say I would have one app with two instances.
I would change an environment variable (cf set-env) and not perform a cf restage.
Eventually one of the two instances would crash and restart. Would it take the new environment variable or the old?
In general if an instance crashes (say the app runs out of memory) and is restarted by Diego (the runtime that's actually running the container instances), the restarted instance will still have the environment variables it was originally "desired" (created) with.
If you explicitly cf restart or cf stop && cf start the app it will pickup the new environment variables which out needing to be restaged.
As user152468 said above, if the environment variables are used during the staging process you will need to cf restage the app for them to functionally take effect.
Edge Case Scenario
If the Diego runtime goes away/loses data for some catastrophic reason, the Cloud Controller will re-sync it and recreate the apps that are meant to be running. In this event the behavior is similar to a cf restart and the app will pick up the new environment variables. This is definitely uncommon, but would also count as a "crash" scenario.
EDIT:
After reading tcdowney's answer below, I tried it out. And tcdowney is right. When the app is restarted due to a crash, it will not pick up the new environment variable, so both of your app instances will share the same environment.
In contrast when you do a cf restart my-app then it will pick it up. Sorry for the confusion!
========================================================
It would take the new value of the environment variable. See here: https://docs.run.pivotal.io/devguide/deploy-apps/start-restart-restage.html
Only if the variable is relevant for the buildpack, you will need to restage.
As you can see on the official doc of Cloud Foundry
https://docs.cloudfoundry.org/devguide/using-tasks.html
A task is an application or script whose code is included as part of a deployed application, but runs independently in its own container.
I'd like to know if there's a way to run command and manipulate files directly on the main container of an application without using a SSH connection or the manifest file.
Thanks
No. Tasks run in their own container, so they cannot affect other running tasks or running application instances. That's the designed behavior.
If you need to make changes to your application, you should look at one of the following:
Use a .profile script. This will let you execute operations prior to your application starting up. It runs for every application instance that is started (I do not believe it runs for tasks) so the operation will be consistently applied.
While not recommended, you can technically background a task through the .profile script that will continue running for the duration of your app. This is not recommended because there is no monitoring of this script and if it crashes it won't cause your app container to restart.
Integrate with your buildpack. Some buildpack's like the PHP buildpack provide hooks for you to integrate and add behavior to staging. For other buildpacks, you can fork the buildpack and make it do whatever you want. This includes changing the command that's returned by the buildpack which tells the platform how to execute your droplet and ultimately what runs in the container.
You can technically modify a running app instance with cf ssh, but I wouldn't recommend it. It's something you should really only do for troubleshooting or maybe testing. Definitely not production apps. If you feel like you need to modify a running app instance for some reason, I'd suggest looking at the reasons why and look for alternative ways of accomplishing your goals.