I would like to define some environment variables for an space to manage multiple test setup within a single foundation.
Is it possible to achieve this and how can we do it?
The srevg and ssevg are to be run via admins and are foundation wide variables. I only need it for my space.
At the time of writing, there is no built-in concept of a org or space scope environment variable. You can apply them system wide, which requires admin access to the environment, or you can apply them for a specific app.
A couple things you can do:
You can put them in your application manifest file. Putting them in manifest.yml makes it less likely that you'd forget the value. You can also use manifest variables if there are values you don't want to commit in your manifest.yml. Don't forget you can define multiple application's in one manifest.yml file, which means you can use YAML anchors to reduce duplication & share the same env block across multiple apps in the same manifest.yml file.
You can use the do-all cf cli plugin to run cf set-env across all apps in the same space (you can use it to run any command across the whole space). If you always use cf do-all set-env when you make changes, you should be able to keep the env variables in sync.
You can use a user-provided service for the information you want to share instead. User provided services are shared by default across the space, so you can just bind the user-provided service to all apps in the space and they will all have access to the information (bonus: it's more flexible and you can bind to a subset of apps as well)
Related
Why do people put a .env file to store all their secrets in a server? If someone hacks it, isn't the .env equally accessible as all the other files? Thanks!
You are correct that storing environmental secrets in a .env file poses a risk of plain text secrets being exposed to a third party if they gained access to raw code.
Just like other areas with sensitive material there are ways to get around this, generally an approach that people might take it to use a secrets management system which instead replaces any secrets values from a .env file to be accessed via a validated request.
AWS supports a couple of official services that can do this:
Secrets Manager - This service is specifically built for this purpose, you define a secret and give it either a string or JSON value that is then retrieved via a call using the SDK. All values are encrypted using a KMS key.
Systems Manager Parameter Store - Similar to secrets manager, you provide a key name and give it a value. It supports both unencrypted and encrypted values (use SecureString type).
In addition there are other services such as Hashicorp Vault that provide similar functionality.
For environmental configuration a .env file can still be appropriate i.e. enable this feature flag but if you want to try and reduce the blast radius of your application then storing secrets outside a plain text file will help to reduce this risk.
That is not the main reason for using environment variables. However, it is secure enough for saving secret values too especially when they’re combined with hashing methods.
Environment variables are most useful in the actual production level of programming. Your application must have different environments to run upon. Development: that your host is local and as a developer you need to test your code and set the debug variable to true to get stateful errors which is not something you want on the production environment. Production: that your host is your domain or server IP and you need different middleware than of the development stage. There are also staging and test environments for bigger projects. There could be a lot of things that should be handled differently on different environments: database is a great example. Besides, environment variables are useful for when there is more than one person working with the code base and people can configure the project based on their machine/OS using environment variables.
I have a WordPress site that is gonna be hosted using ECS in AWS.
To make the management even more flexible, I plan not to store service configurations (i.e. php.ini, nginx.conf) inside the docker image itself. I found that docker swarm offers "docker configs" for such. Are there any equivalent tools doing the same thing? (I know AWS Secrets Manager can handle docker secrets though)
Any advice or alternative approaches? thank you all.
The most similar you could use is probably AWS SSM Parameter store
You will need some logic to retrieve the values when you are running the image.
If you don't want to have the files also inside of the running containers, then you pull from Parameter Store, and add them to the environment, and you will need to do probably some work in the application to read from the environment (the application stays decoupled from the actually source of the config), or you can read directly from Param store in the application (easier, but you have some coupling in your image with Parameter store.
if your concern is only about not having the values in the image, but it is fine if they are inside of the running container, then you can read from Param Store and inject the values in the container inside of the usual location of the files, so for the application is transparent
As additional approaches:
Especially for php.ini and nginx.conf I like a simple approach that is having a separate git repo, with different config files per different environments.
You have a common docker image regardless of the environment
in build time, you pull the proper file for the enviroment, and either save as env variables, or inject in the container
And last: need to mention classic tools like Chef or Puppet, and also ansible. More complex and maybe overkill
The two ways that I store configs and secrets for most services are
Credstash which is combination of KMS and Dynamodb, and
Parameter Store which has already been mentioned,
The aws command line tool can be used to fetch from Parameter Store
and S3(for configs), while credstash is its own utility (quite useful and easy to
use) and needs to be installed separately.
In Google shell which is a part of Google cloud, I set environment variable GOOGLE_APPLICATION_CREDENTIALS because It is need it for PHP NLP project [info: https://cloud.google.com/natural-language/docs/quickstart-client-libraries#client-libraries-install-php]. My project worked fine, but I notice that variable GOOGLE_APPLICATION_CREDENTIALS lasts on my sistem only one day. This is my third time that I am setting it. My project doesn't work when I am missing required variable. Am I doing something wrong?
EDIT:
It is default OS (Debian) when you create new App on Google App engine.
When I type help in Google shell I get info with:
Your 5GB home directory will persist across sessions, but the VM is ephemeral and will be reset
approximately 20 minutes after your session ends. No system-wide change will persist beyond that.
You are completely right, Cloud Shell is running on an ephemeral instance that resets some minutes after the session has ended, reason why you are losing the content of the environment variable you mentioned.
The documentation about limitations in Cloud Shell clearly states that it is intended for interactive use only, and any non-interactive session or intensive usage can be automatically terminated with (or without) a warning.
Therefore, and understanding from your question that you have a background script that is working with Cloud Natural Language, I would strongly advise you to move to a "real" instance of Compute Engine, in which you will have much more control about what is happening. This will allow more flexibility and you will be able to use a bigger machine type, given that Cloud Shell runs on a g1-small GCE instance which, in general, is not enough to run an application. Also, depending on your use case, you may even consider App Engine.
That being said, I have found that when constructing the LanguageClient instance, you may also not use Application Default Credentials and, instead, use the keyFile or keyFilePath variables (explained in the PHP Client Library reference) to pass the path to the JSON key directly to your code, instead of reading it from the environment variable.
Lets assume you are using Linux, make sure that:
The system is not being restarted, and if it is, make sure to set the environment variables accordingly (see how to set permantent environment variables)
Is there a way to set a manifest.yml property for a specific space, e.g. if I want to have 5 instances of my service when deploying to the production space but having only 1 for all other spaces?
A few options come to mind:
You can override manifest.yml settings with cf cli arguments. Thus you could put an instance count of one into your manifest.yml file (actually, you don't need to do this because one is the default value but you can put any value in the file) and when you cf push that overrides the value from manifest.yml with the value you set as the cli argument. Ex: cf push -i <override>.
There's no conditional logic available in an app's manifest.yml, but there's nothing stopping you from using a template language with your manifest. You could, for example, run a templated manifest.yml through Ruby's erb (or any other template engine) to make some dynamic adjustments and then use the output to deploy your app.
Don't use manifest.yml at all. Instead just use a shell script and a composition of cf cli commands and arguments. It's a little more work to get going, but you get all the dynamic behavior of a shell script. Actually, it doesn't have to be a shell script. You could use Python or Ruby or insert your favorite scripting language here.
Probably not exactly the answer you were hoping for here, but hope that it helps.
Depending on the deployment tool, you can parameterise the instances. For ex: in IBM Urban Code Deploy (UCD), you can declare number of instances as a component environment property and you can given different values for each space. DEV, QA can have 1 and PROD can have 5.
There are multiple ways to use this property.
Add the below entry in manifest.yml
instances: ${instance_count}
Value of instance_count parameter can be configured per Space in Urban Code Deploy. I'm sure you can do this with Concourse or other deployment tools also.
Use the parameter in the cf push command.
Urban Code Deploy and other tools run the cf push command to push the application. In the push command, you can give -i parameter and use the instance_count variable which will substitute the value depending on the space to which you are pushing.
You can use the first method to parameterise several fields such as memory, log level etc which can have different values for different spaces.
I am trying to build out our AWS environments using Terraform but am hitting some issues scaling. I have a repository of just modules that I want to use repeatedly when building my environments and a second repository just to handle the actual implementations of those modules.
I am aware of HashiCorp's Github page that has an example but there, each environment is one state file. I want to split environments out but then have multiple state files within each environment. When the state files get big, applying small updates takes way too long.
Every example I've seen where multiple state files are used, the Terraform files are extremely un-DRY and not ideal.
I would prefer to be able to have different variable values between environments but have the same configuration.
Has anyone ever done anything like this? Am I missing something? I'm a bit frustrated because every Terraform example is never at scale and it makes it hard for n00b such as myself to start down the right path. Any help or suggestions is very much appreciated!
The idea of environment unfortunately tends to mean different things to different people and organizations.
To some, it's simply creating multiple copies of some infrastructure -- possibly only temporary, or possibly long-lived -- to allow for testing and experimentation in one without affecting another (probably production) environment.
For others, it's a first-class construct in a deployment architecture, with the environment serving as a container into which other applications and infrastructure are deployed. In this case, there are often multiple separate Terraform configurations that each have a set of resources in each environment, sharing data to create a larger system from smaller parts.
Terraform has a feature called State Environments that serves the first of these use-cases by allowing multiple named states to exist concurrently for a given configuration, and allowing the user to switch between them using the terraform env commands to focus change operations on a particular state.
The State Environments feature alone is not sufficient for the second use-case, since it only deals with multiple states in a single configuration. However, it can be used in conjunction with other Terraform features, making use of the ${terraform.env} interpolation value to deal with differences, to allow multiple state environments within a single configuration to interact with a corresponding set of state environments within another configuration.
One "at scale" approach (relatively-speaking) is described in my series of articles Terraform Environment+Application Pattern, which describes a generalization of a successful deployment architecture with many separate applications deployed together to form an environment.
In that pattern, the environments themselves (which serve as the "container" for applications, as described above) are each created with a separate Terraform configuration, allowing each to differ in the details of how it is configured, but they each expose data in a standard way to allow multiple applications -- each using the State Environments feature -- to be deployed once for each environment using the same configuration.
This compromise leads to some duplication between the environment configurations -- which can be mitgated by using Terraform modules to share patterns between them -- but these then serve as a foundation to allow other configurations to be generalized and deployed multiple times without such duplication.