Good morning,
I'm trying to properly setup GOOGLE_APPLICATION_CREDENTIALS (https://cloud.google.com/docs/authentication/getting-started) in my droplet. I've currently added the GOOGLE_APPLICATION_CREDENTIALS as an environment variable. Its value is the content of the json key file (containing credentials to verify usage of a Google Service), which is required to start the application.
On startup this environment variable is recognized, but Google needs the value to be an actual json file instead. A simple solution would be to add the key file to my repo and link the environment variable to that file. This is however unsafe.
Is there a way to generate a json file on startup, which contains the value I put in the environment variable? This is a solution which appears to work in Heroku (https://github.com/elishaterada/heroku-google-application-credentials-buildpack/issues/2#issuecomment-580212700), but I haven't found something similar in Digital Ocean.
Of course any solution is welcome, as long as I don't have to commit my json file to a repo.
Thanks!
Related
With Kubernetes, I used to mount a file containing feature-flags as key/value pairs. Our UI would then simply get the file and read the values.
Like this: What's the best way to share/mount one file into a pod?
Now I want to do the same with the manifest file for CloudFoundry. How can I mount a file so that it will be available in /dist folder at deployment time?
To add more information, when we mount a file, the UI later can download the file and read the content. We are using React and any call to the server has to go through Apigee layer.
The typical approach to mounting files into a CloudFoundry application is called Volume Services. This takes a remote file system like NFS or SMB and mounts it into your application container.
I don't think that's what you want here. It would probably be overkill to mount in a single file. You totally could go this route though.
That said, CloudFoundry does not have a built-in concept that's similar to Kubernetes, where you can take your configuration and mount it as a file. With CloudFoundry, you do have a few similar options. They are not exactly the same though so you'll have to make the determination if one will work for your needs.
You can pass config through environment variables (or through user-provided service bindings, but that comes through an environment variable VCAP_SERVICES as well). This won't be a file, but perhaps you can have your UI read that instead (You didn't mention how the UI gets that file, so I can't comment further. If you elaborate on that point like if it's HTTP or reading from disk, I could perhaps expand on this option).
If it absolutely needs to be a file, your application could read the environment variable contents and write it to disk when it starts. If your application isn't able to do that like if you're using Nginx, you could include a .profile script at the root of your application that reads it and generates the file. For example: echo "$CFG_VAR" > /dist/file or whatever you need to do to generate that file.
A couple of more notes when using environment variables. There are limits to how much information can go in them (sorry I don't know the exact value off the top of my head, but I think it's around 128K). It is also not great for binary configuration, in which case, you'd need to base64 encode your data first.
You can pull the config file from a config server and cache it locally. This can be pretty simple. The first thing your app does when it starts is to reach out and download the file, place it on the disk and the file will persist there for the duration of your application's lifetime.
If you don't have a server-side application like if you're running Nginx, you can include a .profile script (can be any executable script) at the root of your application which can use curl or another tool to download and set up that configuration.
You can replace "config server" with an HTTP server, Git repository, Vault server, CredHub, database, or really any place you can durably store your data.
Not recommended, but you can also push your configuration file with the application. This would be as simple as including it in the directory or archive that you push. This has the obvious downside of coupling your configuration to the application bits that you push. Depending on where you work, the policies you have to follow, and the tools you use this may or may not matter.
There might be other variations you could use as well. Loading the file in your application when it starts or through a .profile script is very flexible.
I am deploying a Django application using the following steps:
Push updates to GIT
Log into AWS
Pull updates from GIT
The issue I am having is my settings production.py file. I have it in my .gitignore so it does not get uploaded to GITHUB due to security. This, of course, means it is not available when I PULL updates onto my server.
What is a good approach for making this file available to my app when it is on the server without having to upload it to GITHUB where it is exposed?
It is definitely a good idea not to check secrets into your repository. However, there's nothing wrong with checking in configuration that is not secret if it's an intrinsic part of your application.
In large scale deployments, typically one sets configuration using a tool for that purpose like Puppet, so that all the pieces that need to be aware of a particular application's configuration can be generated from one source. Similarly, secrets are usually handled using a secret store like Vault and injected into the environment when the process starts.
If you're just running a single server, it's probably just fine to adjust your configuration or application to read secrets from the environment (or possibly a separate file) and set those values on the server. You can then include other configuration settings (secrets excluded) as a file in the repository. If, in the future, you need more flexibility, you can pick up other tools in the future.
I have this small project that specifies sqlite as the database choice.
For this particular project, the framework is Django, and the server is hosted by Heroku. In order for the database to work, it must be set up with migration commands and credentials whenever the project is deployed to continuous integration tools or development site.
The question is, that many of these environments do not actually use the my_project.sqlite3 file that comes with the source repository, which we version control with git. How do I incorporate changes to the deployed database? Is a script that set up the database suitable for this scenario? Meanwhile, it is worth notice that there are security credentials that should not appear in a script in unencrypted ways, which makes the situation tricky.
that many of these environments do not actually use the my_project.sqlite3 file that comes with the source repository
If your deployment platform does not support your chosen database, then your development environment should probably be moved to using one of the databases they do support. It is possible to run different databases in development and production, but just seems like the source of headaches.
I have found a number of articles that state that Heroku just doesn't support SQLite in production and instead recommends Postgres.
How do I incorporate changes to the deployed database? Is a script that set up the database suitable for this scenario?
I assume that you are just extracting data from one database to give to another, so yes,as long as that script is a one time batch operation each time the code is updated, then it should be fine. You will want something else if you are adding/manipulating data in production and then exporting it to your git.
Meanwhile, it is worth notice that there are security credentials that should not appear in a script in unencrypted ways
An environment variable should solve that. You set your host machine to have environment variables with your credentials and then just retrieve them within the script. You are looking to have something like this:
# Set environment vars
os.environ['USER'] = 'username'
os.environ['PASSWORD'] = 'password'
# Get environment vars
USER = os.getenv('USER')
PASSWORD = os.environ.get('PASSWORD')
I remember being told to store AWS key into a bash**file (something named like that, can't remember exactly) for security reason, but now I forgot how to access that bash**file.
It should be ~/.bash_profile. Open a terminal window and type.
vi ~/.bash_profile.
To prevent committing sensitive application keys/data to your code, and to provide key access to programs, you should store app keys/sensitive information in environment variables. Environment variables are similar to variables in computer programs, except they exist system-wide in Linux and Windows.
In Linux, you can store those keys in the ~/.bash_profile, so they are available in the environment to command line programs.
nano ~/.bash_profile
in that file, add the following:
export AWS_ACCESS_KEY_ID= *ACCESS_KEY*
export AWS_SECRET_ACCESS_KEY= *SECRET_KEY*
Once saved, you’ll need to source it for the environment variable to work in your current session:
source ~/.bash_profile
In any new session, the environment variables will be loaded automatically.
Please note there are new and more preferred ways to store AWS credentials.
The AWS SDK team has recently made some changes that make it more
convenient, more consistent, and easier to specify credentials for the
SDKs in a more secure way.
Instead of keeping AWS credentials in environment variables, you can now put credentials into a single file that’s in a central location. The default location is this:
~/.aws/credentials (Linux/Mac)
See https://aws.amazon.com/blogs/security/a-new-and-standardized-way-to-manage-credentials-in-the-aws-sdks/
Where should the configuration.yml file of Dropwizard be saved?
I'm using Dropwizard which is a Java web framework.
Dropwizard uses configuration.yml files to load in environment specific configuration files.
In the example I found online the configuration.yml files contains username and password of databases.
Now the question is where to save this configuration files which contain password in plain text.
OPTION 1 GIT REPOSITORY
In the example the configuration.yml are part of the project. So one could keep them in the git repository with the rest of the code. This though is a well-known bad security practice.
If someone crack the git repository has access to the code and to the database. Also this way every single developer has access to all the passwords of all the environments.
OPTION 2 FILE ON THE COMPUTER
Safe the configuration.yml on the machine but do not store on the git repository
OPTION 3 ENVIRONMENT VARIABLES
Use configuration.yml file which point to environment variables on the specific machine.
This is not so practical since all this environment variables needs to be set manually on all the machines. Also what is the syntax to use ENVIRONMENT VARIABLES in Dropwizard's configuration.yml files?
I'd go with environment variables if you cannot control read access to the config file or are concerned that your machine is owned by an untrusted third party.
Environment variables are trivial to script.
You should use a file on the computer: this is how many frameworks out there work.
If you use a unix/linux server you can chmod 0600 [filename] and be sure that nobody (almost nobody as root can do anything) can read that file.
On the dropwizard ML it was also cited to use software like puppet/chef to deploy your application and using these frameworks to handle all variables (eg: different configurations for test/staging/production).
Bye
Piero