Where should the configuration.yml file of Dropwizard be saved?
I'm using Dropwizard which is a Java web framework.
Dropwizard uses configuration.yml files to load in environment specific configuration files.
In the example I found online the configuration.yml files contains username and password of databases.
Now the question is where to save this configuration files which contain password in plain text.
OPTION 1 GIT REPOSITORY
In the example the configuration.yml are part of the project. So one could keep them in the git repository with the rest of the code. This though is a well-known bad security practice.
If someone crack the git repository has access to the code and to the database. Also this way every single developer has access to all the passwords of all the environments.
OPTION 2 FILE ON THE COMPUTER
Safe the configuration.yml on the machine but do not store on the git repository
OPTION 3 ENVIRONMENT VARIABLES
Use configuration.yml file which point to environment variables on the specific machine.
This is not so practical since all this environment variables needs to be set manually on all the machines. Also what is the syntax to use ENVIRONMENT VARIABLES in Dropwizard's configuration.yml files?
I'd go with environment variables if you cannot control read access to the config file or are concerned that your machine is owned by an untrusted third party.
Environment variables are trivial to script.
You should use a file on the computer: this is how many frameworks out there work.
If you use a unix/linux server you can chmod 0600 [filename] and be sure that nobody (almost nobody as root can do anything) can read that file.
On the dropwizard ML it was also cited to use software like puppet/chef to deploy your application and using these frameworks to handle all variables (eg: different configurations for test/staging/production).
Bye
Piero
Related
How can I properly manage my development files and production files with source control like GitHub (private repo). I am new to Django and while publishing my code to production there are many settings which needs to be changed in order to use it in production but the same setting won't work on my development server. I am stuck here Please guide me how manage. I want my files to be source control like any time I can make my development environment on another pc and anytime I can make my production server on any server.
how to manage secret keys, passwords on source control (like GitHub private repo)
Also please specify how should an Ideal file structure look like
hope you are well.
To answer the following questions I will do so based in the format you have asked.
You ask about management of production and development files with source control (namely GitHub). It would be best to store these in different branches of source control. Example: "main" branch being used for production and a "development" branch being used for development. This will allow you to work with both branches and you can merge development branch into the production branch.
The best way you can manage sensitive information such as passwords and keys in source control is to avail of .env files (What is the use of python-dotenv?), which stores variables in an environment. You can store variables in this file and tell GitHub to ignore this file in the .gitignore file.
You mention ideal file structure. There are many ways which files can be structured and normally I would say this is preference on the developers behalf as this doesn't really matter as long as the developer and future developers can make sense of the file structure. From personal recommendation and this is my own opinion.
Project setup
staticfiles folder
projectName
settings.py etc
app
app files
manage.py
Hope this can be of some benefit to you and some clarity around your questions.
Have a good day.
I am deploying a Django application using the following steps:
Push updates to GIT
Log into AWS
Pull updates from GIT
The issue I am having is my settings production.py file. I have it in my .gitignore so it does not get uploaded to GITHUB due to security. This, of course, means it is not available when I PULL updates onto my server.
What is a good approach for making this file available to my app when it is on the server without having to upload it to GITHUB where it is exposed?
It is definitely a good idea not to check secrets into your repository. However, there's nothing wrong with checking in configuration that is not secret if it's an intrinsic part of your application.
In large scale deployments, typically one sets configuration using a tool for that purpose like Puppet, so that all the pieces that need to be aware of a particular application's configuration can be generated from one source. Similarly, secrets are usually handled using a secret store like Vault and injected into the environment when the process starts.
If you're just running a single server, it's probably just fine to adjust your configuration or application to read secrets from the environment (or possibly a separate file) and set those values on the server. You can then include other configuration settings (secrets excluded) as a file in the repository. If, in the future, you need more flexibility, you can pick up other tools in the future.
I'm learning about Strongloop, it's pretty good so far.
Question: What is the appropriate place to put AWS keys? config.json? ..and how would I access them from my application?
Thanks
Ideally you would not put those credentials in any file that is committed. I usually find environment variables to be the best balance of convenience and security.
If you are using strong-pm, then you would do this with slc ctl env-set. If you are using some other supervisor, then you'll need to consult its docs.
A lot of times it is enough to use Upstart or systemd directly, which both make it fairly easy to set environment variables in the service process.
Other than above answer, what you can do is put these in your release procedure.
What we have done in our product is all these entries are kept in a config file which is deployed from the shared folder.
Let me elaborate it.
we have local config files in the git. and separate config files on production servers in a folder names as shared, now, when ever a tag release is deployed from git, the shared folder overwrite these config files.
As per best practices, my development team does not store the application config file in a repo for security reasons (we use a config/application.yml file to store configs). However, when we actually develop and deploy, this causes some problems:
A developer needs to add a new external URL that is different depending on what environment the application is running in. Since there is no config file in the repo, he cannot update a single file that gets synced when another developer pulls the code. To make this happen, he updates his local config/application.yml file and then each other developer updates their local file, and then we have to add the new ENV variable to the server's config/application.yml. Has to be a better solution.
If we stored the config/application.yml file in the repo and shared it among everyone and the servers, this solves the problem of sharing/updating global configs, BUT it opens up the possibility that a developer may accidentally start their local application in production mode and touch live data or spam real users with test emails (has happened which is why it's a concern).
Is there a standard best practice for solving these types of problems? It seems I either sacrifice productivity for security but can't really have both.
I've been thinking about creating a config/development.yml file in the repo that all developers share, which stores all environments EXCEPT production. That way they can share config/ENV items for development and sync them up. But in production, I would have a config/production.yml file that ONLY lives on the servers.
If the application is started in anything except production environment, it loads the development.yml file. If it is started in production, it loads the production.yml file. But since the production.yml file does NOT live in the repo (only on the servers), there's no chance that a developer can accidentally touch live data or spam real users, etc...
Have any professional developers tried a scheme like this? I've done a lot of googling but really haven't found a satisfactory solution.
Check out the RailsConfig gem. This allows you do to exactly what you stated, but with the ease of a gem. This also allows you and your dev team to have local yaml files that override settings.
config/settings.yml
config/settings/#{environment}.yml
config/environments/#{environment}.yml
config/settings.local.yml
config/settings/#{environment}.local.yml
config/environments/#{environment}.local.yml
You would then just have config/settings/production.yml within your .gitignore so that it will not be checked into source control.
I am using a config file to store passwords / keys for DB and connection to other services like AWS.
I am using Travis CI for build, and running my tests, and then deploying it to live server.
I would like to encrypt the variables in my config file and only the application should be able to read it. This is the criteria -
1) The application should be able to decrypt it in multiple environments, from the build server to multiple deployment servers.
2) The password used to decrypt the config file is not avalaible to the developers.
The project is in clojure, are there any leiningen plugins / features that will aid in this ?
Thanks,
Murtaza
I think the best solution to what you are trying to do is simply to remove the config files from the application source tree. Save the config files to the build and deployment severs and don't give the developers direct access to those.
The chief problem with what you are trying to do is that you are trying to make it impossible for your developers to have access to something but at the same make it possible for something that the developers have complete control over, the application code, to have access to the same thing. That is, for the application to be able to decrypt the configs, the developers have to have access to the decyption keys.
You could try making it only possible to decrypt your configs through tools available only on the build and deployment servers, (ie a Leiningen plugin,) but it would be simpler to just store the config files separately.