I have an application I am currently working on for which I am integrating Travis CI. I am running into the problem of API keys being accessed by Travis. Given below is my current setup (without Travis):
I have a config.py (and is git ignored) that has API keys for all my interfacing applications. I use ConfigParser to read this file and get the required keys.
Travis asks me to look at environment variables as an option to encrypt the keys and add them to .travis.yml. How would Travis know or what needs to be done in order to make travis know that a particular key belongs to a specific interfacing application. Does there need to be changes to the code?
You would read these important variables into your application as system variables.
However, this will only work for builds that are run against master. These environment variables aren't available for builds that are run as part of pull requests.
Related
I'm using Google Cloud Profiler (located at https://console.cloud.google.com/profiler) and would like to know how my profiling data changes across different builds of my application.
One way to do that would be to check the range of dates during which a particular commit was running on production, but that's time consuming because I have to:
Get the start date/time of release, determine the date/time of the next release
Set those dates manually in the profiler interface from the link above
That's really not terrible, but it'd be great to be able to set BUILD_ID environment variable like I can in Cloud Build and then be able to access that from the UI. Is something like this possible? Or is my approach the best way to do this at the moment?
Comparing across service versions would likely be a simpler and more precise way to do this (as opposed to using the time interval to select for profiles). To compare across service versions, it is necessary that the profiling agents set the service version.
The service version can be specified in the configuration passed to the agent (for the Go, Python, or Node.js agent) or via the -cprof_service_version flag (for the Java agent). If one is setting the service version using the configuration passed to the agent (applicable for the Go, Python, and Node.js agents), it may be convenient to use a flag or command line argument to set the service version so that the source code won't need to updated with each new version.
If one is running on Knative or App Engine standard, the service version should be auto-populated. These environments set the K_REVISION and GAE_VERSION environment variables (respectively), and the profiling agents (for all supported languages) use these environment variables to populate the service version. If one is running in another environment and modifying the source code is inconvenient or infeasible, one can set either the K_REVISION or GAE_VERSION environment variable in the environment running the application with the agent enabled to specify the service version.
My understanding is that the BUILD_ID is available at build time, but not at run time, so I don't know that it's possible for agents to use that directly.
(Disclosure: I work on Cloud Profiler at Google)
You can set the service version for this purpose. Please refer to the agent documentation for how to set it for supported languages.
For example, this shows using ServiceVersion for Go services.
I am deploying a Django application using the following steps:
Push updates to GIT
Log into AWS
Pull updates from GIT
The issue I am having is my settings production.py file. I have it in my .gitignore so it does not get uploaded to GITHUB due to security. This, of course, means it is not available when I PULL updates onto my server.
What is a good approach for making this file available to my app when it is on the server without having to upload it to GITHUB where it is exposed?
It is definitely a good idea not to check secrets into your repository. However, there's nothing wrong with checking in configuration that is not secret if it's an intrinsic part of your application.
In large scale deployments, typically one sets configuration using a tool for that purpose like Puppet, so that all the pieces that need to be aware of a particular application's configuration can be generated from one source. Similarly, secrets are usually handled using a secret store like Vault and injected into the environment when the process starts.
If you're just running a single server, it's probably just fine to adjust your configuration or application to read secrets from the environment (or possibly a separate file) and set those values on the server. You can then include other configuration settings (secrets excluded) as a file in the repository. If, in the future, you need more flexibility, you can pick up other tools in the future.
I have this small project that specifies sqlite as the database choice.
For this particular project, the framework is Django, and the server is hosted by Heroku. In order for the database to work, it must be set up with migration commands and credentials whenever the project is deployed to continuous integration tools or development site.
The question is, that many of these environments do not actually use the my_project.sqlite3 file that comes with the source repository, which we version control with git. How do I incorporate changes to the deployed database? Is a script that set up the database suitable for this scenario? Meanwhile, it is worth notice that there are security credentials that should not appear in a script in unencrypted ways, which makes the situation tricky.
that many of these environments do not actually use the my_project.sqlite3 file that comes with the source repository
If your deployment platform does not support your chosen database, then your development environment should probably be moved to using one of the databases they do support. It is possible to run different databases in development and production, but just seems like the source of headaches.
I have found a number of articles that state that Heroku just doesn't support SQLite in production and instead recommends Postgres.
How do I incorporate changes to the deployed database? Is a script that set up the database suitable for this scenario?
I assume that you are just extracting data from one database to give to another, so yes,as long as that script is a one time batch operation each time the code is updated, then it should be fine. You will want something else if you are adding/manipulating data in production and then exporting it to your git.
Meanwhile, it is worth notice that there are security credentials that should not appear in a script in unencrypted ways
An environment variable should solve that. You set your host machine to have environment variables with your credentials and then just retrieve them within the script. You are looking to have something like this:
# Set environment vars
os.environ['USER'] = 'username'
os.environ['PASSWORD'] = 'password'
# Get environment vars
USER = os.getenv('USER')
PASSWORD = os.environ.get('PASSWORD')
Where should the configuration.yml file of Dropwizard be saved?
I'm using Dropwizard which is a Java web framework.
Dropwizard uses configuration.yml files to load in environment specific configuration files.
In the example I found online the configuration.yml files contains username and password of databases.
Now the question is where to save this configuration files which contain password in plain text.
OPTION 1 GIT REPOSITORY
In the example the configuration.yml are part of the project. So one could keep them in the git repository with the rest of the code. This though is a well-known bad security practice.
If someone crack the git repository has access to the code and to the database. Also this way every single developer has access to all the passwords of all the environments.
OPTION 2 FILE ON THE COMPUTER
Safe the configuration.yml on the machine but do not store on the git repository
OPTION 3 ENVIRONMENT VARIABLES
Use configuration.yml file which point to environment variables on the specific machine.
This is not so practical since all this environment variables needs to be set manually on all the machines. Also what is the syntax to use ENVIRONMENT VARIABLES in Dropwizard's configuration.yml files?
I'd go with environment variables if you cannot control read access to the config file or are concerned that your machine is owned by an untrusted third party.
Environment variables are trivial to script.
You should use a file on the computer: this is how many frameworks out there work.
If you use a unix/linux server you can chmod 0600 [filename] and be sure that nobody (almost nobody as root can do anything) can read that file.
On the dropwizard ML it was also cited to use software like puppet/chef to deploy your application and using these frameworks to handle all variables (eg: different configurations for test/staging/production).
Bye
Piero
I am using a config file to store passwords / keys for DB and connection to other services like AWS.
I am using Travis CI for build, and running my tests, and then deploying it to live server.
I would like to encrypt the variables in my config file and only the application should be able to read it. This is the criteria -
1) The application should be able to decrypt it in multiple environments, from the build server to multiple deployment servers.
2) The password used to decrypt the config file is not avalaible to the developers.
The project is in clojure, are there any leiningen plugins / features that will aid in this ?
Thanks,
Murtaza
I think the best solution to what you are trying to do is simply to remove the config files from the application source tree. Save the config files to the build and deployment severs and don't give the developers direct access to those.
The chief problem with what you are trying to do is that you are trying to make it impossible for your developers to have access to something but at the same make it possible for something that the developers have complete control over, the application code, to have access to the same thing. That is, for the application to be able to decrypt the configs, the developers have to have access to the decyption keys.
You could try making it only possible to decrypt your configs through tools available only on the build and deployment servers, (ie a Leiningen plugin,) but it would be simpler to just store the config files separately.