I am trying to set Environment Variables for a Nginx And Gunicorn Served Django Project on Ubuntu.
The variables are set for the Ubuntu user and i am able to see the value using printenv VAR_EMAIL.
But when i use them django settings it is not working , using them as os.environ['VAR_EMAIL'],This doesn't get the value of the variable in production and server doesn't work.
However this works on Development side.
UPDATE 1st May 2020:
I used systemd and passed the variable like this in gunicorn.service file.This won't work still get key error,)Will post the exact error) as it is production on Ubuntu but i am developing on Windows and it works fine with Environment Variables in Development.
is os.environ['var_name'] correct way to access that ?
I also tried os.environ.get('var_name') as i saw in some video that environ have .get() to access the value. I will try again maybe i made some mistake.Feel free to ask for any info required.
ANSWERED - It was error on my end.
What are you using to supervise and run the gunicorn process in Ubuntu? If you're not using any, I recommend you to use systemd, there's a small guide on how to setup in the gunicorn docs: https://docs.gunicorn.org/en/stable/deploy.html#systemd
After that, you can set the environment variables in the systemd config file doing like the following, under the [Service] section of the systemd config file:
[Service]
Environment="VAR_EMAIL=var-email"
Environment="ANOTHER_VAR=another-var"
You can also use the EnvironmentFile directive if you prefer to have these variables in a separate file: https://www.freedesktop.org/software/systemd/man/systemd.exec.html#EnvironmentFile=
Now after many problems i faced because of which i just hard coded the
variables in code for long time the solution i reached is in this
update.Read the complete answer to get it.
This is an update,it is odd how the environment variables are passed along.
Every variable require the following method but you also need to add the variables to the etc/environment and add the Environment variables in it too.
The environment Variables were to be set in a single line with spaces separating each key+value pair. (Not like mentioned in selected Answer)
Couldn't get it to work with supervisor though.
So the correct way to do it is as mentioned in this Documentation for Systemd : EnvironmentsSystemd
Environment = "KEY_NAME_1=VALUE_KEY_1" "KEY_NAME_2=VALUE_KEY_2" ....
Then User PassEnvironment to activate those in that session.
This way with PassEnvironment you can have different Environment Variables active for Different Services.
PassEnvironment = KEY_NAME_1 KEY_NAME_2 ....
This is Key Names seperated by spaces.
This should help you setup systemd . Check the Selected answer for all the links.
This won't work:
Related
After quite a bit of trial and error and a step by step attempt to find solutions I thought I share the problems here and answer them myself according to what I've found. There is not too much documentation on this anywhere except small bits and pieces and this will hopefully help others in the future.
Please note that this is specific to Django, Celery, Redis and the Digital Ocean App Platform.
This is mostly about the below errors and further resulting implications:
OSError: [Errno 38] Function not implemented
and
Cannot connect to redis://......
The first error happens when you try run the celery command celery -A your_app worker --beat -l info
or similar on the App Platform. It appears that this is currently not supported on digital ocean. The second error occurs when you make a number of potential mistakes.
PART 1:
While Digital Ocean might remedy this in the future here is an approach that will offer a workaround. The problem is the not supported execution pool. Google "celery execution pools" if you want to know more and how they work. The default one is prefork. But what you need is either gevent or eventlet. I went with the former for my purposes.
Whichever you pick you will have to install it as it doesn't come with celery by default. In my case it was: pip install gevent (and don't forget adding it to your requirements as well).
Once you have that you can re-run the celery command but note that gevent and beat are not supported within a single command (will result in an error). Instead do the following:
celery -A your_app worker --pool=gevent -l info
and then separately (if you want to run beat that is) in another terminal/console
celery -A your_app beat -l info
In the first line you can also specify the concurrency like so: --concurrency=100. This is not required but useful. Read up on it what it does as that goes beyond the solution here.
PART 2:
In my specific case I've tested the above locally (development) first to make sure they work. The next issue was getting this into production. I use Redis as the db/broker.
In my specific setup I have most of my celery configuration in the_main_app/celery/__init__.py file but sometimes people put it directly into the_main_app/celery.py. Whichever it is you do make sure that the REDIS_URL is set correctly. For development it usually looks something like this:
YOUR_VAR_NAME = os.environ.get('REDIS_URL', 'redis://localhost:6379') where YOUR_VAR_NAME is then set to the broker with everything as below:
YOUR_VAR_NAME = os.environ.get('REDIS_URL', 'redis://localhost:6379')
app = Celery('the_main_app')
app.conf.broker_url = YOUR_VAR_NAME
The remaining settings are all documented on the "celery first steps with django" help page but are not relevant for what I am showing here.
PART 3:
When you setup your Redis Database on the App Platform (which is very simple) you will see the connection details as 'public network' and 'VPC network'.
The celery documentation says to use the following URL format for production: redis://:password#hostname:port/db_number. This didn't work. If you are not using a yaml file then you can simply copy paste the entire connection string (select from the dropdown!) from the Redis DB connection details and then setup an App-Level environment variable in your Digital Ocean project named REDIS_URL and paste in that entire string (and also encrypt it!).
The string should look like something like this (redis with 2 s!)
rediss://USER:PASS#URL.db.ondigitialocean.com:PORT.
You are almost done. The last step is to setup the workers. It was fine for me to run the PART 1 commands as console commands on the App Platform to test them but eventually I've setup a small worker (+ Add Component) for each line pasted them into the Run Command.
That is basically the process step by step. Good luck!
When working with a ColdFusion server you can access the CFIDE/administrator to set config values, which update the cfusion/lib/ xml files (e.g. neo-runtime.xml, neo-mail.xml, etc.)
I'd like to automate a deployment process that includes setting these administrator values so that I don't have to log in and manually set them for each new box that shares settings. I'm unsure of the best way to go about it.
Some thoughts I had are:
Replacing the full files with ones containing my custom settings. I've done this for local development, but it may not be an ideal method due to CF hot-fixes potentially adding/removing/changing attributes.
A script to read the wddx xml file and replace the attribute values. I'm having trouble finding information about how to do this method.
Has anyone done anything like this before? Or does anyone have any recommendations on how to best go about this?
At one company, we checked all the neo-*.xml files into source control, with a set for each environment Devs only had access to the dev settings and we could deploy a local development environment with all the correct settings for new employees quickly.
but it may not be an ideal method due to CF hot-fixes potentially adding/removing/changing attributes.
You have to keep up with those changes and migrate each environment appropriately.
While I was there, we upgraded from 8 to 9, 9 to 11 and from 11 to 2016. Environments would have to be mixed as it took time to verify the applications worked with each new version of CF. Each server got their correct XML files for that environment and scripts would copy updates as needed. We had something like 55 servers in production running 8 instances each, so this scaled well.
There is a very usefull tool developed by Ortus Solutions for this kind of automatizations called cfconfig that can be installed with their commandbox command line utility. This tool isn't only capable of setting configurations of the administrator: It is also capable of exporting/importing settings to a json file (cfconfig.json). It might be what you need.
Here is the link to their docs
https://cfconfig.ortusbooks.com/introduction/getting-started-guide
CFConfig worked perfectly for my needs. I marked #AndreasRu answer as accepted for introducing me to that tool! I'm just adding this response with some additional detail for posterity.
Install CommandBox as part of deployment script
Install CFConfig as part of deployment script
Use CFConfig to export a config.json file from an existing box that will share settings with the new deployment. Store this json file in source control for each type/env of box.
Use CFConfig to import the config.json as part of deployment script
Here's a simple example of what this looks like on debian
# Installs CommandBox
curl -fsSl https://downloads.ortussolutions.com/debs/gpg | apt-key add -
echo "deb https://downloads.ortussolutions.com/debs/noarch /" | tee -a /etc/apt/sources.list.d/commandbox.list
apt-get update && apt-get install apt-transport-https commandbox
# Installs CFConfig module
box install commandbox-cfconfig
# Import config settings
box cfconfig import from=/<path-to-config>/config.json to=/opt/ColdFusion/cfusion/ toFormat=adobe#11.0.19
I have deployed two rails apps to Digital Ocean, Ubuntu 18.04 with Passenger and Nginx.
Both apps were built on rails 5.2.2 with ruby 2.5.1, and the second app has all the same gems at the same versions. While the first app runs fine, the second will not launch.
The last useful line of the Passenger log says:
[ E 2020-08-06 22:41:56.6186 30885/T1i age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /var/www/html/AppName_Prod/current: The application encountered the following error: ActiveSupport::MessageEncryptor::InvalidMessage (ActiveSupport::MessageEncryptor::InvalidMessage)
I know this is somethign to do with the master.key file, but that is present and contains the correct key. I'm not using environment vars to store the master keys - they are in the master.key file inside each app's dir structure.
I've read every SO post I could find on this and none have solved my issue.
Any suggestions for getting these two apps (and more) to work on the same droplet?
I'm all out of ideas.
Thank you for any help you can offer.
For anyone who might have the same issue, it was a bit deceptive.
I had tried rails credentials:edit and it didn't fix the issue, but I found that the app's containing folder was owned by user:user, whereas my other app was owned by user:root.
When I changed this, everything started to work.
I hope it helps someone because I didn't find this info anywhere online and it was a lot of trial and error.
Use ls -l to list the current owner of folders in the current working directory, so you can compare them.
For me, this turned out to be somewhat complicated. I had provisioned my server using Ansible, which has a task to copy the Nginx conf. After provisioning the server, I changed RAILS_MASTER_KEY.
It turns out that my Ansible task does not re-write the Nginx conf if it already exists on the server (the file is not compared, I guess). So although I updated RAILS_MASTER_KEY in my Ansible playbook (and it was even getting copied across to the server's environment variables!), it was not accessible to Rails through passenger because it does not pass on the user's environment variables.
Whew!
To fix this (and create a snowflake server in the process...) I manually logged into the server and updated RAILS_MASTER_KEY to my new value in the Nginx passenger_env_var.
I am currently writing my fabric deploy script for my gunicorn/django app which is managed by supervisord. I have certain passwords that are required to be in the environment of the app when gunicorn runs.
It seems the standard way of doing things is to add environment variables to the [program:app] of /etc/supervisord/app.conf, but this means I will have plaintext passwords in my repo unless I kept the app.conf outside of the repo (which doesn't seem clean).
I can't hash the passwords because they are used by the app to access third party services. Also even if I kept app.conf outside of the repo, I still have plaintext passwords in app.conf which doesn't seem ideal.
Is there some way to let supervisord do its thing (restarting gunicorn on its own) without storing plaintext passwords?
Edit: Inevitably I need either the passwords or a way for supervisord to unlock the passwords, both of which can be used by attackers, so I think I will use something like AWS key management service so at least there is an extra step of getting to them.
I haven't used supervisord but I'm assuming you're just running a bash script to restart your server.
Create a file in your project folder called .env (the name doesn't really matter but it's convention). Make sure to add this file to your .gitignore.
In the env file, set any config vars:
#!/bin/bash
export ENV_VAR="value"
Then to use the env file, use source .env. This will apply any exported vars to your environment.
Here's what I'm working with right now:
Ubuntu Trusty 14.04
Rails 4.2.6
Ruby 2.2.3
Passenger
Nginx
When I try to visit the IP I get this message:
Incomplete response received from application
When I look at nginx/error.log I see:
Missing `secret_token` and `secret_key_base` for 'production' environment, set these values in `config/secrets.yml`
On the server I did:
RAILS_ENV=production bundle exec rake secret
I placed that result into each of these files for good measure:
~/.bashrc
~/.bash_profile
~/.profile
/app/shared/config/local_env.yml
For all shell scripts the format is:
export SECRET_KEY_BASE="[key]"
For the local_env.yml I used just:
SECRET_KEY_BASE="[key]"
I've also tried entering it without quotation marks.
I've restarted the server each time I made a change. No cigar.
What else might be the issue?
-- UPDATE
I've even added the secret key to the secrets.yml file directly. So now I'm thinking my issue is either something to do with passenger/nginx or with a typo somewhere.
It is more likely that the environment variables are not actually set rather than Rails is not picking them up. You're raking secrets, which I don't do. I set them up manually in the Unix etc/environment, and do not check any secrets into source control. But the following are a few steps that should help you either resolve or hone in on the problem.
On your Ubuntu server for system wide environment variables
1- $env
Look for your SECRET_TOKEN and SECRET_KEY_BASE. The error tells you that these are not set, this is just a technique to check env. (RAILS_ENV will also be shown in the list if it is set.)
2- $sudo nano /etc/environment
Add the following lines -- use your actual values between double quotes. Do not use a [key] or any programmatic replacement.
export SECRET_TOKEN="T99ABC..."
export SECRET_KEY_BASE="99ABC..."
3- $logout / $login to reload environment vars
4- $env - Check the environment again
Look for your SECRET_TOKEN and SECRET_KEY_BASE to be set.
5- Try deploying again. If it fails, check the environment vars using $env again. It will tell you if something in your deploy is smashing your SECRET_* env vars.