I have google credentials that are saved as OS environment variables. I would like to work with more than one project credential depending on the view that is called. I have implemented as shown below
#app.route('/project/one', methods=['POST'])
def project_one():
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "path/to/project/one/json/file"
#do stuff authenticated by project one credentials
#app.route('/project/two', methods=['POST'])
def project_two():
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "/path/to/prject/two/json/file"
#do stuff authenticated by project two credentials
When I test this individually, the two endpoints call to the correct google project and I get the expected results.
My worry is in high traffic situation where the two endpoints are called concurrently, am I likely to get errors from the os variable being set in two places and if so which approach should I set for this.
I do not have control on the name of the variable so can't give it different names.
Thanks for your answers in advance
Related
At my project login, some settings.py environment variables are loaded to enable some behaviors:
unit_id = settings.COMPANY
When another user logged in the system changes the value of this variable, through a function, it reflects in all other users who are already active:
settings.COMPANY = "coke"
in this case, all users will see "coke" in settings.COMPANY. I believed this would be in memory and would only apply to the user section in question, because I did not write in the physical file.
I wonder if this is how Django handles the settings.py environment variables: Does it propagate dynamically to all instances opened by all users?
This variable is accessed by context_processors.py, below:
def units(request):
unit_id = settings.COMPANY
You should not change settings at runtime.
This is (mainly) because Django doesn't know anything about it's runtime, so it's definitely possible to run multiple instances of the same Django installation. Changing a setting like this will not propagate it to any other processes.
I wonder if this is how Django handles the settings.py environment
variables: Does it propagate dynamically to all instances opened by
all users?
Django doesn't run an instance for every user. There is one or more (if you for example use something like gunicorn or if you use multiple servers with a load balancer.) processes that listen on a certain port.
To have some changeable setting, you could specify a default value, but you should store something like the active company in the database.
I am using django version 2.1.7:
I have read a lot of article and questions about this. I found people using various methods.
Approches used in the general are:
1) setting environmental variables
2) multiple settings
3) load configuration variables from a file like using django-environ etc
Approach 1: I will not be using, because i dont want to use environmental variables to store variables.
Approach 3: uses 3rd party library and sometimes they may not be maintained any more. But if there is any standard way to do this i am fine.
Approach 2: Here i am not sure how to store the sensitive data seprately.
So can someone guide me in this regard.
I'm using a combination of all:
When launching gunicorn as a service in systemd, you can set your environment variables using the conf file in your .service.d directory:
[Service]
Environment="DJANGO_SETTINGS_MODULE=myapp.settings.production"
Environment="DATABASE_NAME=MYDB"
Environment="DATABASE_USER=MYUSER"
Environment="DATABASE_PASSWORD=MYPASSWORD"
...
That file is fetched from S3 (where it is encrypted) when the instance is launched. I have a startup script using aws-cli that does that.
In order to be able to run management commands, e.g. to migrate your database in production, you also need these variables, so if the environment variables aren't set, I fetch them directly from this file. So in my settings, I have something like this:
def get_env_variable(var_name):
try:
return os.environ[var_name]
except KeyError:
env = fetch_env_from_gunicorn()
return env[var_name]
def fetch_env_from_gunicorn():
gunicorn_conf_directory = '/etc/systemd/system/gunicorn.service.d'
gunicorn_env_filepath = '%s/gunicorn.conf' % (gunicorn_conf_directory,)
if not os.path.isfile(gunicorn_env_filepath):
raise ImproperlyConfigured('No environment settings found in gunicorn configuration')
key_value_reg_ex = re.compile(r'^Environment\s*=\s*\"(.+?)\s*=\s*(.+?)\"$')
environment_vars = {}
with open(gunicorn_env_filepath) as f:
for line in f:
match = key_value_reg_ex.search(line)
if match:
environment_vars[match.group(1)] = match.group(2)
return environment_vars
...
DATABASES = {
...
'PASSWORD': get_env_variable('DATABASE_PASSWORD'),
...}
Note that all management commands I run in my hosted instances, I need to explicitly pass --settings=myapp.settings.production so it knows which settings file to use.
Different settings for different types of environments. That's because things like DEBUG but also mail settings, authentication settings, etc... are too different between the environments to use one single settings file.
I have default.py settings in a settings directory with all common settings, and then production.py for example starts with from .default import * and just overrides what it needs to override like DEBUG = False and different DATABASES (to use the above mechanism), LOGGING etc...
With this, you have the following security risks:
Anyone with ssh access and permissions for the systemd directory can read the secrets. Make sure you have your instances in a VPN and restrict ssh access to specific IP addresses for example. Also I only allow ssh with ssh keys, not username/password, so no risk of stealing passwords.
Anyone with ssh access to the django app directory could run a shell and from django.conf import settings to then read the settings. Same as above, should be restricted anyway.
Anyone with access to your storage bucket (e.g. S3) can read the file. Again, this can be restricted easily.
I want to build a multi tenant architecture for a SAAS system. We are using Django as our backend and mongoengine as our main database and gunicorn as our web-server.
Our clients are a few big companies, so the number of databases pre-allocating space shouldn't be a problem.
The first approach we took was to write a middleware to determine the source of the request to properly connect to a mongoengine database. Here is the code:
class MongoConnectionMiddleware(object):
def process_request(self, request):
if request.user.is_authenticated():
mongo_connect(request.user.profile.establishment)
And the mongo_connect method:
def mongo_connect(establishment):
db_name = 'db_client_%d' % establishment.id
connect(db_name)
This will register the "default" alias as the db_name for every mongoengine request.
But it seems that when many concurrent users from different companies are making requests, each one sets the default db_name to it's own name.
As an example:
Company A makes a request and connects to database A. While A is making it's work company B connects to database B. This makes A also connect to B's database in the process, so A fails to find some ids.
¿Is there a way to isolate the connection to the mongo database per request to avoid this problem?
Unfortunately MongoEngine seems to be designed around a very basic use case of a single primary connection and multiple auxiliary connections.
http://docs.mongoengine.org/en/latest/guide/connecting.html#connecting-to-mongodb
To get around the default connection logic, I define the first connection I come across as the default, I also add it as a named connection. I then add any subsequent connection as named connections only.
https://github.com/MongoEngine/mongoengine/issues/607#issuecomment-38651532
You can use the with_db decorator to switch from one connection to another, but it's a contextmanager call, which means as soon as you leave the with statement, it will revert. It also still requires a default connection.
http://docs.mongoengine.org/en/latest/guide/connecting.html#switch-database-context-manager
You might be able to put it inside a function and then yield inside the with to prevent it reverting immediately, I'm not sure if this is valid.
You could use a wrapper of some kind, either a function, class or a custom QuerySet, that checks the current django/flask session and switches the db to the appropriate connection.
I'm not sure if a QuerySet can do this, but it would probably be the nicest way if it can.
http://docs.mongoengine.org/en/latest/guide/querying.html#custom-querysets
I included some code in this issue here where I change the database connection for my models.
https://github.com/MongoEngine/mongoengine/issues/605
def switch(model, db):
model._meta['db_alias'] = db
# must set _collection to none so it is re-evaluated
model._collection = None
return model
MyDocument = switch(MyDocument, 'db-alias')
You'll also want to take a look at the code that mongoengine uses to switch dbs.
Beware that mongo engine likes to cache things, so changing a few variables here and there doesn't always cause an effect. It's full of surprises like this.
Edit:
I should also add, that the 'connect' call won't pick up value changes. So calling connect with new parameters wont take effect unless its a new alias. Even the disconnect function (which isn't exposed publically) doesn't let you do this as the models will cache the connection. I mention this in some of the issues linked above and also here: https://github.com/MongoEngine/mongoengine/issues/566
In django-webtest, every test TestCase subclass comes with self.app, which is an instance of webtest.TestApp, then I could make it login as user A by self.app.get('/',user='A').
However, if I want to test the behavior if for both user A and user B in a test, how should I do it?
It seems that self.app is just DjangoTestApp() with extra_environ passed in. Is it appropriate to just create another instance of it?
I haven't tried setting up another instance of DjangoTestApp as you suggest, but I have written complex tests where, after making requests as user A I have then switched to making requests as user B with no issue, in each case passing the user or username in when making the request, e.g. self.app.get('/', user'A') as you have already written.
The only part which did not work as expected was when making unauthenticated requests, e.g. self.app.get('/', user=None). This did not work as expected and instead continued to use the user from the request immediately prior to this one.
To reset the app state (which should allow you to emulate most workflows with several users in a sequential manner) you can run self.renew_app() which will refresh your app state, effectively logging the current user out.
To test simultaneous access by more than one user (your question does not specify exactly what you are trying to test) then setting up another instance of DjangoTestApp would seem to be worth exploring.
I'm looking for a very simple and free cloud store for small packets of data.
Basically, I want to write a Greasemonkey script that a user can run on multiple machines with a shared data set. The data is primarily just a single number, eight byte per user should be enough.
It all boils down to the following requirements:
simple to develop for (it's a fun project for a few hours, I don't want to invest twice as much in the sync)
store eight bytes per user (or maybe a bit more, but it's really tiny)
ideally, users don't have to sign up (they just get a random key they can enter on all their machines)
I don't need to sign up (it's all Greasemonkey, so there's no way to hide a secret, like a developer key)
there is no private data in the values, so another user getting access to that information by guessing the random key is no big deal
the information is easily recreated (sharing it in the cloud is just for convenience), so another user taking over the 'key' is easily fixed as well
First ideas:
Store on Google Docs with a form as the frontend. Of course, that's kinda ugly and every user needs to set it up again.
I could set up a Google App Engine instance that allows storing a number to a key and retrieving the number by key. It wouldn't be hard, but it still sounds overkill for what I need.
I could create a Firefox add-on instead of a Greasemonkey script and use Mozilla Weave/Sync—which unfortunately doesn't support storing HTML5 local storage yet, so GM isn't enough. Of course I'd have to implement the same for Opera and Chrome then (assuming there are similar services for them), instead of just reusing the user script.
Anybody got a clever idea or a service I'm not aware of?
Update for those who are curious: I ended up going the GAE route (about half a page of Python code). I only discovered OpenKeyval afterwards (see below). The advantage is that it's pretty easy for users to connect on all their machines (just a Google account login, no other key to transfer from machine A to machine B), the disadvantage is that everybody needs a Google account.
OpenKeyval is pretty much what I was looking for.
OpenKeyval was what I was looking for but has apparently been shut down.
I think GAE will be nice choice. With your requirements for storage size you will never pass free 500 mb of GAE's store. And it will be easy to port your script across browsers because of REST nature of your service;)
I was asked to share my GAE key/value store solution, so here it comes. Note that this code hasn't run for years, so it might be wrong and/or very outdated GAE code:
app.yaml
application: myapp
version: 1
runtime: python
api_version: 1
handlers:
- url: /
script: keyvaluestore.py
keyvaluestore.py
from google.appengine.api import users
from google.appengine.ext import db
from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app
class KeyValue(db.Model):
v = db.StringProperty(required=True)
class KeyValueStore(webapp.RequestHandler):
def _do_auth(self):
user = users.get_current_user()
if user:
return user
else:
self.response.headers['Content-Type'] = 'text/plain'
self.response.out.write('login_needed|'+users.create_login_url(self.request.get('uri')))
def get(self):
user = self._do_auth()
callback = self.request.get('jsonp_callback')
if user:
self.response.headers['Content-Type'] = 'text/plain'
self.response.out.write(self._read_value(user.user_id()))
def post(self):
user = self._do_auth()
if user:
self._store_value(user.user_id(), self.request.body)
def _read_value(self, key):
result = db.get(db.Key.from_path("KeyValue", key))
return result.v if result else 'none'
def _store_value(self, k, v):
kv = KeyValue(key_name = k, v = v)
kv.put()
application = webapp.WSGIApplication([('/', KeyValueStore)],
debug=True)
def main():
run_wsgi_app(application)
if __name__ == "__main__":
main()
The closest thing I've seen is Amazon's Simple Queue Service.
http://aws.amazon.com/sqs/
I've not used it myself so I'm not sure how the developer key aspect works, but they give you 100,000 free queries a month.