Hey this is a more general question.
The first part is just knowing exactly how Global variables work. If multiple users are accessing the server at the same time is a global variable going to be shared across all of the users? or will each user have their own instance of that global variable?
I am aware of sessions and how this is probably the best answer to solve my issue, however, I am currently working with the Django FormWizard and it doesnt seem to have access to request so I am unable to use the sessions. I am not entirely sure how to access request so if anyone knows how to do that I appreciate the help.
Thanks!
Depends on how you deploy your application. Gunicorn for example by default will load every worker process with its own environment, but with --preload it will load app and only after that prefork workers. The second way global variables will be shared, but with a limitation: all shared variables will copy-on-write, so if you try to modify global variable in a worker process, that variable will be copied and you'll modify copy of instance.
Answering your second question. You cannot get request instance from anywhere you want in Django if it's not directly passed into function. Modifying global variables often isn't threadsafe, be aware.
Related
This question already has answers here:
Are global variables thread-safe in Flask? How do I share data between requests?
(4 answers)
Closed 1 year ago.
I used Flask's g variable to store variables. I hosted the app in IIS, the values are not working correct and some time previously edited values were shown.
Then I moved to sessions to hold my data. In my case I should not use database for storing my data.
For each session I have more than 10 variables. Whether it is good to hold the data in session? or storing variables globally across request is good in production environment, if yes, anyone please explain?
Also I would like mention that global declaration of variables not working correct in production environment for me.
I am using flask for a long time and the session approach works great for me. Still, there are some security concerns because you are storing information in your session cookie which is easy to decrypt. If the data are not sensitive like passwords I strongly suggest using session.
From a development perspective, defining variables and connections inside the UI is effective but not robust, as it is impossible to keep track of what has been added and removed.
Airflow came up with a way to store variables as environment variables. But a few natural questions arise from this:
Does this need to be defined before every DAG? What if I have multiples DAGs sharing the same env values? Seems a bit redundant to be defining it every time.
If defined this way, do they still display on the UI? The UI is still a great idea for taking quick look at some of the key value pairs.
I guess in a perfect world, the solution I would be looking for is somehow, just define the value of the variables and connections in the airflow.cfg file which would automatically populate the variables and connections in the UI.
Any kind of help is appreciated. Thank you in advance!
There is one more way of storing and managing and connections, one that is most versatile, secure and gives you all the versioning and auditing support - namely Secret Backends.
https://airflow.apache.org/docs/apache-airflow/stable/security/secrets/secrets-backend/index.html
It has built-in integration with Vault, GCP Secret Store, AWS Secret store, you can use Local Filesystem Secret Backend, and you can also roll your own backend.
When you use one of those then you get all the versioning, management, security, access management coming from the Secret Backend you use (most of the secret backends have all those built-in).
This also means that you CANNOT see/edit the values via Airflow UI as it's all delegated to those backends. But the backends usually come with their own UIs for that.
Answering your questions:
If you define connections/variables via env vars, you should define the variables in your Workers and Scheduler, not in the DAGs. That means that (if your system is distributed) you need to have a mechanism to update those variables and restart all airflow processes when they change (for example via deploying new images with those variables or upgrading helm chart or similar)
No. The UI only displays variables/connections defined in the DB.
I understand that Heroku preserves it's filesystem in a running dyno ephemerally, wiping changes once a day when the dyno cycles.
I use a simple .txt file to store some frequently used and updated authentication keys.
Aside: A text file is an awful way to do this, would love to be told a bit about security best practices for storing keys, but, I understand how massive that lesson would be and the amount of resources already present for me to research myself. So, don't worry about it.
The problem is, I constantly refresh these keys, and rewrite the text file. So when the dyno cycles and resets the keys to their initial state, they are no longer valid. What alternative methods do I have to make sure that the keys are synced across dynos and are always up to date?
I am using a Django app, so I have access to creating a Model, which would probably be my first guess at how to go about this.
Thank you in advance.
Use config vars - these get set into the environment on each dyno. Updating the key restarts the app. https://devcenter.heroku.com/articles/config-vars
Hi I'm writing a web application using Django. I'm still learning the framework and reading the howto book. I know I might be asking this question prematurely however i'd really like to know. I want to create a python data structure in memory that is shared across all the sessions. What would be the best and most scalable way to perform this. So far I have read about redis however I would like to more flexibility and understand redis can only store strings instead of python objects..
This post is partially close to what you want (excluding the java part and the later update on the post). The summary of the answer is that django is a muti-process environment, and thus sharing objects across sessions is not feasible. One option is to use the database for storing such shared objects.
We have two ColdFusion applications that share a common database. There are three instances of each application. (One instance of each application runs on each of three servers.)
I can see that the three instances of a given application should share a client variable store. (Load-balancing can cause a single user session to bounce between the three instances.) My question is: Is there any danger to having all instances of both applications share the same data store? Or should only one application be pointing at a given data store?
You can use the same client data store. The CDATA table has an 'app' column that stores the coldfusion application name. That column will keep your data unique to each application.
I'm working at an enterprise level ColdFusion shop with multiple CF applications running on the same server that are all pointed at the same client variable store. The only concern within the organization is how the client variable store affects regular backups, and that falls under the data team's purview. We don't have any problems with the different apps actually using the same client variable storage.
Related, from the ColdFusion documentation:
Some browsers allow only 20 cookies to
be set from a particular host.
ColdFusion uses two of these cookies
for the CFID and CFToken identifiers,
and also creates a cookie named
cfglobals to hold global data about
the client, such as HitCount,
TimeCreated, and LastVisit. This
limits you to 17 unique applications
per client-host pair.
I guess this deals more with how many applications you actually run rather than whether you have them all share the same client data store, but it does suggest that there may be some kind of hard limit on the total number of apps you can run at once, although I'd recommend splitting across hosts (or just using a different domain name) if you're planning on more than 16 apps!
As Eric stated above, running multiple apps off of one datasource is fine. What I would warn you is that these databases can fill up fast if you're not careful to block spiders and search engines from using them. Because CF creates client variables on each request for a new session, a search engine will get a new one every time because it never sends its old credentials/cookies so CF thinks it's a new user who needs a new client variable set. Also, be absolutely certain to check "Disable global client variable updates" in CF admin. This will save you a lot of unnecessary overhead.
I would think that multiple applications sharing the same data store would open up the possibility of users from one application having access to the other applications. While it may not be likely, the possibility could exist. (I don't have any facts to back that up, it just seems like a logical conclusion).
The question is, are you comfortable with that possibility, or do you have to absolutely make sure each application is secure?