I am testing my app for concurrency
I am using django-concurrency
For every model added
from concurrency.fields import IntegerVersionField
version = IntegerVersionField( )
So i open with 2 browser windows my app. log with 2 different users and the edit with both same record . Save with one then I save with second one and always second one just overwrites the first one . Basically does nothing .
Expected to get an error on my second save.
What could be the issue ?
I have never used Django concurrency and I would run a mile if I run across apps that offer to abstract concurrency, transaction management etc. Based on general concepts of concurrency. This behaviour seems right. There's no error here.
django-concurrency is an optimistic lock [1] implementation for
Django.
Supported Django versions: 1.8.x, 1.9.x, 1.10.x., 1.11.x
It prevents users from doing concurrent editing in Django both from UI
and from a django command.
What this actually means is that two users cannot modify the same instance at the same time. In adhoc two browser testing of the sort that's just tried out, such a situation can never be produced! What usually happens is that one or the other transaction usually runs to completion because when testing on small databases on localhost, there is no latency and everything happens instantly.
The proper way to test this is to open up to separate change shells, open up the same instance and edit as outlined in the django concurrency guide. Or test with a multi threaded client.
Related
My Django app generates a complex report that can take upto 5 minutes to create. Therefore it runs once a night using a scheduled management command.
That's been ok, except I now want the user to be able to select the date range for the report, which means the report needs to be created while the user waits.
What are my options for running the tast in the background? So far I've found these:
Celery - might work but is complex
django-background-tasks looks like the right tool for the job but hasn't been updated for years, last supported Django is 2.2
The report/background task could be generated by AWS Lambda, basically in a microservice. Django calls the Microservice which can execute the background task then call the Django app back once finished. This is what I did last time but not sure it would work now as I'd need to send the microservice 10mb of data to process.
Use subprocess.popen which someone here said worked for them but other reports say it doesn't work from Django.
EDIT: Looks like Django 3.1 onwards supports ASync views and may be the simple solution for this.
I've been reading through related articles regarding using same database for multiple django projects however, I have not been able to come up with a fix yet.
I've tried relative and absolute pathing when importing but it gives "attempted relative import beyond top-level package" error when I try to access parent directory.
Project 1 gets the users to write to database by filling in a form and Project 2 retrieves data written in by users from database.
I'm using Postgresql for database. I've tried writing exactly same models.py for both projects but it seems like in database they appear as separate relations/tables. E.g. for table named school in both models.py, it would look like project1_school and project2_school in Postgres database.
Is there a way to write to and read from same tables of same database?
Thank you so much in advance.
I think that you might be confuse with the difference between Projects and Applications.
Projects vs. apps
What’s the difference between a project and an app? An app is a Web application that does something – e.g., a Weblog system, a database of public records or a small poll app. A project is a collection of configuration and apps for a particular website. A project can contain multiple apps. An app can be in multiple projects.
Writing your first Django app, part 1
So in you particular case, I would say that your actual projects, both of them, could be applications of one project. The main reason, why I think this is a better approach, is that both are gonna use the same data, one application writes while the other retrieve it. One could even argue that they actually could be the same application. But this may depend on many factors of your business.
BTW, is really hard for me to imagine a situation where it would be a good idea to have two projects using the same database. Even if both projects need to share data, I would not think in using on database. I would try to solve it at an application level. But I you need for some reason to share information at database level, there are tools to connect both databases.
I have a Django (v1.4, using Postgresql) project which I've written a bunch of working unittests for. These use FactoryBoy to generate most of their data.
I'm now starting to write some integration tests using LiveServerTestCase with Selenium. I've just realised that my tests and the live test server use different databases. Which means that data created by factories in my tests aren't available to Selenium.
I'm not sure of the best way to progress. I think I could use fixtures to supply data that would work, although this is a pain having got this far using factories instead.
Is there a way I can continue to use factories to generate data that will work for my Selenium tests? Really I'd like my tests and LiveServerTestCase to use the same database.
I found out why this happened to me, and some possible workarounds, including Ilya Baryshev's answer above.
If your test descends from Django's TestCase, and if your database supports transactions, then each test runs in its own transaction, and nobody outside (no other thread, external process, or other test) can see the objects created in the database by your test.
LiveServerTestCase uses threads, so it would suffer from this problem. So the designers made it inherit from TransactionTestCase instead of TestCase, which disables these transactions, so that changes are globally visible.
What happened to me was that I added some mixins to my test class, and one of them pulled in TestCase. This doesn't cause an error, but it silently replaces the base class of LiveServerTestCase with TestCase, which enables transactions again, causing the problem that you describe.
Ilya's SQLite memory database workaround works because Django contains code that detects when using a SQLite :memory: database that actually shares the same connection between threads, so you see your test's objects in the LiveServerThread because they're inside the same transaction. However this comes with some caveats:
It’s important to prevent simultaneous database queries via this shared connection by the two threads, as that may sometimes randomly cause the tests to fail. So you need to ensure that the two threads don’t access the database at the same time. In particular, this means that in some cases (for example, just after clicking a link or submitting a form), you might need to check that a response is received by Selenium and that the next page is loaded before proceeding with further test execution. Do this, for example, by making Selenium wait until the HTML tag is found in the response (requires Selenium > 2.13)...
https://docs.djangoproject.com/en/1.4/topics/testing/#live-test-server
In my case, once we identifier that autocommit was being turned off when the test started, and tracked down why (because we had entered TestCase code that we shouldn't have done), we were able to fix the inheritance hierarchy to avoid pulling in TestCase, and then the same database was visible from both the live server thread and the test.
This also works with Postgres databases, so it would provide a solution for velotron.
Have you tried using sqlite as your database backend for tests?
When using an in-memory SQLite database to run the tests, the same
database connection will be shared by two threads in parallel: the
thread in which the live server is run and the thread in which the
test case is run.
from Django docs
If you're not using anything beyond regular ORM, you might benefit from test speedups as well.
How to make Django execute something automatically at a particular time.?
For example, my django application has to ftp upload to remote servers at pre defined times. The ftp server addresses, usernames, passwords, time, day and frequency has been defined in a django model.
I want to run a file upload automatically based on the values stored in the model.
One way to do is to write a python script and add it to the crontab. This script runs every minute and keeps an eye on the time values defined in the model.
Other thing that I can roughly think of is maybe django signals. I'm not sure if they can handle this issue. Is there a way to generate signals at predefined times (Haven't read indepth about them yet).
Just for the record - there is also celery which allows to schedule messages for the future dispatch. It's, however, a different beast than cron, as it requires/uses RabbitMQ and is meant for message queues.
I have been thinking about this recently and have found django-cron which seems as though it would do what you want.
Edit: Also if you are not specifically looking for Django based solution, I have recently used scheduler.py, which is a small single file script which works well and is simple to use.
I've had really good experiences with django-chronograph.
You need to set one crontab task: to call the chronograph python management command, which then runs other custom management commands, based on an admin-tweakable schedule
The problem you're describing is best solved using cron, not Django directly. Since it seems that you need to store data about your ftp uploads in your database (using Django to access it for logs or graphs or whatever), you can make a python script that uses Django which runs via cron.
James Bennett wrote a great article on how to do this which you can read in full here: http://www.b-list.org/weblog/2007/sep/22/standalone-django-scripts/
The main gist of it is that, you can write standalone django scripts that cron can launch and run periodically, and these scripts can fully utilize your Django database, models, and anything else they want to. This gives you the flexibility to run whatever code you need and populate your database, while not trying to make Django do something it wasn't meant to do (Django is a web framework, and is event-driven, not time-driven).
Best of luck!
I just finished a Django app that I want to get some outside user feedback on. I'd like to launch one version and then fork a private version so I can incorporate feedback and add more features. I'm planning to do lots of small iterations of this process. I'm new to web development; how do websites typically do this? Is it simply a matter of copying my Django project folder to another directory, launching the server there, and continuing my dev work in the original directory? Or would I want to use a version control system instead? My intuition is that it's the latter, but if so, it seems like a huge topic with many uses (e.g. collaboration, which doesn't apply here) and I don't really know where to start.
1) Seperate URLs www.yoursite.com vs test.yoursite.com. you can also do www.yoursite.com and www.yoursite.com/development, etc.. You could also create a /beta or /staging..
2) Keep seperate databases, one for production, and one for development. Write a script that will copy your live database into a dev database. Keep one database for each type of site you create. (You may want to create a beta or staging database for your tester).. Do your own work in the dev database. If you change the database structure, save the changes as a .sql file that can be loaded and run on the live site database when you turn those changes live.
3) Merge features into your different sites with version control. I am currently playing with a subversion setup for web apps that has my stable (trunk), one for staging, and one for development. Development tags + branches get merged into staging, and then staging tags/branches get merged into stable. Version control will let you manage your source code in any way you want. You will have to find a methodology that works for you and use it.
4) Consider build automation. It will publish your site for you automatically. Take a look at http://ant.apache.org/. It can drive a lot of automatically checking out your code and uploading it to each specific site as you might need.
5) Toy of the month: There is a utility called cUrl that you may find valuable. It does a lot from the command line. This might be okay for you to do in case you don't want to use all or any of Ant.
Good luck!
You would typically use version control, and have two domains: your-site.com and test.your-site.com. Then your-site.com would always update to trunk which is the current latest, shipping version. You would do your development in a branch of trunk and test.your-site.com would update to that. Then you periodically merge changes from your development branch to trunk.
Jas Panesar has the best answer if you are asking this from a development standpoint, certainly. That is, if you're just asking how to easily keep your new developments separate from the site that is already running. However, if your question was actually asking how to run both versions simultaniously, then here's my two cents.
Your setup has a lot to do with this, but I always recommend running process-based web servers in the first place. That is, not to use threaded servers (less relevant to this question) and not embedding in the web server (that is, not using mod_python, which is the relevant part here). So, you have one or more processes getting HTTP requests from your web server (Apache, Nginx, Lighttpd, etc.). Now, when you want to try something out live, without affecting your normal running site, you can bring up a process serving requests that never gets the regular requests proxied to it like the others do. That is, normal users don't see it.
You can setup a subdomain that points to this one, and you can install middleware that redirects "special" user to the beta version. This allows you to unroll new features to some users, but not others.
Now, the biggest issues come with database changes. Schema migration is a big deal and something most of us never pay attention to. I think that running side-by-side is great, because it forces you to do schema migrations correctly. That is, you can't just shut everything down and run lengthy schema changes before bringing it back up. You'd never see any remotely important site doing that.
The key is those small steps. You need to always have two versions of your code able to access the same database, so changes you make for the new code need to not break the old code. This breaks down into a few steps you can always make:
You can add a column with a default value, or that is optional. The new code can use it, and the old code can ignore it.
You can update the live version with code that knows to use a new column, at which point you can make it required.
You can make the new version ignore a column, and when it becomes the main version, you can delete that column.
You can make these small steps to migrate between any schemas. You can iteratively add a new column that replaces an old one, roll out the new code, and remove the old column, all without interrupting service.
That said, its your first web app? You can probably break it. You probably have few users :-) But, it is fantastic you're even asking this question. Many "professionals" fair to ever ask it, and even then fewer answer it.
What I do is have an export a copy of my SVN repository and put the files on the live production server, and then keep a virtual machine with a development working copy, and submit the changes to the repo when Im done.