I want to run a django query from a string and put the output into a variable
in my DRF project, the client sends a django query:
{'query': 'model.objects.all()'}
and I need to return the result of this query.
I tried using exec('model.objects.all()') but i can't assign the output to a variable, i also tried using subprocess.run([sys.executable, "-c", 'model.objects.all()'], capture_output=True, text=True) but subrocess doesn't find the model
There's a huge amount of setting up needed before a process using Django models can work correctly. That's why manage.py shell exists.
If you want to perform Django operations outside the context of a Django server, write a management command. You can then invoke it from the command line, from cron, from other Python scripts ... wherever.
Related
I'm using django-rest-framework as backend and postgresql as the database. The database might be changed by raw SQL script and I want to get notified in the backend when those changes happen so that I can notify different users about the change.
I've checked about posts like https://gist.github.com/pkese/2790749 for receiving notification in python and some SQL scripts for
CREATE TRIGGER rec_notify_trig AFTER INSERT OR UPDATE OR DELETE ON rec
FOR EACH ROW EXECUTE PROCEDURE rec_notify_func()
My question is that I don't know how to hook them together in the django-rest-framework, like where should I put the SQL script, where to put the python settup so that I can connect them together. Any advice will be appreciated.
I would create an endpoint on the djangorestframework side to accept a notification.
Then, in your rec_notify_func() you can call out and hit your endpoint where you can perform any enduser notification necessary.
CREATE EXTENSION plpython3u;
CREATE FUNCTION rec_notify_func(notification_endpoint_uri text) RETURNS text AS $$
from urllib.request import urlopen
data = urlopen(notification_endpoint_uri)
return data.read()
$$ LANGUAGE plpython3u;
NOTE:
You need to have plpython installed on the system in order to enable the extension.
On ubuntu something like this:
sudo apt-get install postgresql-plpython3-9.6
I have a webapp that I created using Django2. At a high level, it will be used to process .tsv files of data and display them nicely on a screen.
I want to be able to have a command line interface where I can perform a POST request to the already running webapp, and essentially add data to a model, save it, and create a unique webpage to display that data. Something like:
uploadtodjangoapp <myfilename> --user='heidi' --other-options='....'
uploading myfilename to myapp!
done
see data here: www.mysite.com/info/myfilename
In this situation ^ the webpage will be running already somewhere (either locally or on a vm).
Currently, I know you can create a form on the user interface to perform post requests/get user data. And I know you can also use python manage.py shell and do something like:
>> from myapp.model import mymodel
>> m = mymodel(data="some data here")
>> m.save()
.... but is this the only way?
Any help would be greatly appreciated!
You can easily achieve this using curl
Just for Example :
In terminal
curl --data "field_1=data_1&field_2=data_2&field_3=data_3" <API FOR POST REQUEST>
I put Celery in my Django app so that the two other python programs can process the input from my Django app via doing subprocess method.
My question is how do I access the output from the subprocess? Back then when I made just a python program, I access the log files (output from the two apps) via stdout and stderr. Is this the same when I use Celery in Django? Is the value of CELERY_RESULT_BACKEND (if I should assign my Django app's db here) affected by the log files?
So far what I've done is:
Access the two apps via subprocess in my tasks.py
I assigned my broker's db, Redis, as my db for now for CELERY_RESULT_BACKEND. My plan is to get the log files and then save them to my Django app's db so that I can just access that db.
Can you offer some help?
Typically, you only care about the task result, which is the return value of the celery task, and that is stored in your result_backend for at least result_expires time (usually 1 day). So, to the extent that you want to access any particular task's result, you can just do so using the task ID.
I've managed to write a Scrapy project that scrapes data from a web page, and when I call it with the scrapy crawl dmoz -o items.json -t json at the command line, it successfully outputs the scraped data to a JSON file.
I then wrote another script that takes that JSON file, loads it, changes the way the data is organized (I didn't like the default way it was being organized), and spits it out as a second JSON file. I then use Django's manage.py loaddata fixture.json command to load the contents of that second file into a Django database.
Now, I'm sensing that I'm going to get laughed out of the building for doing this in three separate steps, but I'm not quite sure how to put it all together into one script. For starters, it does seem really stupid that I can't just have my Scrapy project output my data in the exact way that I want. But where do I put the code to modify the 'default' way that Feed exports is outputting my data? Would it just go in my pipelines.py file?
And secondly, I want to call the scraper from inside a python script that will then also load the resulting JSON fixture into my database. Is that as simple as putting something like:
from twisted.internet import reactor
from scrapy.crawler import Crawler
from scrapy.settings import Settings
from scrapy import log
from testspiders.spiders.followall import FollowAllSpider
spider = FollowAllSpider(domain='scrapinghub.com')
crawler = Crawler(Settings())
crawler.configure()
crawler.crawl(spider)
crawler.start()
log.start()
reactor.run()
at the top of my script, and then following it with something like:
from django.something.manage import loaddata
loaddata('/path/to/fixture.json')
? And finally, is there any specific place this script would have to live relative to both my Django project and the Scrapy project for it to work properly?
Exactly that. Define a custom item pipeline in pipelines.py to output the item data as desired, then add the pipeline class to settings.py. The scrapy documentation has a JSONWriterPipeline example that may be of use.
Well, on the basis that the script in your example was taken from the scrapy documentation, it should work. Have you tried it?
The location shouldn't matter, so long as all of the necessary imports work. You could test this by firing a Python interpreter in the desired location and then checking all of the imports one by one. If they all run correctly, then the script should be fine.
If anything goes wrong, then post another question in here with the relevant code and what was tried and I'm sure someone will be happy to help out. :)
I am setting up our project to integrate with a shipping platform called Endicia which has the ability to insert new rows into our database when a package is shipped.
How can I detect from python when a new row has been inserted?
My solution for now would be to query the DB every 30 seconds or so for new rows... is there another solution to send a signal from postgres to python?
You'd set up a custom command that is run by the manage.py file.
You'd put it in `yourapp/management/commands/' folder. Make sure to add an init.py file to both the management and commands folder or the command won't work. Then you create the code for the custom command.
Then, see this related question about running a shell script when changes are made to a postgres database. The answer there was to use PL/sh. You'll need to figure that part out on your own, but basically however you do it, the end result is that the script should call something like /path/to/app/manage.py command_name