I am working on a project where urls are put into a Django model called UrlItems. The models.py file containing UrlItems is located in the home app. I typed scrapy startproject scraper in the same directory as the models.py file. Please see this image to better understand my Django project structure.
I understand how to create new UrlItems from my scraper but what if my goal is to get and iterate over my Django project's existing UrlItems inside my spider's def start_requests(self) function?
What I have tried:
1) I followed the marked solution in this question to try and see if my created DjangoItem already had the UrlItems loaded. I tried to use UrlItemDjangoItem.objects.all() in my spider's start_requests function and realized that I would not be able to retrieve my Django project's UrlItems this way.
2) In my spider I tried to import my UrlItems like this from ...models import UrlItem and I received this error ValueError: attempted relative import beyond top-level package.
Update
After some consideration I may end up having the Scrapy spider query my Django application's API to receive a list of the existing Django objects in JSON.
Related
I am learning Django from https://www.youtube.com/watch?v=nAn1KpPlN2w&index=5&list=PL6gx4Cwl9DGBlmzzFcLgDhKTTfNLfX1IK
I have did exactly the same things mentioned in it.But still these errors are coming at the end when I restart my server to see changes in music and admin page. These errors are:
Here is an image of the command line with the errors show.
from django.import views
This is an invalid syntax and is not a correct way to import in Python.
Instead, according to the tutorial, you should be importing views from your django app:
from . import views
I am looking to write a Django app that is a utility.
I would like the user to be presented with a form when they can upload 4 or 5 files.
They would then be able to hit a process button and the 5 files would be processed and combine or zipped and downloaded as 1 file(result_file)
After this the 5 file would be deleted as well at the result_file
I would prefer to avoid having a database on this app as it never really stores ant information
Could all this be done in a view ? and how would you guys approach it ?
This is what I would try:
Create the script that does the file processing work.
You can place the script in a newly created folder inside the django app folder and just import it in your views.py.
You can create an empty file ,"init.py" in the aforementioned new folder, and make it as package to import it in views.py
In the same views.py, there could be another function that has the download feature .
I have a Django project called myproj, with an app called myapp, containing a model called Mymodel. In myproj/myapp, I want to create a Python script that accesses the database. I know how to access the database from myproj/myapp/views.py, but if I have a blank file at myproj/myapp/myscript.py, then what do I need to do to enable database interaction? Currently, if I just write the following in the script:
from models import mymodel
objs = Mymodel.objects.all()
But I get a load of errors for that. Any help?
Create a custom management command
I am am new to Django / Scrapy and well to programming in general. I am trying to make a Django site to help me learn.
What I want to do is Scrape product information from different sites and store them in my postgres database using DjangoItem from Scrapy.
I have read all the docs from both Scrapy and Django. I have searched here and other sites for a couple days and just couldn't find exactly what I was looking for that made the light bulb go off.
Anyway, my question is, what is the standard for deploying Scrapy and Django together. Ideally I would like to scrape 5-10 different sites and store their information in my database.
Scrapy's docs are a little short on information on the best way to implement DjangoItem.
1) Should the Scrapy project be inside my Django app, at the root level of my Django project or outside all together.
2) Other than setting DjangoItem to my Django model, do I need to change any other settings?
Thanks
Brian
I generally put my scrapy project somewhere inside my Django project root folder. Just remember you will need to make sure both projects are in the python path. This is easy to do if you are using virtualenv properly.
Aside from that as long as you can import your Django models from Scrapy i think everything else in the Scrapy docs is very clear. When you import your Django model, the Django settings are set up at that point, this means your database connection etc should all be working fine as long as they are already working in Django.
The only real trick is getting the python path set up properly (which is probably a topic for another questions).
I'm new to django and had a question regarding organizing views. manage.py startapp creates a views.py in my app folder. But django-admin.py startproject <name> does not create a corresponding views.py file in the <project_name>/<project_name> folder.
I find it intuitive to have global views which do not correspond to a particular app. For example, a login page would and should be independent of any app that I create (its associated with the django auth app). So, would it make sense to create another views.py in the <project_name>/<project_name> folder where I can define such views?
(Just wanted to run it by experienced djangoers before I proceed.)
Thanks.
You can write your global views anywhere. it can be in any file name (I use, global_views.py)
I used to write a global to overrride/customize the default framework apps like custom authentication backend and custom sites.
Better to create a custom app and write all the global views.