How do I put custom spider middlewares in my scrapy project?
Let's say I want to put this one. As the documentation says, I added
SPIDER_MIDDLEWARES = { 'myproject.middlewares.IgnoreVisitedItems': 543,}
to settings.py and the code from the above link to _init_.py in spider folder.
But I'm getting:
ImportError: Error loading object 'myproject.middlewares.IgnoreVisitedItems': No module named middlewares
I tried things on this but it didn;t work.
This is my project structure:
How should I get this custom middleware working? I just need a generalized method to get it working, not specifically for this project.
it seem that your directory structure does not have a file called middlewares as you referred in your project settings.py file, moreover, it seem that your project name is cnn and not myproject.
if this is the case, you'll have to do two things:
create middlewares.py file and put IgnoreVisitedItems in it, place it in the same directory as items.py and pipelines.py
change you settings SPIDER_MIDDLEWARES entry to:
SPIDER_MIDDLEWARES = { 'cnn.middlewares.IgnoreVisitedItems': 543,}
EDIT: note however that in most cases scrapy will ignore visited Requests by default, unless you'll force it to recrawl them using Request's dont_filter parameter
Related
I'm trying to run a .py file and in the file I have this import
from config.wsgi import *
import os
from django.template.loader import get_template
from weasyprint import HTML, CSS
from config import settings
The whole project works, if I set runserver, the project starts without any problem, but this file does not work. The structure of the project is as follows
NombreDelProyecto
--app
---config
----__init__.py
----asgi.py
----settings.py
----wsgy.py
----db.py
---core
----general
----login
----user
----archivodetest.py
the case as I say the project works, but in the views of the applications that I have been doing to put imports I get in red underlined but as I say it works for example:
from core.general.forms import ActividadForm
That comes out in red, if I put in front of the core, app.core as follows
from app.core.general.forms import ActividadForm
it does not show red but the project does not work and I get the following error
from app.core.general.forms import ActividadForm
ModuleNotFoundError: No module named 'app'
I understand that it is the routes or something I did wrong from the beginning, please could someone help me.
Thank you very much.
I tried adding the route, changing the app's route in settings, but to no avail.
You've named the file wsgy.py but it needs to be wsgi.py.
Rename your file in config and retry.
To your question, I think its because you're missing the __init__.py file in the general app.
If you haven't already go one, you'll likely need to have add the same again in your core app too.
You probably manually created all of these files and structures I suspect, and if that's the case, please take a look at the documentation regarding creating new apps inside a django project.
If you go a bit further up the page, it will also tell you how to create the initial django project structure with a command.
Thank you very much for the answer, I managed to solve it after a lot of testing.
There are two ways, open the project again from the app folder (I had it open in the ProjectName folder).
Or as a second option in pyCharm on the left where the project folders are, I went to the app folder (which is the root) and right clicked and in the menu, Mark Directorie as - Sources root. Then my problem is fixed.
I had all the arcvhiso init.py, and where I put the wrong name wsgi.py is that I wrote it wrong here but in the project it was right.
Thank you very much for the help.
I'm having a weird problem with import views in Django project. I'm not sure whether it is a problem caused by PyCharm of Django. So PyCharm says that it can't import views.py file, but it works when I run the server.
Here is the picture:
Do you know where could be the problem?
EDIT:
According to Inlangers answer, I've tried to change import to from vlado_web.translations import views which did not helped, moreover, it raises
Exception Value:
No module named translations
When I have from translations import views there, it works correctly but PyCharm says that it can't be resolved.
PyCharm doesn't know where your source files are. Try this:
Right click on folder vlado_web (the folder that contains manage.py) within PyCharm. Go to Mark Directory As and choose Sources Root.
This will let PyCharm know that the vlado_web directory is the root folder for your source code, and will allow you to perform absolute imports from there, e.g.
from vlado_web.translations import views
Try from vlado_web.translations import views
It seems to be a Django best practice to add the settings.py file to the ignore list of the version control system. But how does one start to set up a project when the settings.py file was not included in the (Github) project*?
Starting from a default settings file is not at all easy because I have no experience with Django, and see that there are an specific settings for included modules/frameworks that need to be included.
Is there a way to create a settings.py file from the code in an existing Django project? Is there any other alternative to trial and error if the author cannot be contacted or is not responding to the request for an example file?
In fact is not a best practice to put it on the ignore list based on the opinion of Daniel Greenfeld (AKA Pydany) and Audrey Roy on "Two Scoops of Django". In fact is recommended that you keep one setting file for every setup your team members have.
Of course the easiest way to proceed is to contact the author and if he is that kind you can get it.
I would recommend the following:
Take a default settings.py from an new project
Locate the requirements.txt and verify which one of the apps located there are
django base and include them to the settings.py
Include all the
apps explicitly defined on the project (all the folders different to
the appname folder)
After that is just trial en error( or more
like trial and exception).
I have a pycharm project and, presumably, a Django project. Perhaps they are one and the same, perhaps not - I'm unsure as to the distinction.
Anyway, in my settings.py file (which is in the root of project directory, which is what I presume is my pycharm project) I have:
ROOT_URLCONF = 'dumpstown.urls'
Does that mean dumpstown is my project name? Or my pycharm project name? What is the distinction? Because I also have a folder called dumpstownapp and this has all my models.py and view.py files. I would have thought that dumpstownapp was the Django project, but I really don't know!
So, to be concise:
In this folder setup
folderA
folderB
views.py
models.py
<other stuff>
templates folder
settings.py
<other stuff>
which is the "Django project name" ~ and by that I mean, if I have a UserProfile defined in my models.py (shown above) what would be the AUTH_PROFILE_MODULE entry I'd need for it? I'm getting several understandings from the django docs - I'd assume
dumpstownapp.models.UserProfile
But from the docs I'd get
dumpstownapp.UserProfile
Or maybe my app is called dumpstown? and then what do I get?
FolderA is the Django project folder, and folderB is the Django app folder.
I haven't used a UserProfile, but according to the docs ( https://docs.djangoproject.com/en/dev/topics/auth/#storing-additional-information-about-users) it should contain a string containing the Django app name and the model name separated by a dot, so "dumpstownapp.UserProfile" should be correct. If you show the error someone can probably give you better help..
Django documentation used to say that the parent of the project folder (i.e. the parent of folderA) should be on the path, but I believe that has been changed to also include the project folder itself (i.e. folderA) -- since it makes sharing of Django apps much easier. PyCharm seems to assume that is the case, since pressing Alt+F7 to auto-add an import for a newly used module create an import statement that assumes folderA is on the import path (I'm a relative newcomer to PyCharm, and I'm using it on a project that started in the Django 0.96 era, so I might just have things set up wrong..) But folderA is both the Django and the PyCharm project (the .idea file is where PyCharm stores its project data).
In one of my django-app git-submodule projects I needed to find out the name of the Django project that django-app/library was used in. To that end, I tried to get the path of the file that was being executed, find my package in the path and take the folder above it. However, it turned out that on the production server the project was deployed in a folder with a different name (a standard name like www or something along those lines). So this way is not fully reliable.
So I ended up setting a PROJECT variable in the django settings file and using that instead.
I'm aware I can "store it anywhere in my python path" and all, but what's an organized pattern I can use to store middleware classes for my project?
I am appending my project root directory and project directory to the sys path through mod_wsgi:
sys.path.append( '/srv/' )
sys.path.append( '/srv/workarounds/' )
The latter line being the project root. As an example, let's say I want to apply this middleware class: http://djangosnippets.org/snippets/1179/
Would I just copy the snippet contents into a middleware.py file and dump it in my project root? Create a directory for middleware, add that directory to my python path?
My usual layout for a django site is:
projects/
templates/
common/
local/
Where:
projects contains your main project and any others
common contains things you may share across sites, or are at least not project-specific, like if you need to download django-profile and django-registration rather than having it directly in python/site-packages
templates contains just that
local contains things that are going to be specific to the current machine, so that you can have properly separated data, like database location and password - I then soft-link the machine-specific versions (say "machine1-localconfig.py") to local/localconfig.py and then can "import localconfig" in settings.py
I generally put middleware that's project-specific inside a project, and middleware that's not project-specific in common/middleware/
Make sure to add the templates directory to the right place in settings (or most probably, localconfig.py and then import it in settings), and makse sure to add the projects, common, and local directories to your PYTHONPATH.
If you have only a couple of tightly coupled middleware classes put them into a middleware.py module under the app root. (This is how the django.contrib apps do it - see the sessions' app middleware here).
If you have many varying middleware classes, create a middleware pacakge with submodules of the related middleware classes. Though if you do end up in this situation, consider how you can refactor your project into several mini-apps that all solve a specific need (and open source them :)).
Personally, I have a common django package where I dump common middleware (like the your linked LoginRequiredMiddleware class) into a middleware package. If this makes sense in your project's context I would highly suggest it. It's saved my countless hours of duplication, and bug fixing. django-common and django-annoying are good examples of this kind of project layout