Im using bs4 for crawling news data.
At the first time, I made crawler function on views.py but it make error 504 error due to long loading time.
So, I decided to crawling and save data with Django ORM with new python file named 'crawling.py' in the same directory with models.
My crawler importing below functions
# from django.portal import settings
from .models import *
import requests
from bs4 import BeautifulSoup
import urllib.request as req
import ssl
from bs4.builder import builder_registry
import time
but it occurs error like below
(project) macs-MacBook-Pro:portalpage mac$ python crawling.py
Traceback (most recent call last):
File "crawling.py", line 2, in <module>
from .models import *
ImportError: attempted relative import with no known parent package
I found way to run my crawler in root directory but I will use crontab for batch jobs so I would like to locate my crawling.py inside app directory.
How could I run my crawler on back-end smoothly?
Related
Django Version: 3.1.5
folder structure
So, I'm studying Django. When I try to generate random data for my project I get this error:
Traceback (most recent call last):
File "C:\PythonProjects\DJANGO\myblogsite\blog\management\commands\create_data.py", line 2, in <module>
from core.models import Category, Post, Comment
ModuleNotFoundError: No module named 'core'
Process finished with exit code 1
create_data.py
from django.core.management.base import BaseCommand
from core.models import Category, Post, Comment
from random import randint
import datetime
Has anybody a clue how to deal with this problem?
from django.core.management.base import BaseCommand
from blog.models import Category, Post, Comment
from random import randint
import datetime
I have installed Django 1.8.3 Then i created my project. I need to import the AppCache but not able to import it. In Django shell i wrote
from django.db.models.loading import AppCache
Error is:
ImportError: cannot import name AppCache
Please help me out.
AppCache is not a part of django.db.models.loading from django 1.7 onwards.For more information you can read this link
I'm using scrapy-0.16 for data extraction from LinkedIn.
from scrapy.selector import HtmlXPathSelector
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.http import Request
from scrapy import log
from linkedin.items import LinkedinItem, PersonProfileItem
from os import path
from linkedin.parser.HtmlParser import HtmlParser
import os
import urllib
from bs4 import UnicodeDammit
from linkedin.db import MongoDBClient
https://github.com/pondering/scrapy-linkedin
The error comes
Traceback (most recent call last):
File "C:\Users\TAWANE DUDEZ\Desktop\linkedin\linkedin\spiders\LinkedinSpider.py", line 6, in <module>
from linkedin.items import LinkedinItem, PersonProfileItem
ImportError: No module named linkedin.items
Cannot find linkedin.items module.
My suspicion is that you're trying to run the scrapy crawl LinkedinSpider command from the wrong directory. Try navigating to C:\Users\TAWANE DUDEZ\Desktop\linkedin and then running the command again.
Since the crawler is now starting, you also need to be running a MongoDB instance before starting the crawl. The README of the github project being used says to typemongod to start an instance. Just to check, you do have MongoDB and pymongo installed right?
I want to setup a Cron job to run a django file.
What I want to know is django-cron a good option for this? i.e. or are there any other apps that can be used?
If no, how to run a django file through command line ?
admin#ADMIN-PC ~/workspace/bolt (master)
$ python ./log/cron.py
Traceback (most recent call last):
File "./log/cron.py", line 3, in <module>
from account.models import UserProfile
ImportError: No module named account.models
I have set the following variable
admin#ADMIN-PC ~/workspace/bolt (master)
$ export DJANGO_SETTINGS_MODULE=settings
I keep getting these errors as the files that are being referenced have direct imports
from foo.models import *
Any help will be highly appreciated.
Custom Command
from django.core.management.base import BaseCommand, CommandError
import pdb
import datetime
from too.models import UserProfile
from foo.api import end_all_foo_conversations
class Command(BaseCommand):
# in minutes
def handle(self,*args,**options):
print datetime
The error I am getting while trying to run the command is as follows:-
'queryset': self.rel.to._default_manager.using(db).complex_fi
imit_choices_to),
AttributeError: 'str' object has no attribute '_default_manager'
I think you should write a custom management command and run it through manage.py.
I am going a tad crazy here. I keep getting this error: ImportError: No module named models and I am not sure why. Here is what I have found so far...
>>> from django.shortcuts import get_object_or_404, redirect
>>> from mystore.cart import cart
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/Users/Jeff/django/mystore/cart/cart.py", line 3, in <module>
from mystore.cart.models import CartItem
ImportError: No module named models
>>>
I am not sure what's going on with this... line 3 in cart.py is this:
from mystore.cart.models import CartItem
If I try to do: from mystore.cart.models import CartItem it works fine...
Any suggestions?
Almost certainly, you have a circular dependency: mystore.cart.cart is importing mystore.cart.models, which in turn is trying to import mystore.cart.cart.
You should determine if both of those imports are necessary, and if either of them could be moved out of the global scope into a function or method.
Why are you doing from mystore.cart import cart? That should be just from mystore import cart.
Very early in the mystore.cart.models an error is occurring that's why nothing in models.py can be imported. The error can be a circular import, a conditional statement that's triggered during runtime but not at the command prompt or is happening inside something else your are importing at the beginning of models.py
You have to put a point before.
bad
from models import *
good
from .models import *
that means that is at the same level.