File writing in Django keeps having IOError - django

I'm running my app locally and I'm currently having an IOError during my file creation from the database. I am using Django 1.10, MongoDB as my database, and Celery 4.0.2 for my background tasks. The problem occurs in the tasks.py since that is where I access the db then store it in my django subfolder 'analysis_samples'.
Here is the traceback:
[2017-04-15 15:31:08,798: ERROR/PoolWorker-2] Task tasks.process_sample_input[0619194e-4300-4a1d-91b0-20766e048c4a] raised unexpected: IOError(2, 'No such file or directory')
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 367, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 622, in __protected_call__
return self.run(*args, **kwargs)
File "/home/user/django_apps/myapp/analysis/tasks.py", line 218, in process_sample_input
with open(sample_file_path, "w+") as f:
IOError: [Errno 2] No such file or directory: u'/home/user/django_apps/myapp/myapp/analysis_samples/58f1cc3c45015d127c3d68c1'
And here is the snippet of tasks.py:
from django.core.files import File
sys.path.append(settings.ANALYSIS_SAMPLES)
import base64
import os, sys
#shared_task(name='tasks.process_sample_input')
def process_sample_input(instance_id):
instance = Sample.objects.get(pk=instance_id)
#many code here..
try:
conn=pymongo.MongoClient(settings.MONGO_HOST, settings.MONGO_PORT)
db = conn.thugfs #connect to GridFS db of thug
thugfs_db = GridFS(db)
except pymongo.errors.ConnectionFailure, e:
logger.error("Could not connect to ThugFS MongoDB: %s" % e)
sample_file_folder = settings.ANALYSIS_SAMPLES
for sample_fs_id in sample_fs_ids:
sample_file = thugfs_db.get(ObjectId(sample_fs_id)).read()
sample_file = base64.b64decode(sample_file) #decode file from database
sample_file_path = os.path.join(sample_file_folder, sample_fs_id)
with open(sample_file_path, "w+") as f:
fileOut = File(f)
fileOut.write(sample_file)
settings.py:
ANALYSIS_SAMPLES = os.path.join(BASE_DIR, 'myapp/analysis_samples')
Can anyone see the point that caused the error? Any help will be appreciated.

Related

Scrapy error when run with Celery and Django Channels is installed: OSError: [Errno 9] Bad file descriptor

When I run this script with Django shell it will run without error, but when I run with celery worker it get this error: OSError: [Errno 9] Bad file descriptor. If I remove channels from INSTALLED_APPS the problem will be solved and task runs with celery worker without error. Running the same script with python Process (due to ReactorNotRestartable exception) in django shell get the same error which celery worker had: OSError: [Errno 9] Bad file descriptor.
I think there's a conflict between Django Channels & Scrapy when spider is executed in another process.
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings
def run_spider(spider)
settings = get_project_settings()
crawler_process = CrawlerProcess(settings=settings)
crawler_process.crawl(spider)
crawler_process.start()
exception stack trace when run with celery worker:
Task run spider raised unexpected: OSError(9, 'Bad file descriptor')
Traceback (most recent call last):
File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py", line 256, in _add_reader
key = self._selector.get_key(fd)
File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/selectors.py", line 192, in get_key
raise KeyError("{!r} is not registered".format(fileobj)) from None
KeyError: '9 is not registered'
exception stack trace when run with python Process in Django shell:
/usr/local/lib/python3.7/site-packages/billiard/pool.py in apply(self, func, args, kwds)
1387 '''
1388 if self._state == RUN:
-> 1389 return self.apply_async(func, args, kwds).get()
1390
1391 def starmap(self, func, iterable, chunksize=None):
/usr/local/lib/python3.7/site-packages/billiard/pool.py in get(self, timeout)
1787 return self._value
1788 else:
-> 1789 raise self._value.exception
1790
1791 def safe_apply_callback(self, fun, *args, **kwargs):
OSError: [Errno 9] Bad file descriptor

Flask Security - Send confirmation email?

I am trying to create a flask boilerplate app and I am running into an error when I try to run a flask-script. I am getting an error that I don't fully understand, the code is broken up into several modules so it may be best to post the link to the repository. The two parts in the traceback that are my code are an import from the app and the creation of the security object.
The two files that are causing conflict are:
manage.py
app/__init__.py
What caused the problem is that I set several security variables to true:
SECURITY_REGISTERABLE = True
SECURITY_SEND_REGISTER_EMAIL = True
SECURITY_CONFIRMABLE = True
SECURITY_CONFIRM_URL = True
SECURITY_EMAIL_SENDER = 'some_email_account'
SECURITY_CONFIRM_LOGIN_WITHOUT_CONFIRMATION = False
SECURITY_TRACKABLE = True
SECURITY_CHANGEABLE = True
The project can be found here:
github link
Traceback (most recent call last):
File "manage.py", line 4, in <module>
from app import app, db, Role, user_datastore
File "C:\Users\...\dev\flask-boilerplate\app\__init__.py", line 22, in <module>
security = Security(app, user_datastore)
File "C:\Users\...\dev\flask-boilerplate\venv\lib\site-packages\flask_security\core.py", line 469, in __init__
self._state = self.init_app(app, datastore, **kwargs)
File "C:\Users\...\dev\flask-boilerplate\venv\lib\site-packages\flask_security\core.py", line 507, in init_app
app.register_blueprint(create_blueprint(state, __name__))
File "C:\Users\...\dev\flask-boilerplate\venv\lib\site-packages\flask_security\views.py", line 383, in create_blueprint
'<token>'),
File "C:\Users\...\dev\flask-boilerplate\venv\lib\site-packages\flask_security\utils.py", line 249, in slash_url_suffix
return url.endswith('/') and ('%s/' % suffix) or ('/%s' % suffix)
AttributeError: 'bool' object has no attribute 'endswith'
SECURITY_CONFIRM_URL must be a string (i.e. SECURITY_CONFIRM_URL='https://google.com)

Log warning from Selenium on Django [duplicate]

Whenever I try to construct a string based on self.live_server_url, I get python TypeError messages. For example, I've tried the following string constructions (form 1 & 2 below), but I experience the same TypeError. My desired string is the Live Server URL with "/lists" appended. NOTE: the actual test does succeed to create a server and I can manually access the server, and more specifically, I can manually access the exact URL that I'm trying to build programmatically (e.g. 'http://localhost:8081/lists').
TypeErrors occur with these string constructions.
# FORM 1
lists_live_server_url = '%s%s' % (self.live_server_url, '/lists')
# FORM 2
lists_live_server_url = '{0}{1}'.format(self.live_server_url, '/lists')
self.browser.get(lists_live_server_url)
There is no python error with this form (nothing appended to string), albeit my test fails (as I would expect since it isn't accessing /lists).
self.browser.get(self.live_server_url)
Here is the python error that I'm getting.
/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/bin/python3.4 /Applications/PyCharm.app/Contents/helpers/pycharm/django_test_manage.py test functional_tests.lists_tests.LiveNewVisitorTest.test_can_start_a_list_and_retrieve_it_later /Users/myusername/PycharmProjects/mysite_proj
Testing started at 11:55 AM ...
Creating test database for alias 'default'...
Traceback (most recent call last):
File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/wsgiref/handlers.py", line 137, in run
self.result = application(self.environ, self.start_response)
File "/usr/local/lib/python3.4/site-packages/django/test/testcases.py", line 1104, in __call__
return super(FSFilesHandler, self).__call__(environ, start_response)
File "/usr/local/lib/python3.4/site-packages/django/core/handlers/wsgi.py", line 189, in __call__
response = self.get_response(request)
File "/usr/local/lib/python3.4/site-packages/django/test/testcases.py", line 1087, in get_response
return self.serve(request)
File "/usr/local/lib/python3.4/site-packages/django/test/testcases.py", line 1099, in serve
return serve(request, final_rel_path, document_root=self.get_base_dir())
File "/usr/local/lib/python3.4/site-packages/django/views/static.py", line 54, in serve
fullpath = os.path.join(document_root, newpath)
File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/posixpath.py", line 82, in join
path += b
TypeError: unsupported operand type(s) for +=: 'NoneType' and 'str'
Am I unknowingly attempting to modify the live_server_url, which is leading to these TypeErrors? How could I programmatically build a string of live_server_url + "/lists"?
Here is the test that I am attempting...
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from django.test import LiveServerTestCase
class LiveNewVisitorTest(LiveServerTestCase):
def setUp(self):
self.browser = webdriver.Chrome()
self.browser.implicitly_wait(3)
def tearDown(self):
self.browser.close()
def test_can_start_a_list_and_retrieve_it_later(self):
#self.browser.get('http://localhost:8000/lists')
#self.browser.get('http://www.google.com')
#lists_live_server_url = '%s%s' % (self.live_server_url, '/lists')
#lists_live_server_url = '{0}{1}'.format(self.live_server_url, '/lists')
lists_live_server_url = self.live_server_url
self.browser.get(lists_live_server_url)
self.assertIn('To-Do', self.browser.title)
header_text = self.browser.find_element_by_tag_name('h1').text
self.assertIn('To-Do', header_text)
See this discussion on Reddit featuring the same error Traceback.
Basically, this is not a problem with anything within the Selenium tests but rather with your project's static file configuration.
From your question, I believe the key line within the Traceback is:
File "/usr/local/lib/python3.4/site-packages/django/views/static.py", line 54, in serve
fullpath = os.path.join(document_root, newpath)
This line indicates that an unsuccessful os.path.join is being attempted within django.views.static.
Set STATIC_ROOT in your project's settings.pyfile and you should be good.
Use StaticLiveServerTestCase instead may help

Python SnakeSQL Table not found error

I am learning Python and SnakeSQL for my database operations. I am following the examples of both web.py and SnakeSQL's. I combined both codes from their web sites and tried to make the below code worked but I am getting this error:
Traceback (most recent call last):
File ".\app.py", line 9, in
cursor.execute("INSERT INTO Test (dateColumn, numberColumn) VALUES ('2004->11-8', 4)")
File "D:\Python\virtualenvs\new4\textweb\bin\SnakeSQL\driver\base.py", line >1552, in execute
self.info = self.connection._insert(parsedSQL['table'], >parsedSQL['columns'], parsedSQL['sqlValues'], parameters)
File "D:\Python\virtualenvs\new4\textweb\bin\SnakeSQL\driver\base.py", line >1040, in _insert
raise SQLError("Table '%s' not found."%(table))
error.SQLError: Table 'Test' not found.
And here my app.py code:
import web
import SnakeSQL
render = web.template.render('templates/')
connection = SnakeSQL.connect(database='test', autoCreate=True)
connection = SnakeSQL.connect(database='test')
cursor = connection.cursor()
cursor.execute("INSERT INTO Test (dateColumn, numberColumn) VALUES ('2004-11-8', 4)")
urls = (
'/', 'index'
)
class index:
def GET(self, name):
cursor.execute("SELECT * FROM Test")
results = cursor.fetchall()
return render.index(results)
if __name__ == "__main__":
app = web.application(urls, globals())
app.run()
What could be wrong in here?

Scrapy-Scraper Does Not Run

I can run python using Beautiful Soup and Mechanized, but for some reason when I try to use Spray-Scraper it just doesn't work. Here's an example of what happens when I attempt to test the scraper with a tutorial:
Project name & BOT name = "tutorial"
The following scripts are the items.py and settings.py that I used.
items.py
import scrapy
class DmozSpider(scrapy.Spider):
name = "dmoz"
allowed_domains = ["dmoz.org"]
start_urls = [
"http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
"http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
]
def parse(self, response):
filename = response.url.split("/")[-2]
with open(filename, 'wb') as f:
f.write(response.body)
settings.py
BOT_NAME = 'tutorial'
SPIDER_MODULES = ['tutorial.spiders']
NEWSPIDER_MODULE = 'tutorial.spiders'
CMD
C:\Users\Turbo>scrapy startproject tutorial
New Scrapy project 'tutorial' created in:
C:\Users\Turbo\tutorial
You can start your first spider with:
cd tutorial
scrapy genspider example example.com
C:\Users\Turbo>cd tutorial
C:\Users\Turbo\tutorial>scrapy crawl dmoz
Traceback (most recent call last):
File "C:\Python27\Scripts\scrapy-script.py", line 9, in <module>
load_entry_point('scrapy==0.24.4', 'console_scripts', 'scrapy')()
File "C:\Python27\lib\site-packages\scrapy-0.24.4-py2.7.egg\scrapy\cmdline.py"
, line 143, in execute
_run_print_help(parser, _run_command, cmd, args, opts)
File "C:\Python27\lib\site-packages\scrapy-0.24.4-py2.7.egg\scrapy\cmdline.py"
, line 89, in _run_print_help
func(*a, **kw)
File "C:\Python27\lib\site-packages\scrapy-0.24.4-py2.7.egg\scrapy\cmdline.py"
, line 150, in _run_command
cmd.run(args, opts)
File "C:\Python27\lib\site-packages\scrapy-0.24.4-py2.7.egg\scrapy\commands\cr
awl.py", line 58, in run
spider = crawler.spiders.create(spname, **opts.spargs)
File "C:\Python27\lib\site-packages\scrapy-0.24.4-py2.7.egg\scrapy\spidermanag
er.py", line 44, in create
raise KeyError("Spider not found: %s" % spider_name)
KeyError: 'Spider not found: dmoz'
The problem is that you are putting your spider into the items.py.
Instead, create a package spiders, inside it create a dmoz.py and put your spider into it.
See more at Our first Spider paragraph of the tutorial.