I want to parallelize execution of a for loop on the quadcore processor of my computer's CPU. I am using pp (Python-Parallel) - rather than joblib.Parallel for reasons considered here.
But I am getting an error:
Traceback (most recent call last):
File "batching.py", line 60, in cleave_out_bad_data
job1 = job_server.submit(cleave_out, (data_dir,dirlist,), (endswithdat,))
File "/homes/ad6813/.local/lib/python2.7/site-packages/pp.py", line 459, in submit
sfunc = self.__dumpsfunc((func, ) + depfuncs, modules)
File "/homes/ad6813/.local/lib/python2.7/site-packages/pp.py", line 637, in __dumpsfunc
sources = [self.__get_source(func) for func in funcs]
File "/homes/ad6813/.local/lib/python2.7/site-packages/pp.py", line 704, in __get_source
sourcelines = inspect.getsourcelines(func)[0]
File "/usr/lib/python2.7/inspect.py", line 690, in getsourcelines
lines, lnum = findsource(object)
File "/usr/lib/python2.7/inspect.py", line 529, in findsource
raise IOError('source code not available')
IOError: source code not available
It looks like the reason is a python-2.7 bug.
Has anyone come across this and solved it?
Here is my code:
def clean_dir(data_dir, dirlist):
job_server = pp.Server()
job1 = job_server.submit(clean, (data_dir,dirlist,), (endswith,))
def clean(data_dir, dirlist):
[good_or_bad(file, data_dir) for file in dirlist if endswith(file)]
Inspired by here, the way I fixed a similar problem is to save the code and function in one file like "test.py" and call this file with python, rather than input the function line by line in python shell. I works for me.
Related
I'm trying to use pyglet instead of pygame, 'cause it supports several screens.
this is a sample code that I run:
import pyglet
display = pyglet.canvas.get_display()
screens = display.get_screens()
window = pyglet.window.Window(fullscreen=True, screen=screens[1])
pyglet.app.run()
and I get this error:
Traceback (most recent call last): File
"/home/pi/netcomShopTV/idk.py", line 5, in
window = pyglet.window.Window() File "/usr/local/lib/python2.7/dist-packages/pyglet/init.py", line 359,
in getattr
import(import_name) File "/usr/local/lib/python2.7/dist-packages/pyglet/window/init.py",
line 1890, in
gl._create_shadow_window() File "/usr/local/lib/python2.7/dist-packages/pyglet/gl/init.py", line
209, in _create_shadow_window
_shadow_window = Window(width=1, height=1, visible=False) File "/usr/local/lib/python2.7/dist-packages/pyglet/window/xlib/init.py",
line 171, in init
super(XlibWindow, self).init(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/pyglet/window/init.py",
line 642, in init
self._create() File "/usr/local/lib/python2.7/dist-packages/pyglet/window/xlib/init.py",
line 265, in _create
self.context.set_vsync(self._vsync) # XXX ? File "/usr/local/lib/python2.7/dist-packages/pyglet/gl/xlib.py", line 265,
in set_vsync
warnings.warn(e) TypeError: expected string or buffer
Pyglet Version: 1.4.8
I searched in internet, couldn't find anything to solve this problem.
It seems this bug was introduced with this recent change. You should definitely raise it on pyglet github issue tracker.
Meanwhile, try installing the version prior to 1.4.8. (I though suspect this may just lead to crashing on failed sync as opposed to trying to warn you and then crashing :)).
As #alecxe mentioned, it was a bug. After I opened a ticket on github, I got the solution:
This is an exception for a Raspberry Pi specific issue. It's supposed
to raise a warning, and pass without crashing. If possible, could you
try editing line 265 in
/usr/local/lib/python2.7/dist-packages/pyglet/gl/xlib.py, and
changing:
warnings.warn(e) to warnings.warn(e.message)
I am trying to run the following Python 2.7.10 code on Win7:
from robot import run
run("Login.robot", variable=['USERNAME:testuser#fakemail.com'])
Login.robot is a very simple robot framework test that opens a browser window, loads our log in page, enters in a username and password, then confirms the user has logged in. I have tested it by running it from the command line with pybot and the test suite runs correctly. But trying to run the test suite from this script generates
Traceback (most recent call last):
File "C:\Users\dmdunn\Desktop\test_user_data.py", line 4, in <module>
run("Login.robot", variable=['USERNAME:testuser#fakemail.com'])
File "C:\Python27\lib\site-packages\robot\run.py", line 471, in run
return RobotFramework().execute(*datasources, **options)
File "C:\Python27\lib\site-packages\robot\utils\application.py", line 83, in execute
return self._execute(list(arguments), options)
File "C:\Python27\lib\site-packages\robot\utils\application.py", line 96, in _execute
details, rc=FRAMEWORK_ERROR)
File "C:\Python27\lib\site-packages\robot\utils\application.py", line 110, in _report_error
self._logger.error(message)
File "C:\Python27\lib\site-packages\robot\output\loggerhelper.py", line 59, in error
self.write(msg, 'ERROR')
File "C:\Python27\lib\site-packages\robot\output\loggerhelper.py", line 62, in write
self.message(Message(message, level, html))
File "C:\Python27\lib\site-packages\robot\output\logger.py", line 109, in message
logger.message(msg)
File "C:\Python27\lib\site-packages\robot\output\monitor.py", line 66, in message
self._writer.error(msg.message, msg.level, clear=self._running_test)
File "C:\Python27\lib\site-packages\robot\output\monitor.py", line 142, in error
self._highlight('[ ', level, ' ] ' + message, error=True)
File "C:\Python27\lib\site-packages\robot\output\monitor.py", line 158, in _highlight
self._write(before, newline=False, error=error)
File "C:\Python27\lib\site-packages\robot\output\monitor.py", line 154, in _write
stream.flush()
IOError: [Errno 9] Bad file descriptor
Any suggestions? I've tried using an absolute path to the robot file, single quotes, double quotes, referencing via an open file object, referencing via a closed file object. Thanks!
I discovered it was due to using IDLE, which prevented robot framework from writing to the console. My script runs when started from the Windows command line.
Open a file handle and call robot.run method inside while handle is open.
I hope this will solve your issue.
mydir = os.path.join('c:/','....')
os.makedirs(mydir)
with open('mydir', 'w') as stdout_file:
k = run("Login.robot",
variable=['USERNAME:testuser#fakemail.com'],
stdout=stdout_file)
I have a setup inside a virtual environment:
Django-nonrel-1.6
mongodb-engine
djangotoolbox
Everything works fine, the only problem is during running tests. Every time django tries to flush database after running a test function it throws an error:
Traceback (most recent call last):
File "/home/lehins/.virtualenvs/studentpal/local/lib/python2.7/site-packages/django/test/testcases.py", line 187, in __call__
self._post_teardown()
File "/home/lehins/.virtualenvs/studentpal/local/lib/python2.7/site-packages/django/test/testcases.py", line 796, in _post_teardown
self._fixture_teardown()
File "/home/lehins/.virtualenvs/studentpal/local/lib/python2.7/site-packages/django/test/testcases.py", line 889, in _fixture_teardown
return super(TestCase, self)._fixture_teardown()
File "/home/lehins/.virtualenvs/studentpal/local/lib/python2.7/site-packages/django/test/testcases.py", line 817, in _fixture_teardown
inhibit_post_syncdb=self.available_apps is not None)
File "/home/lehins/.virtualenvs/studentpal/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 159, in call_command
return klass.execute(*args, **defaults)
File "/home/lehins/.virtualenvs/studentpal/local/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
output = self.handle(*args, **options)
File "/home/lehins/.virtualenvs/studentpal/local/lib/python2.7/site-packages/django/core/management/base.py", line 415, in handle
return self.handle_noargs(**options)
File "/home/lehins/.virtualenvs/studentpal/local/lib/python2.7/site-packages/django/core/management/commands/flush.py", line 79, in handle_noargs
six.reraise(CommandError, CommandError(new_msg), sys.exc_info()[2])
File "/home/lehins/.virtualenvs/studentpal/local/lib/python2.7/site-packages/django/core/management/commands/flush.py", line 67, in handle_noargs
savepoint=connection.features.can_rollback_ddl):
File "/home/lehins/.virtualenvs/studentpal/local/lib/python2.7/site-packages/django/db/transaction.py", line 251, in __enter__
"The outermost 'atomic' block cannot use "
CommandError: Database test_dev_db couldn't be flushed. Possible reasons:
* The database isn't running or isn't configured correctly.
* At least one of the expected database tables doesn't exist.
* The SQL was invalid.
Hint: Look at the output of 'django-admin.py sqlflush'. That's the SQL this command wasn't able to run.
The full error: The outermost 'atomic' block cannot use savepoint = False when autocommit is off.
So for each test case, I have a pass or a fail, like it suppose to, but I also get this annoying error.
I did run django-admin.py sqlflush --settings=dev_settings --pythonpath=., which flushed my development database just fine, with no errors.
In a couple test functions I checked a few models pulled from database, and it seems to be flushing and recreating objects just fine, so that's why it is not affecting actual test cases.
I went though the whole traceback and I kind of understand why it happens, but I cannot figure out how to deal with. Any help is appreciated.
Edit
Just tried running tests with Django-nonrel-1.5, there was no problems. It seems like a bug in 1.6 version.
Use SimpleTestCase or a custom TestCase
class CustomTestCase(TestCase):
def _fixture_teardown(self):
for db_name in self._databases_names(include_mirrors=False):
call_command('custom_flush', verbosity=0, interactive=False,
database=db_name, skip_validation=True,
reset_sequences=False,
allow_cascade=self.available_apps is not None,
inhibit_post_syncdb=self.available_apps is not None)
Since the problem is transaction.atomic in command flush, you may have to write your own flush.
I was using Apache Solr for quite some time and only recently started running into some severe issues with it. I'm using it with haystack and a django project. When I do it from manage.py shell i'm getting the below:
>>> from haystack.query import SearchQuerySet
>>> emps = SearchQuerySet().filter(django_ct='web.employer').filter(name__icontains='Mi')[:10]
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/haystack/query.py", line 241, in __getitem__
self._fill_cache(start, bound)
File "/usr/local/lib/python2.7/dist-packages/haystack/query.py", line 140, in _fill_cache
results = self.query.get_results(**kwargs)
File "/usr/local/lib/python2.7/dist-packages/haystack/backends/__init__.py", line 469, in get_results
self.run(**kwargs)
File "/usr/local/lib/python2.7/dist-packages/haystack/backends/solr_backend.py", line 501, in run
results = self.backend.search(final_query, **search_kwargs)
File "/usr/local/lib/python2.7/dist-packages/haystack/backends/__init__.py", line 47, in wrapper
return func(obj, query_string, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/haystack/backends/solr_backend.py", line 202, in search
raw_results = self.conn.search(query_string, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/pysolr.py", line 578, in search
response = self._select(params)
File "/usr/local/lib/python2.7/dist-packages/pysolr.py", line 308, in _select
return self._send_request('get', path)
File "/usr/local/lib/python2.7/dist-packages/pysolr.py", line 293, in _send_request
error_message = self._extract_error(resp)
File "/usr/local/lib/python2.7/dist-packages/pysolr.py", line 372, in _extract_error
reason, full_html = self._scrape_response(resp.headers, resp.content)
File "/usr/local/lib/python2.7/dist-packages/pysolr.py", line 404, in _scrape_response
p_nodes = body_node.cssselect('p')
AttributeError: 'NoneType' object has no attribute 'cssselect'
I tried reinstalling haystack, lxml, cssselect, pysolr and still i'm getting these errors. Is there anything else I can try for this? Thanks for any help!
I also tried reading few other SO questions including this:
XML error object has no attribute 'cssselect'
Seems like the issue is with pysolr. You might find some help here.
I had the same issue persist even after bringing up pysolr and lxml to latest version.
Turned out it was because I was not using haystack generated schema which has a few additional fields compared to the default solr one.
You can confirm if this is the case by looking at your solr logs.
It is an issue with pysolr. It hasn't been fixed till 3.3.0.
The only alternative would be to override the pysolr code and make adjustments for when Solr returns a reponse status!=200.
You can check if the response has body attribute or not and make adjustments according to that.
So i have a django site that is giving me this AmbiguousTimeError. I have a job activates when a product is saved that is given a brief timeout before updating my search index. Looks like an update was made in the Daylight Savings Time hour, and pytz cannot figure out what to do with it.
How can i prevent this from happening the next time the hour shifts for DST?
[2012-11-06 14:22:52,115: ERROR/MainProcess] Unrecoverable error: AmbiguousTimeError(datetime.datetime(2012, 11, 4, 1, 11, 4, 335637),)
Traceback (most recent call last):
File "/usr/local/lib/python2.6/dist-packages/celery/worker/__init__.py", line 353, in start
component.start()
File "/usr/local/lib/python2.6/dist-packages/celery/worker/consumer.py", line 369, in start
self.consume_messages()
File "/usr/local/lib/python2.6/dist-packages/celery/worker/consumer.py", line 842, in consume_messages
self.connection.drain_events(timeout=10.0)
File "/usr/local/lib/python2.6/dist-packages/kombu/connection.py", line 191, in drain_events
return self.transport.drain_events(self.connection, **kwargs)
File "/usr/local/lib/python2.6/dist-packages/kombu/transport/virtual/__init__.py", line 760, in drain_events
self._callbacks[queue](message)
File "/usr/local/lib/python2.6/dist-packages/kombu/transport/virtual/__init__.py", line 465, in _callback
return callback(message)
File "/usr/local/lib/python2.6/dist-packages/kombu/messaging.py", line 485, in _receive_callback
self.receive(decoded, message)
File "/usr/local/lib/python2.6/dist-packages/kombu/messaging.py", line 457, in receive
[callback(body, message) for callback in callbacks]
File "/usr/local/lib/python2.6/dist-packages/celery/worker/consumer.py", line 560, in receive_message
self.strategies[name](message, body, message.ack_log_error)
File "/usr/local/lib/python2.6/dist-packages/celery/worker/strategy.py", line 25, in task_message_handler
delivery_info=message.delivery_info))
File "/usr/local/lib/python2.6/dist-packages/celery/worker/job.py", line 120, in __init__
self.eta = tz_to_local(maybe_iso8601(eta), self.tzlocal, tz)
File "/usr/local/lib/python2.6/dist-packages/celery/utils/timeutils.py", line 52, in to_local
dt = make_aware(dt, orig or self.utc)
File "/usr/local/lib/python2.6/dist-packages/celery/utils/timeutils.py", line 211, in make_aware
return localize(dt, is_dst=None)
File "/usr/local/lib/python2.6/dist-packages/pytz/tzinfo.py", line 349, in localize
raise AmbiguousTimeError(dt)
AmbiguousTimeError: 2012-11-04 01:11:04.335637
EDIT: I fixed it temporarily with this code in celery:
celery/worker/job.py # line 120
try:
self.eta = tz_to_local(maybe_iso8601(eta), self.tzlocal, tz)
except:
self.eta = None
I don't want to have changes in a pip installed app, so i need to fix what i can in my code:
This runs when i save my app:
self.task_cls.apply_async(
args=[action, get_identifier(instance)],
countdown=15
)
I'm assuming that i need to somehow detect if i'm in the ambiguous time and adjust countdown.
I think i'm going to have to clear the tasks to fix this, but how can i prevent this from happening the next time the hour shifts for DST?
It's not clear what you're doing (you haven't shown any code), but basically you need to take account for the way the world works. You can't avoid having ambiguous times when you convert from local time to UTC (or to a different zone's local time) when the time goes back an hour.
Likewise you ought to be aware that there are "gap" or "impossible" times, where a reasonable-sounding local time simply doesn't occur.
I don't know what options Python gives you, but ideally an API should let you resolve ambiguous times however you want - whether that's throwing an error, giving you the earlier occurrence, the later occurrence, or something else.
Apparently, Celery solved this issue:
https://github.com/celery/celery/issues/1061