What I'm trying to achieve here is to submit multiple jobs to Gearman and print the results returned by the workers once they are done with processing the job.
I've read through the examples on:
https://pythonhosted.org/gearman/1to2.html#client-multiple-tasks
https://pythonhosted.org/gearman/client.html
I then tried to implement the check_request_status in the following way:
list_of_jobs = []
for i in xrange(1,4,1):
list_of_jobs.extend(dict(task='run_task', data=str(i)))
completed_requests = gm_client.wait_until_jobs_completed(submitted_requests,
poll_timeout=30.0)
for job_request in completed_requests:
if job_request.complete:
print job_request.result
elif job_request.timed_out:
print "Job %s timed out!" % job_request.unique
elif job_request.state == JOB_UNKNOWN:
print "Job %s connection failed!" % job_request.unique
I'm getting the following error which I can't seem to figure out even after much Google-fu, searching through the Gearman Google Groups and poring through other people's implementation of Gearman:
Traceback (most recent call last):
File "supervisor.py", line 16, in <module>
completed_requests = gm_client.submit_multiple_jobs(list_of_jobs)
File "/usr/local/lib/python2.7/dist-packages/gearman/client.py", line 48, in submit_multiple_jobs
requests_to_submit = [self._create_request_from_dictionary(job_info, background=background, max_retries=max_retries) for job_info in jobs_to_submit]
File "/usr/local/lib/python2.7/dist-packages/gearman/client.py", line 169, in _create_request_from_dictionary
job_unique = job_info.get('unique')
AttributeError: 'str' object has no attribute 'get'
Does anyone know what's going on here?
Turns out the error is a simple one. In the for-loop above, simply use the append function instead of extend for adding a dict to the list_of_jobs array:
list_of_jobs.append(dict(task='run_task', data=str(i)))
Related
This error is not seen every time. When the application runs multiple times we have seen this issue.
Can one help me regarding this?
sql_query = 'SELECT * FROM hourly_table where mac=? and name=? and processedhour >= ? and processedhour < ? ;'
Traceback (most recent call last):
File "/opt/Extraction/main/CassandraOps.py", line 1229, in selectConcurrent
select_statement = session.prepare(sql_query)
File "cassandra/cluster.py", line 2496, in cassandra.cluster.Session.prepare (cassandra/cluster.c:43515)
raise
File "cassandra/cluster.py", line 2493, in cassandra.cluster.Session.prepare (cassandra/cluster.c:43365)
query_id, bind_metadata, pk_indexes, result_metadata, result_metadata_id = future.result()
File "cassandra/cluster.py", line 4198, in cassandra.cluster.ResponseFuture.result (cassandra/cluster.c:80578)
raise self._final_exception
AttributeError: 'PrepareMessage' object has no attribute 'consistency_level'
Expected Query:
select called_station_id,unixtimestamp,acct_session_id,nas_port_id,session_mac,acct_input_octets,acct_output_octets from cdr_sessionterminated where called_station_id = '421888800011' and unixtimestamp>='2021-07-28 11:00:00Z'and unixtimestamp < '2021-07-28 12:00:00Z';
Code:
`
sql_query = "SELECT {} FROM {} where called_station_id = ? and unixtimestamp >= '{}Z' and unixtimestamp < '{}Z' ;".format(columns, table, timeUnitFrom, timeUnitTo)
select_statement = session.prepare(sql_query)
statements_and_params = []
for row in queryFrame.itertuples(index=False):
row = list(row)
params = ()
for column in range(0, len(columnList)):
params+=(row[column],)
statements_and_params.append((select_statement, params))
results = execute_concurrent(session, statements_and_params,raise_on_first_error=False, concurrency=10)
`
Error:
'PrepareMessage' object has no attribute 'consistency_level'
Note:
I'm not getting this error always. it's not even consistent for the past 24 hours I got this error twice.
This looks like a Python coding issue and not specific to Cassandra.
I would suggest that you review your code particularly where you are creating prepared statements and check for errors. For example, it's likely that sections of the code are not indented correctly leading to the error you posted. Cheers!
I have two questions about Django tracebacks.
First, if I have a traceback like the following, I can see that the error is occurring in the 'review_new_profile' view as shown in the second traceback error
below. Is it possible to tell Django to also display the line number in that view where the error occurred?
/srv/http/example.com/venvs/725be8a8537aeef8021231ba68de3184bbd547b1/local/lib/python2.7/site-packages/django/core/handlers/base.py in get_response
1. response = wrapped_callback(request, *callback_args, **callback_kwargs)
/srv/http/example.com/repo/profile/views.py in review_new_profile
1. user = User.objects.get(id=request.session['uid'])
/srv/http/example.com/venvs/725be8a8537aeef8021231ba68de3184bbd547b1/local/lib/python2.7/site-packages/django/contrib/sessions/backends/base.py in __getitem__
1. return self._session[key]
What I'd like to see is something like the following where '540' is the line number in the views.py:
/srv/http/example.com/repo/profile/views.py in review_new_profile (540)
1. user = User.objects.get(id=request.session['uid'])
Second, am I correct in in thinking that the error actually occurred at the bottom-most line? So an interpretation of this trace is that Django was trying to generate a response to a request but when it tried to get a user instance for a given uid, it couldn't find the uid in the session? In other words, the error actually occurred on the last line in the traceback, where Django was trying to get the uid (key) from the session?
For your first question, Django does show the line. In your case, those 1s are the line numbers. The real question is why they are all 1s.
Second question: you are correct. The exception occurred on the last line.
Basically, when you did request.session['uid'], it tried to call that session object's __getitem__ method. Somewhere inside that __getitem__, there is this line:
return self._session[key]
This is what gave the error.
Running the below script works for 60% of the entries from the MasterGroupList however suddenly fails with the below error. although my questions seem to be poor ou guys have been able to help me before. Any idea how I can avoid getting this error? or what is trhoughing off the script? The masterGroupList looks like:
Groups Pulled from AD
SET00 POWERUSER
SET00 USERS
SEF00 CREATORS
SEF00 USERS
...another 300 entries...
Error:
Traceback (most recent call last):
File "C:\Users\ks185278\OneDrive - NCR Corporation\Active Directory Access Scr
ipt\test.py", line 44, in <module>
print group.member
File "C:\Python27\lib\site-packages\active_directory.py", line 805, in __getat
tr__
raise AttributeError
AttributeError
Code:
from active_directory import *
import os
file = open("C:\Users\NAME\Active Directory Access Script\MasterGroupList.txt", "r")
fileAsList = file.readlines()
indexOfTitle = fileAsList.index("Groups Pulled from AD\n")
i = indexOfTitle + 1
while i <= len(fileAsList):
fileLocation = 'C:\\AD Access\\%s\\%s.txt' % (fileAsList[i][:5], fileAsList[i][:fileAsList[i].find("\n")])
#Creates the dir if it does not exist already
if not os.path.isdir(os.path.dirname(fileLocation)):
os.makedirs(os.path.dirname(fileLocation))
fileGroup = open(fileLocation, "w+")
#writes group members to the open file
group = find_group(fileAsList[i][:fileAsList[i].find("\n")])
print group.member
for group_member in group.member: #this is line 44
fileGroup.write(group_member.cn + "\n")
fileGroup.close()
i+=1
Disclaimer: I don't know python, but I know Active Directory fairly well.
If it's failing on this:
for group_member in group.member:
It could possibly mean that the group has no members.
Depending on how phython handles this, it could also mean that the group has only one member and group.member is a plain string rather than an array.
What does print group.member show?
The source code of active_directory.py is here: https://github.com/tjguk/active_directory/blob/master/active_directory.py
These are the relevant lines:
if name not in self._delegate_map:
try:
attr = getattr(self.com_object, name)
except AttributeError:
try:
attr = self.com_object.Get(name)
except:
raise AttributeError
So it looks like it just can't find the attribute you're looking up, which in this case looks like the 'member' attribute.
I'm trying to run a celery (3.1.17) task that executes further tasks in a group but I always run into errors. This is how I set up the code:
from celery import task, group
#task
def daily_emails():
[...]
all_tasks = []
for chunk in range(0, users.count(), 1000):
some_users = users[chunk:chunk+1000]
all_tasks.append(write_email_bunch.subtask(some_users, execnum))
job = group(all_tasks)
# result = job.apply_async()
# job.get()
result = job.delay()
print result
results = result.join()
print results
print "done writing email tasks"
count = sum(results)
print count
#task
def write_email_bunch(some_users, execnum):
[...]
return len(some_users) - skipped_email_count
And this is the output:
<GroupResult: 3d766c85-21af-4ed0-90cb-a1ca2d281db1 [69527252-8468-4358-9328-144f727f372b, 6d03d86e-1b69-4f43-832e-bd27c4dfc092, 1d868d1b-b502-4672-9895-430089e9532e]>
Traceback (most recent call last):
File "send_daily_emails.py", line 8, in <module>
daily_emails()
File "/var/www/virtualenvs/nt_dev/local/lib/python2.7/site-packages/celery/app/task.py", line 420, in __call__
return self.run(*args, **kwargs)
File "/var/www/nt_dev/nt/apps/emails/tasks.py", line 124, in daily_emails
results = result.join()
File "/var/www/virtualenvs/nt_dev/local/lib/python2.7/site-packages/celery/result.py", line 642, in join
interval=interval, no_ack=no_ack,
File "/var/www/virtualenvs/nt_dev/local/lib/python2.7/site-packages/celery/result.py", line 870, in get
raise self.result
TypeError: write_email_bunch() argument after ** must be a mapping, not long
So I get a GroupResult but somehow Im unable to join it or further process it.
And when I use write_email_bunch.s(some_users, execnum) I get this exception:
File "/var/www/virtualenvs/nt_dev/local/lib/python2.7/site-packages/celery/result.py", line 870, in get
raise self.result
TypeError: 'tuple' object is not callable
How would I wait for all the Group Tasks to be completed to continue afterwards?
job.get() gives me this exception:
TypeError: get expected at least 1 arguments, got 0
subtask takes a tuple of args, a dict of kwargs and task options so it should be called like this:
all_tasks.append(write_email_bunch.subtask((some_users, execnum)))
note that we are passing it a tuple containing the args
Also you shouldn't wait on a task inside a task - this can cause deadlocks. In this case I reckon daily_emails does not need to be a celery task - it can be a regular function that creates a canvas object and runs apply async.
def daily_emails():
all_tasks = []
for chunk in range(0, users.count(), 1000):
some_users = users[chunk:chunk+1000]
all_tasks.append(write_email_bunch.subtask(some_users, execnum))
job = group(all_tasks)
result = job.apply_async()
return result.id
In addition to the other answer you could be using chunks here:
http://docs.celeryproject.org/en/latest/userguide/canvas.html#chunks
#app.task
def daily_emails():
return write_email.chunks(users, 1000).delay()
#task
def write_email(user):
[...]
It may be beneficial to do it manually if getting several objects at once
from the db is important. You should also consider that the model objects will be serialized here, to avoid that you can send the pk only and refetch the model in the task, or send the fields that you care about (like email address or whatever is required to send that email to the user).
I have just installed django-cnotes
But it wont work.
It just throws up this error
Traceback (most recent call last):
File "/Library/Python/2.5/site-packages/django/core/servers/basehttp.py", line 279, in run
self.result = application(self.environ, self.start_response)
File "/Library/Python/2.5/site-packages/django/core/servers/basehttp.py", line 651, in __call__
return self.application(environ, start_response)
File "/Library/Python/2.5/site-packages/django/core/handlers/wsgi.py", line 245, in __call__
response = middleware_method(request, response)
File "/Library/Python/2.5/site-packages/django_cnote-0.3.4-py2.5.egg/cnotes/middleware.py", line 47, in process_response
signed_data = self.sign('cnotes', base64.urlsafe_b64encode(Pickle.dumps(cnotes.cnotes)))
PicklingError: Can't pickle <class 'django.utils.functional.__proxy__'>: attribute lookup django.utils.functional.__proxy__ failed
And it is not even in the normal django error debug page. What you see above is all there is on the screen.
And I have just used it as described on github, I just dont get it. Any one have an idea for what is causing this?
UPDATE:
Okay, so I have found something, I think.
message = _("You have successfully altered ")
message += edituser.username
cnotes.add(message)
message2 = _("You may now close ")
cnotes.add(message2)
This will cause the error. So I thought "Okay, I can only call it once per view" That would have been stupid and it was indeed not the cause.
The following code will produce no error
message = _("You have successfully altered ")
message += edituser.username
cnotes.add(message)
message2 = '_("You may now close ")'
cnotes.add(message2)
But is not because of the translation it uses that fine just 2 lines above, but it has to be something with doing another translation or something. Im lost.
It appears as though pickle is receiving an object of type django.utils.functional.__proxy__. This means either your input is weird, or there is a bug in cnotes.
If there is something wrong with your input to cnotes, you should see it if you take a look at the types of your messages (I used the manage.py shell):
>>> message = _("You have successfully altered ")
>>> message += "Bob Knoblick"
>>> type(message)
<type 'unicode'>
>>> message2 = _("You may now close ")
>>> type(message2)
<type 'unicode'>
>>>
If your types come back as anything other than unicode or str, I'd dig into your code and figure out where that other type is coming from, or ensure that it can be pickled.
If there is something wrong within cnotes, you should get the same error doing this:
cnotes.add(u'Foo')
cnotes.add(u'Bar')
cnotes.add(u'Baz')
Per the original author:
The translated string, _("You may now close ") was not ending up as a unicode string. One can use this to force unicode before sending to cnotes:
message2 = unicode(_("You may now close "))