I'm using celery 4.3.0,And use it to generate a image which maybe 10M+,Then I got an error like below:
Pool callback raised exception: MemoryError('Process got: ')
Traceback (most recent call last):
File "*/lib/python3.7/site-packages/billiard/pool.py", line 1750, in safe_apply_callback
fun(*args, **kwargs)
File "*/lib/python3.7/site-packages/celery/worker/request.py", line 564, in on_success
return self.on_failure(retval, return_ok=True)
File "*/lib/python3.7/site-packages/celery/worker/request.py", line 351, in on_failure
raise MemoryError('Process got: %s' % (exc_info.exception,))
My server has 20G+ memory left when running this task.And I had test some small images which work well.Do I need set some config to prevent this?
Related
I ran the following python sample code on this page (you need a private key to run this): https://cloud.google.com/speech-to-text/docs/quickstart-client-libraries
# Imports the Google Cloud client library
from google.cloud import speech
import os
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = 'key.json'
# Instantiates a client
client = speech.SpeechClient()
# The name of the audio file to transcribe
gcs_uri = "gs://cloud-samples-data/speech/brooklyn_bridge.raw"
audio = speech.RecognitionAudio(uri=gcs_uri)
config = speech.RecognitionConfig(
encoding=speech.RecognitionConfig.AudioEncoding.LINEAR16,
sample_rate_hertz=16000,
language_code="en-US",
)
# Detects speech in the audio file
response = client.recognize(config=config, audio=audio)
for result in response.results:
print("Transcript: {}".format(result.alternatives[0].transcript))
This results in the following error:
Traceback (most recent call last):
File "C:\Users\98274\anaconda3\envs\carla\lib\site-packages\google\api_core\grpc_helpers.py", line 57, in error_remapped_callable
return callable_(*args, **kwargs)
File "C:\Users\98274\anaconda3\envs\carla\lib\site-packages\grpc\_channel.py", line 826, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "C:\Users\98274\anaconda3\envs\carla\lib\site-packages\grpc\_channel.py", line 729, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses"
debug_error_string = "{"created":"#1614347442.192000000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":4090,"referenced_errors":[{"created":"#1614347442.192000000","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":394,"grpc_status":14}]}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\98274\anaconda3\envs\carla\lib\site-packages\google\api_core\retry.py", line 184, in retry_target
return target()
File "C:\Users\98274\anaconda3\envs\carla\lib\site-packages\google\api_core\grpc_helpers.py", line 59, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.ServiceUnavailable: 503 failed to connect to all addresses
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:/Users/98274/Desktop/carla_ppo/test_connection.py", line 22, in <module>
response = client.recognize(config=config, audio=audio)
File "C:\Users\98274\anaconda3\envs\carla\lib\site-packages\google\cloud\speech_v1\services\speech\client.py", line 334, in recognize
response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,)
File "C:\Users\98274\anaconda3\envs\carla\lib\site-packages\google\api_core\gapic_v1\method.py", line 145, in __call__
return wrapped_func(*args, **kwargs)
File "C:\Users\98274\anaconda3\envs\carla\lib\site-packages\google\api_core\retry.py", line 286, in retry_wrapped_func
on_error=on_error,
File "C:\Users\98274\anaconda3\envs\carla\lib\site-packages\google\api_core\retry.py", line 206, in retry_target
last_exc,
File "<string>", line 3, in raise_from
google.api_core.exceptions.RetryError: Deadline of 120.0s exceeded while calling functools.partial(<function _wrap_unary_errors.<locals>.error_remapped_callable at 0x0000026B56606A68>, config {
encoding: LINEAR16
sample_rate_hertz: 16000
language_code: "en-US"
}
audio {
uri: "gs://cloud-samples-data/speech/brooklyn_bridge.raw"
}
, metadata=[('x-goog-api-client', 'gl-python/3.7.9 grpc/1.31.0 gax/1.23.0 gapic/2.0.1')]), last exception: 503 failed to connect to all addresses
What's the problem? Does it related to internet issue because I am in China? How can I solve it?
I am posting the answer as CommunityWiki to further contribuite to the community.
As discussed in the comment section, the code provided in the documentation is running smoothly. However, you have to make sure that you set your credentials using GOOGLE_APPLICATION_CREDENTIALS so your request is authenticated. Also,the client libraries have to be properly installed.
In #antaressgzz case's the request to the API was being blocked by a firewall rule. Thus, it is a good practice to check these rules, changing them if applicable.
I am trying to run the SONOS self test for a music service on Sonos.
After getting the dependencies, and filling out the config file, I try to run the python Sonos selftest, however it runs into an error and I have no clue what the underlying issue might be to get it running:
No handlers could be found for logger "smapi"
Traceback (most recent call last):
File "suite_selftest.py", line 226, in <module>
nightly_mode(parser.config_file)
File "suite_selftest.py", line 51, in nightly_mode
development_mode(config_file)
File "suite_selftest.py", line 186, in development_mode
fixtures.append(getlastupdate.PollingIntervalTest(suite.client, suite.smapiservice))
File "/Users/thomas/Desktop/PythonSelfTest/smapi/content_workflow/getlastupdate.py", line 20, in __init__
self.poll_interval = self.smapiservice.get_polling_interval()
File "../../sonos-1.1.0.dev_r300235-py2.7.egg/sonos/smapi/smapiservice.py", line 465, in get_polling_interval
File "/usr/local/Cellar/python#2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ConfigParser.py", line 362, in getfloat
return self._get(section, float, option)
File "/usr/local/Cellar/python#2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ConfigParser.py", line 356, in _get
return conv(self.get(section, option))
ValueError: could not convert string to float:
Found the fix already, forgot to add the Polling Interval in the config file...
I have the following celery task:
#task
def get_users_facebook_as_profile_icon(user_id, facebook_id):
logger.info('Grabbing users facebook picture')
url = "http://graph.facebook.com/%s/picture?type=large" % facebook_id
import requests
response = requests.get(url)
if response.status_code != 200:
raise Exception("Could not get facebook profile picture")
...
I have more after this, but I keep getting the following error:
"AssertionError('PID check failed. RNG must be re-initialized after fork(). Hint: Try Random.atfork()',)"
Task was called with args: (3246, 17500596) kwargs: {}.
The contents of the full traceback was:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 437, in __protected_call__
return self.run(*args, **kwargs)
File "/home/ubuntu/mounzawebsite/mounza/celery_tasks/login_registration.py", line 42, in get_users_facebook_as_profile_icon
hashname = user.generate_picture_name()
File "/home/ubuntu/mounzawebsite/mounza/web/models.py", line 515, in generate_picture_name
return generate_random_name(None)
File "/home/ubuntu/mounzawebsite/mounza/web/models.py", line 40, in generate_random_name
str(random.randint(1, 99982098098908237)) +
File "/usr/lib/python2.7/dist-packages/Crypto/Random/__init__.py", line 41, in get_random_bytes
return _UserFriendlyRNG.get_random_bytes(n)
File "/usr/lib/python2.7/dist-packages/Crypto/Random/_UserFriendlyRNG.py", line 213, in get_random_bytes
return _get_singleton().read(n)
File "/usr/lib/python2.7/dist-packages/Crypto/Random/_UserFriendlyRNG.py", line 163, in read
return _UserFriendlyRNG.read(self, bytes)
File "/usr/lib/python2.7/dist-packages/Crypto/Random/_UserFriendlyRNG.py", line 122, in read
self._check_pid()
File "/usr/lib/python2.7/dist-packages/Crypto/Random/_UserFriendlyRNG.py", line 138, in _check_pid
raise AssertionError("PID check failed. RNG must be re-initialized after fork(). Hint: Try Random.atfork()")
AssertionError: PID check failed. RNG must be re-initialized after fork(). Hint: Try Random.atfork()
I tried digging into this online, not able to find the root cause. but this is the only task where this error occurs. The only difference is that i'm downloading an image from Facebook, but I never see this issue anywhere else, including other tasks where I download images.
The URL works perfectly if I do it through a web browser, but it's only via this task it fails. Is there anything else that could contribute to this??
I have exhausted all attempts in fixing this :(
Here is why:
http://comments.gmane.org/gmane.comp.python.amqp.celery.user/3664
always run teh below:
Crypto.Random.atfork()
When a new worker process is initialized. Done and done.
Out of the blue today I started having this problem.. in trying to run the Python Shell from my .py file. The shell window opens with a blinking cursor. If i type anything and hit enter the following error hits. Python IDLE doesn't work at all and I only get this error.
IDLE internal error in runcode()
Traceback (most recent call last):
File "C:\Python27\lib\idlelib\rpc.py", line 235, in asyncqueue
self.putmessage((seq, request))
File "C:\Python27\lib\idlelib\rpc.py", line 332, in putmessage
n = self.sock.send(s[:BUFSIZE])
error: [Errno 10053] An established connection was aborted by the software in your host machine
Python (command line) works fine, but the IDLE does not. I've tried rebooting windows and system restore with no luck. I've been googling for answers with little success. My firewall has always been disabled. Trying to start IDLE from command line returns the following error.
C:\Python27>python.exe -m idlelib.idle
Failed to load extension 'CallTips'
Traceback (most recent call last):
File "C:\Python27\lib\idlelib\EditorWindow.py", line 1061, in load_standard_ex
tensions
self.load_extension(name)
File "C:\Python27\lib\idlelib\EditorWindow.py", line 1076, in load_extension
cls = getattr(mod, name)
AttributeError: 'module' object has no attribute 'CallTips'
----------------------------------------
Unhandled server exception!
Thread: SockThread
Client Address: ('127.0.0.1', 49552)
Request: <socket._socketobject object at 0x0176BCA8>
Traceback (most recent call last):
File "C:\Python27\lib\SocketServer.py", line 295, in _handle_request_noblock
self.process_request(request, client_address)
File "C:\Python27\lib\SocketServer.py", line 321, in process_request
self.finish_request(request, client_address)
File "C:\Python27\lib\SocketServer.py", line 334, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\Python27\lib\idlelib\rpc.py", line 503, in __init__
SocketServer.BaseRequestHandler.__init__(self, sock, addr, svr)
File "C:\Python27\lib\SocketServer.py", line 649, in __init__
self.handle()
File "C:\Python27\lib\idlelib\run.py", line 276, in handle
executive = Executive(self)
File "C:\Python27\lib\idlelib\run.py", line 315, in __init__
self.calltip = CallTips.CallTips()
AttributeError: 'module' object has no attribute 'CallTips'
Windows -32bit OS. thanks in advance for your help.
This is really weird. But a great question you posted! Nice to read.
Here could be a solution: https://stackoverflow.com/a/3277996/1320237
Why would it be impossible to import the Calltips module from the standart library? Maybe you have a Python 3 idle open? Then they interfere?
Well my impatience got the best of me. I copied my site-packages file to the desktop and uninstalled/reinstalled Python. Pasted the site-packages back into my Python directory and all is right with the world again.
Environment:
Scrapy 0.16.2
Twisted-12.2.0
python 2.7
macosx-10.6
Okey here is my problem:
I try to run
scrapy shell http://aaa.17domn.com/bt9/file.php/MERH77V.html
Error:
[ScrapyHTTPPageGetter,client] Unhandled Error
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-macosx-10.6-intel.egg/twisted/internet/selectreactor.py", line 150, in _doReadOrWrite
why = getattr(selectable, method)()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-macosx-10.6-intel.egg/twisted/internet/tcp.py", line 202, in doRead
return self._dataReceived(data)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-macosx-10.6-intel.egg/twisted/internet/tcp.py", line 208, in _dataReceived
rval = self.protocol.dataReceived(data)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-macosx-10.6-intel.egg/twisted/protocols/basic.py", line 564, in dataReceived
why = self.lineReceived(line)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Scrapy-0.16.2-py2.7.egg/scrapy/core/downloader/webclient.py", line 50, in lineReceived
return HTTPClient.lineReceived(self, line.rstrip())
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-macosx-10.6-intel.egg/twisted/web/http.py", line 450, in lineReceived
self.extractHeader(self._header)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-macosx-10.6-intel.egg/twisted/web/http.py", line 406, in extractHeader
key, val = header.split(':',1)
exceptions.ValueError: need more than 1 value to unpack
I found the solution from https://groups.google.com/forum/#!msg/scrapy-users/xFKo8ggzPxs/VXDl3CZ4V4cJ
They describe this is caused by twisted. Then I patched function extractHeader in /twisted/web/http.py from http://twistedmatrix.com/trac/ticket/2842. Its WORKS
BUT BUT, Hold on NOt yet!!!
I run another web
scrapy shell http://www1.wkdown.info/fs3/file.php/M994ATR.html
Error:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-macosx-10.6-intel.egg/twisted/internet/defer.py", line 551, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Scrapy-0.16.2-py2.7.egg/scrapy/core/downloader/webclient.py", line 122, in _build_response
status = int(self.status)
ValueError: invalid literal for int() with base 10: 'html'
I think something happen on response headers. Scrapy cannot handle it well.
Any idea?
Thank you!