Python: Urllib2 | [Errno 54] Connection reset by peer - python-2.7

I'm calling a list of urls from the same domain and returning a snip of their html for a few thousand domains but am getting this error about 1,000 rows or so in.
Is there anything I can do to avoid this error?
Does it make sense to create a wait step after every row? every few hundred rows?
Is there a better way to get around this?
File "/Users.../ap.py", line 144, in <module> simpleProg()
File "/Users.../ap.py", line 21, in simpleProg()
File "/Users.../ap.py", line 57, in first_step()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 400, in open
response = self._open(req, data)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 418, in _open
'_open', req)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 378, in _call_chain
result = func(*args)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1207, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1180, in do_open
r = h.getresponse(buffering=True)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1030, in getresponse
response.begin()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 407, in begin
version, status, reason = self._read_status()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 365, in _read_status
line = self.fp.readline()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 447, in readline
data = self._sock.recv(self._rbufsize)
socket.error: [Errno 54] Connection reset by peer

Related

how to fix error with ssl certificate in aws lambda

I am using this guide on the AWS deeplens. I am experimenting with the code and at one point it stopped working for the second standalone lambda function. At some point, the code started getting the exception and I only recently noticed it.
Here is the code in question:
def sendMessageToIoTTopic(iotMessage):
# this is a default topic if its not specified in the lambda envirionment
topicName = "worker-safety"
if "iot_topic" in os.environ:
topicName = os.environ['iot_topic']
print("Send message to topic: " + topicName + " before")
iotClient = boto3.client('iot-data', region_name='us-east-1')
response = iotClient.publish(
topic=topicName,
qos=1,
payload=json.dumps(iotMessage)
)
print("Send message to topic: " + topicName + "after")
print("using this topic **{}** response is {}".format(topicName, response))
The guide says to modify the default timeout from 1 second to 3 seconds. Here is the log for this code with a 3 second timeout.
2021-01-31T14:28:35.579-05:00 Before we send to iot MQTT?
2021-01-31T14:28:35.579-05:00 Send message to topic: worker-safety-demo-cloud before
2021-01-31T14:28:36.202-05:00 END RequestId: 89794691-a149-4480-b0bd-fd1a69522774
2021-01-31T14:28:36.202-05:00 REPORT RequestId: 89794691-a149-4480-b0bd-fd1a69522774 Duration: 3003.48 ms Billed Duration: 3000 ms Memory Size: 128 MB Max Memory Used: 35 MB
2021-01-31T14:28:36.202-05:00 2021-01-31T19:28:36.201Z 89794691-a149-4480-b0bd-fd1a69522774 Task timed out after 3.00 seconds
2021-01-31T14:28:38.016-05:00 START RequestId: 71562d34-28d6-471a-8a43-b1e0bab1d621 Version: $LATEST
Adjusting the timeout for the lambda function from 3 seconds to 15 seconds allows us to see the timeout error in detail. It shows that it is an SSL error.
This is the resultant log with more info:
2021-01-31T14:36:08.775-05:00 Before we send to iot MQTT?
2021-01-31T14:36:08.775-05:00 Send message to topic: worker-safety-demo-cloud before
2021-01-31T14:36:17.295-05:00 [ERROR] SSLError: SSL validation failed for https://data.iot.us-east-1.amazonaws.com/topics/worker-safety-demo-cloud?qos=1 [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091) Traceback (most recent call last): File "/var/task/lambda_function.py", line 177, in lambda_handler sendMessageToIoTTopic(iotMessage) File "/var/task/lambda_function.py", line 119, in sendMessageToIoTTopic payload=json.dumps(iotMessage) File "/var/runtime/botocore/client.py", line 357, in _api_call return self._make_api_call(operation_name, kwargs) File "/var/runtime/botocore/client.py", line 663, in _make_api_call operation_model, request_dict, request_context) File "/var/runtime/botocore/client.py", line 682, in _make_request return self._endpoint.make_request(operation_model, request_dict) File "/var/runtime/botocore/endpoint.py", line 102, in make_request return self._send_request(request_dict, operation_model) File "/var/runtime/botocore/endpoint.py", line 137, in _send_request success_response, exception): File "/var/runtime/botocore/endpoint.py", line 256, in _needs_retry caught_exception=caught_exception, request_dict=request_dict) File "/var/runtime/botocore/hooks.py", line 356, in emit return self._emitter.emit(aliased_event_name, **kwargs) File "/var/runtime/botocore/hooks.py", line 228, in emit return self._emit(event_name, kwargs) File "/var/runtime/botocore/hooks.py", line 211, in _emit response = handler(**kwargs) File "/var/runtime/botocore/retryhandler.py", line 183, in __call__ if self._checker(attempts, response, caught_exception): File "/var/runtime/botocore/retryhandler.py", line 251, in __call__ caught_exception) File "/var/runtime/botocore/retryhandler.py", line 277, in _should_retry return self._checker(attempt_number, response, caught_exception) File "/var/runtime/botocore/retryhandler.py", line 317, in __call__ caught_exception) File "/var/runtime/botocore/retryhandler.py", line 223, in __call__ attempt_number, caught_exception) File "/var/runtime/botocore/retryhandler.py", line 359, in _check_caught_exception raise caught_exception File "/var/runtime/botocore/endpoint.py", line 200, in _do_get_response http_response = self._send(request) File "/var/runtime/botocore/endpoint.py", line 269, in _send return self.http_session.send(request) File "/var/runtime/botocore/httpsession.py", line 281, in send raise SSLError(endpoint_url=request.url, error=e)
2021-01-31T14:36:17.296-05:00 END RequestId: 0da11ea2-c9d3-42ce-a8d5-cd3e709bf42e
The SSL error in detail is this:
[ERROR] SSLError: SSL validation failed for https://data.iot.us-east-1.amazonaws.com/topics/worker-safety-demo-cloud?qos=1 [SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)Traceback (most recent call last):  
File "/var/task/lambda_function.py", line 177, in lambda_handler    sendMessageToIoTTopic(iotMessage)  
File "/var/task/lambda_function.py", line 119, in sendMessageToIoTTopic    payload=json.dumps(iotMessage)  
File "/var/runtime/botocore/client.py", line 357, in _api_call    return self._make_api_call(operation_name, kwargs)  
File "/var/runtime/botocore/client.py", line 663, in _make_api_call    operation_model, request_dict, request_context)  
File "/var/runtime/botocore/client.py", line 682, in _make_request    return self._endpoint.make_request(operation_model, request_dict)  
File "/var/runtime/botocore/endpoint.py", line 102, in make_request    return self._send_request(request_dict, operation_model)  
File "/var/runtime/botocore/endpoint.py", line 137, in _send_request    success_response, exception):  
File "/var/runtime/botocore/endpoint.py", line 256, in _needs_retry    caught_exception=caught_exception, request_dict=request_dict)  
File "/var/runtime/botocore/hooks.py", line 356, in emit    return self._emitter.emit(aliased_event_name, **kwargs)  
File "/var/runtime/botocore/hooks.py", line 228, in emit    return self._emit(event_name, kwargs)  
File "/var/runtime/botocore/hooks.py", line 211, in _emit    response = handler(**kwargs)  File "/var/runtime/botocore/retryhandler.py", line 183, in __call__    if self._checker(attempts, response, caught_exception):  
File "/var/runtime/botocore/retryhandler.py", line 251, in __call__    caught_exception)  File "/var/runtime/botocore/retryhandler.py", line 277, in _should_retry    return self._checker(attempt_number, response, caught_exception)  
File "/var/runtime/botocore/retryhandler.py", line 317, in __call__    caught_exception)  File "/var/runtime/botocore/retryhandler.py", line 223, in __call__    attempt_number, caught_exception)  
File "/var/runtime/botocore/retryhandler.py", line 359, in _check_caught_exception    raise caught_exception  
File "/var/runtime/botocore/endpoint.py", line 200, in _do_get_response    http_response = self._send(request)  
File "/var/runtime/botocore/endpoint.py", line 269, in _send    return self.http_session.send(request) 
 File "/var/runtime/botocore/httpsession.py", line 281, in send    raise SSLError(endpoint_url=request.url, error=e)

Temporary failure in name resolution channel 2 django

I have implemented channel2 in my django project , after some time i start it i have received this error on my server , i have apache on ubuntu for my code.
2019-02-12 10:30:51,863 ERROR Exception inside application: [Errno -3] Temporary failure in name resolution
File "/var/www/RPG/python/lib/python3.6/site-packages/channels/consumer.py", line 59, in _call_
[receive, self.channel_receive], self.dispatch
File "/var/www/RPG/python/lib/python3.6/site-packages/channels/utils.py", line 59, in await_many_dispatch
await task
File "/var/www/RPG/python/lib/python3.6/site-packages/channels/utils.py", line 51, in await_many_dispatch
result = task.result()
File "/var/www/RPG/python/lib/python3.6/site-packages/channels_redis/core.py", line 429, in receive
real_channel
File "/var/www/RPG/python/lib/python3.6/site-packages/channels_redis/core.py", line 484, in receive_single
index, channel_key, timeout=self.brpop_timeout
File "/var/www/RPG/python/lib/python3.6/site-packages/channels_redis/core.py", line 324, in _brpop_with_clean
async with self.connection(index) as connection:
File "/var/www/RPG/python/lib/python3.6/site-packages/channels_redis/core.py", line 820, in _aenter_
self.conn = await self.pool.pop()
File "/var/www/RPG/python/lib/python3.6/site-packages/channels_redis/core.py", line 70, in pop
conns.append(await aioredis.create_redis(**self.host, loop=loop))
File "/var/www/RPG/python/lib/python3.6/site-packages/aioredis/commands/__init__.py", line 178, in create_redis
loop=loop)
File "/var/www/RPG/python/lib/python3.6/site-packages/aioredis/connection.py", line 108, in create_connection
timeout, loop=loop)
File "/usr/lib/python3.6/asyncio/tasks.py", line 339, in wait_for
return (yield from fut)
File "/var/www/RPG/python/lib/python3.6/site-packages/aioredis/stream.py", line 19, in open_connection
lambda: protocol, host, port, **kwds)
File "/usr/lib/python3.6/asyncio/base_events.py", line 739, in create_connection
infos = f1.result()
File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/lib/python3.6/socket.py", line 745, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
[Errno -3] Temporary failure in name resolution
Here i attach a image which solve the problem but i need to find better solution.I don't think it's a properway to do this.

Sentry error - Temporary failure in name resolution

I got an error on my server.
DEBUG 2019-01-20 08:38:01,498 base 32461 140474717611392 Sending message of length 7076 to https://sentry.theinvaders.pro/api/30/store/
ERROR 2019-01-20 08:38:01,505 base 32461 140474190894848 Sentry responded with an error: <urlopen error [Errno -3] Temporary failure in name resolution> (url: https://sentry.theinvaders.pro/api/30/store/)
Traceback (most recent call last):
File "/usr/lib/python3.6/urllib/request.py", line 1318, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/usr/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/usr/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/var/www/mass/var/venv/lib/python3.6/site-packages/raven/utils/http.py", line 31, in connect
timeout=self.timeout,
File "/usr/lib/python3.6/socket.py", line 704, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
File "/usr/lib/python3.6/socket.py", line 745, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution
>During handling of the above exception, another exception occurred:
>Traceback (most recent call last):
File "/var/www/mass/var/venv/lib/python3.6/site-packages/raven/transport/threaded.py", line 165, in send_sync
super(ThreadedHTTPTransport, self).send(url, data, headers)
File "/var/www/mass/var/venv/lib/python3.6/site-packages/raven/transport/http.py", line 43, in send
ca_certs=self.ca_certs,
File "/var/www/mass/var/venv/lib/python3.6/site-packages/raven/utils/http.py", line 66, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.6/urllib/request.py", line 526, in open
response = self._open(req, data)
File "/usr/lib/python3.6/urllib/request.py", line 544, in _open
'_open', req)
File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/var/www/mass/var/venv/lib/python3.6/site-packages/raven/utils/http.py", line 46, in https_open
return self.do_open(ValidHTTPSConnection, req)
File "/usr/lib/python3.6/urllib/request.py", line 1320, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno -3] Temporary failure in name resolution>
ERROR 2019-01-20 08:38:01,509 base 32461 140474190894848 ['OperationalError: could not translate host name "localhost" to address: Temporary failure in name resolution\n', ' File "django/core/handlers/exception.py", line 34, in inner', ' File "django/core/handlers/base.py", line 126, in _get_response', ' File "django/core/handlers/base.py", line 124, in _get_response', ' File "django/views/decorators/csrf.py", line 54, in wrapped_view', ' File "django/views/generic/base.py", line 68, in view', ' File "rest_framework/views.py", line 483, in dispatch', ' File "rest_framework/views.py", line 443, in handle_exception', ' File "rest_framework/views.py", line 480, in dispatch', ' File "rest_framework/decorators.py", line 53, in handler', ' File "mass/api/version1/candidate/views.py", line 66, in search_candidate', ' File "django/db/transaction.py", line 147, in __enter__', ' File "django/db/backends/base/base.py", line 378, in get_autocommit', ' File "django/db/backends/base/base.py", line 216, in ensure_connection', ' File "django/db/utils.py", line 89, in __exit__', ' File "django/db/backends/base/base.py", line 216, in ensure_connection', ' File "django/db/backends/base/base.py", line 194, in connect', ' File "django/db/backends/postgresql/base.py", line 178, in get_new_connection', ' File "psycopg2/__init__.py", line 130, in connect']
This lasted for 2 hours. Then the server rebooted. And everything began to work normally.
Please tell me what could be the cause of this.
The hosting provider had an error a DNS server.

python-selenium on aws lambda throws error when switch_back_to.default_content and try to select element

python-selenium using phantomjs used to open an iFrame inside the page, then after switching back to the default content(driver.switch_to.default_content()) does not work.
The following is a pseudo code snippet to explain the scenario.
frame = WebDriverWait(driver, wait_time).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR, "iframe.class_name")))
......
driver.switch_to.default_content()
print('Switched back to default content')
# Error occurs when line below is interpreted
driver.find_element_by_css_selector("div[title^='Test %s']" %(array_of_content[1]))
Following error occurs:-
Traceback (most recent call last):
File "/var/task/main.py", line 217, in test_method
driver.find_element_by_css_selector("div[title^='Test %s']" %(array_of_content[1]))
File "/var/task/selenium/webdriver/support/wait.py", line 71, in until
value = method(self._driver)
File "/var/task/selenium/webdriver/support/expected_conditions.py", line 213, in __call__
element = visibility_of_element_located(self.locator)(driver)
File "/var/task/selenium/webdriver/support/expected_conditions.py", line 78, in __call__
return _element_if_visible(_find_element(driver, self.locator))
File "/var/task/selenium/webdriver/support/expected_conditions.py", line 307, in _find_element
return driver.find_element(*by)
File "/var/task/selenium/webdriver/remote/webdriver.py", line 752, in find_element
'value': value})['value']
File "/var/task/selenium/webdriver/remote/webdriver.py", line 234, in execute
response = self.command_executor.execute(driver_command, params)
File "/var/task/selenium/webdriver/remote/remote_connection.py", line 401, in execute
return self._request(command_info[0], url, body=data)
File "/var/task/selenium/webdriver/remote/remote_connection.py", line 471, in _request
resp = opener.open(request, timeout=self._timeout)
File "/usr/lib64/python2.7/urllib2.py", line 431, in open
response = self._open(req, data)
File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
'_open', req)
File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
result = func(*args)
File "/usr/lib64/python2.7/urllib2.py", line 1229, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib64/python2.7/urllib2.py", line 1202, in do_open
r = h.getresponse(buffering=True)
File "/var/task/raven/breadcrumbs.py", line 328, in getresponse
rv = real_getresponse(self, *args, **kwargs)
File "/usr/lib64/python2.7/httplib.py", line 1132, in getresponse
response.begin()
File "/usr/lib64/python2.7/httplib.py", line 453, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python2.7/httplib.py", line 417, in _read_status
raise BadStatusLine(line)
BadStatusLine: ''
Any ideas, what is happening.
As I am able to see the same error, even if I write driver.save_screenshot('test.png').
But it is not throwing any error on local machines, also tried to use AMI instance to replicate if something is there in concern Amazon Linux.

Unable to create superuser on google app engine

I am working on a django project and have used django nonrel to deploy on google app engine.
However when I try to create a super user using the command "manage.py remote create superuser", I get the following error,
Please let me know if anyone knows the reason for this
Username: newuser
Traceback (most recent call last):
File "C:\SVN GAE\development\legaltracker\manage.py", line 26, in <module>
execute_manager(settings)
File "C:\SVN GAE\development\legaltracker\django\core\management\__init__.py",
line 438, in execute_manager
utility.execute()
File "C:\SVN GAE\development\legaltracker\django\core\management\__init__.py",
line 379, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "C:\SVN GAE\development\legaltracker\djangoappengine\management\commands\
remote.py", line 15, in run_from_argv
execute_from_command_line(argv)
File "C:\SVN GAE\development\legaltracker\django\core\management\__init__.py",
line 429, in execute_from_command_line
utility.execute()
File "C:\SVN GAE\development\legaltracker\django\core\management\__init__.py",
line 379, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "C:\SVN GAE\development\legaltracker\django\core\management\base.py", lin
e 191, in run_from_argv
self.execute(*args, **options.__dict__)
File "C:\SVN GAE\development\legaltracker\django\core\management\base.py", lin
e 218, in execute
output = self.handle(*args, **options)
File "C:\SVN GAE\development\legaltracker\django\contrib\auth\management\comma
nds\createsuperuser.py", line 97, in handle
User.objects.get(username=username)
File "C:\SVN GAE\development\legaltracker\django\db\models\manager.py", line 1
32, in get
return self.get_query_set().get(*args, **kwargs)
File "C:\SVN GAE\development\legaltracker\django\db\models\query.py", line 336
, in get
num = len(clone)
File "C:\SVN GAE\development\legaltracker\django\db\models\query.py", line 81,
in __len__
self._result_cache = list(self.iterator())
File "C:\SVN GAE\development\legaltracker\django\db\models\query.py", line 269
, in iterator
for row in compiler.results_iter():
File "C:\SVN GAE\development\legaltracker\djangotoolbox\db\basecompiler.py", l
ine 219, in results_iter
for entity in self.build_query(fields).fetch(low_mark, high_mark):
File "C:\SVN GAE\development\legaltracker\djangoappengine\db\compiler.py", lin
e 95, in fetch
results = query.Run(**kw)
File "C:\Program Files\Google\google_appengine\google\appengine\api\datastore.
py", line 1138, in Run
return self._Run(**kwargs)
File "C:\Program Files\Google\google_appengine\google\appengine\api\datastore.
py", line 1167, in _Run
datastore_pb.QueryResult(), rpc)
File "C:\Program Files\Google\google_appengine\google\appengine\api\datastore.
py", line 186, in _MakeSyncCall
rpc.check_success()
File "C:\Program Files\Google\google_appengine\google\appengine\api\apiproxy_s
tub_map.py", line 474, in check_success
self.__rpc.CheckSuccess()
File "C:\Program Files\Google\google_appengine\google\appengine\api\apiproxy_r
pc.py", line 149, in _WaitImpl
self.request, self.response)
File "C:\Program Files\Google\google_appengine\google\appengine\ext\remote_api
\remote_api_stub.py", line 223, in MakeSyncCall
handler(request, response)
File "C:\Program Files\Google\google_appengine\google\appengine\ext\remote_api
\remote_api_stub.py", line 232, in _Dynamic_RunQuery
'datastore_v3', 'RunQuery', query, query_result)
File "C:\Program Files\Google\google_appengine\google\appengine\ext\remote_api
\remote_api_stub.py", line 155, in MakeSyncCall
self._MakeRealSyncCall(service, call, request, response)
File "C:\Program Files\Google\google_appengine\google\appengine\ext\remote_api
\remote_api_stub.py", line 167, in _MakeRealSyncCall
encoded_response = self._server.Send(self._path, encoded_request)
File "C:\Program Files\Google\google_appengine\google\appengine\tools\appengin
e_rpc.py", line 346, in Send
f = self.opener.open(req)
File "C:\Python25\lib\urllib2.py", line 387, in open
response = meth(req, response)
File "C:\Python25\lib\urllib2.py", line 498, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Python25\lib\urllib2.py", line 425, in error
return self._call_chain(*args)
File "C:\Python25\lib\urllib2.py", line 360, in _call_chain
result = func(*args)
File "C:\Python25\lib\urllib2.py", line 506, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 302: Found
Im not very familiar with django nonrel but HTTP code 302 is not really an error but indicates that the client should try under a different URL. Unfortunately the error message does not show which URL is redirected to. I see this happens on the local development server - I suggest you check the logfile of the development webserver.