I'm running a GAE python application with 3 modules, one module run every half an hour and is in charge of fetching a file form an FTP server, processing it and saving it to CloudStorage.
Recently the communication between my application and the FTP server start failing more often, increasingly until it became unusable.
The FTP server is Highwinds, they did some changes recently, asked me some questions like the ping and traceroute from the instance running the code to their server, but I'm not able to provide that given the GAE restrictions.
I was able to overcome some errors by just repeating the same operation multiple times.
I'm including some error traces at the end of this message.
I would appreciate some help troubleshooting this issue.
files = ftp_service.nlst()
File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/ftplib.py", line 509, in nlst
self.retrlines(cmd, files.append)
File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/ftplib.py", line 432, in retrlines
conn = self.transfercmd(cmd)
File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/ftplib.py", line 371, in transfercmd
return self.ntransfercmd(cmd, rest)[0]
File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/ftplib.py", line 330, in ntransfercmd
conn = socket.create_connection((host, port), self.timeout)
File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/socket.py", line 569, in create_connection
raise err
error: [Errno 110] Connection timed out
ftp_service.retrbinary("RETR %s" % file_path, callback=handle_binary)
File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/ftplib.py", line 409, in retrbinary
conn = self.transfercmd(cmd, rest)
File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/ftplib.py", line 371, in transfercmd
return self.ntransfercmd(cmd, rest)[0]
File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/ftplib.py", line 334, in ntransfercmd
resp = self.sendcmd(cmd)
File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/ftplib.py", line 244, in sendcmd
return self.getresp()
File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/ftplib.py", line 217, in getresp
raise error_temp, resp
error_temp: 425 Rejected data connection from foreign address 74.125.183.23:53274.
Related
I successfully created a Flask app locally (Ubuntu) and wanted to host this app via amazon EC2. I copied all the files to aws and run the application via
(testVenv38) ubuntu#ip-xxx-yy-39-70:~/virtuelleUmgebungen/testVenv38/flaskTest$ python3 application.py
* Serving Flask app "flaskTest" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: on
* Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
* Restarting with stat
* Debugger is active!
* Debugger PIN: xxx-yyy-030
Afterwards I was expecting to access my application from any machine via the following link (yellow) followed by the "5000" from definition in my application.py
app.run(host='0.0.0.0', port=5000, debug=True)
(yellow link is https://3.69.xyz.abc/5000), but I get no response. How can I make my app accessible for everybody?
Here is the whole log. But thats the same I get when I run it locally, so this does not indicate the problem I guess.
(testVenv38) ubuntu#ip-xxxx:~/virtuelleUmgebungen/testVenv38/flasklogin-tutorial-master$ python3 application.py
* Serving Flask app "flask_login_tutorial" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: on
* Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
* Restarting with stat
* Debugger is active!
* Debugger PIN: xxx
failed to send traces to Datadog Agent at http://localhost:8126
Traceback (most recent call last):
File "/home/ubuntu/virtuelleUmgebungen/testVenv38/lib/python3.8/site-packages/tenacity/__init__.py", line 412, in call
result = fn(*args, **kwargs)
File "/home/ubuntu/virtuelleUmgebungen/testVenv38/lib/python3.8/site-packages/ddtrace/internal/writer.py", line 356, in _send_payload
response = self._put(payload, headers)
File "/home/ubuntu/virtuelleUmgebungen/testVenv38/lib/python3.8/site-packages/ddtrace/internal/writer.py", line 332, in _put
conn.request("PUT", self._endpoint, data, headers)
File "/usr/lib/python3.8/http/client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1007, in _send_output
self.send(msg)
File "/usr/lib/python3.8/http/client.py", line 947, in send
self.connect()
File "/usr/lib/python3.8/http/client.py", line 918, in connect
self.sock = self._create_connection(
File "/usr/lib/python3.8/socket.py", line 808, in create_connection
raise err
File "/usr/lib/python3.8/socket.py", line 796, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/ubuntu/virtuelleUmgebungen/testVenv38/lib/python3.8/site-packages/ddtrace/internal/writer.py", line 458, in flush_queue
self._retry_upload(self._send_payload, encoded, n_traces)
File "/home/ubuntu/virtuelleUmgebungen/testVenv38/lib/python3.8/site-packages/tenacity/__init__.py", line 409, in call
do = self.iter(retry_state=retry_state)
File "/home/ubuntu/virtuelleUmgebungen/testVenv38/lib/python3.8/site-packages/tenacity/__init__.py", line 369, in iter
six.raise_from(retry_exc, fut.exception())
File "<string>", line 3, in raise_from
tenacity.RetryError: RetryError[<Future at 0x7f04d4bdfd60 state=finished raised ConnectionRefusedError>]
failed to send traces to Datadog Agent at http://localhost:8126
Traceback (most recent call last):
File "/home/ubuntu/virtuelleUmgebungen/testVenv38/lib/python3.8/site-packages/tenacity/__init__.py", line 412, in call
result = fn(*args, **kwargs)
File "/home/ubuntu/virtuelleUmgebungen/testVenv38/lib/python3.8/site-packages/ddtrace/internal/writer.py", line 356, in _send_payload
response = self._put(payload, headers)
File "/home/ubuntu/virtuelleUmgebungen/testVenv38/lib/python3.8/site-packages/ddtrace/internal/writer.py", line 332, in _put
conn.request("PUT", self._endpoint, data, headers)
File "/usr/lib/python3.8/http/client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1007, in _send_output
self.send(msg)
File "/usr/lib/python3.8/http/client.py", line 947, in send
self.connect()
File "/usr/lib/python3.8/http/client.py", line 918, in connect
self.sock = self._create_connection(
File "/usr/lib/python3.8/socket.py", line 808, in create_connection
raise err
File "/usr/lib/python3.8/socket.py", line 796, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/ubuntu/virtuelleUmgebungen/testVenv38/lib/python3.8/site-packages/ddtrace/internal/writer.py", line 458, in flush_queue
self._retry_upload(self._send_payload, encoded, n_traces)
File "/home/ubuntu/virtuelleUmgebungen/testVenv38/lib/python3.8/site-packages/tenacity/__init__.py", line 409, in call
do = self.iter(retry_state=retry_state)
File "/home/ubuntu/virtuelleUmgebungen/testVenv38/lib/python3.8/site-packages/tenacity/__init__.py", line 369, in iter
six.raise_from(retry_exc, fut.exception())
File "<string>", line 3, in raise_from
tenacity.RetryError: RetryError[<Future at 0x7f4c81faf490 state=finished raised ConnectionRefusedError>]
Edit2:
The opened ports (other than my own ip for ssh access) are the following:
Portbereich Protokoll Quelle Sicherheitsgruppen
Alle Alle sg-XXX7113746eda1a92 default
22 TCP 0.0.0.0/0 default
Edit3:
I added the following entry to incoming traffic:
0 - 65535 TCP 0.0.0.0/0 default
Still does not work.
I'm running a server on heroku where I have intermittent channel layer communications between instances of asyncConsumers. Running heroku locally, I have no problem with redis channel connection since I can have as many of them as I want, but once I upload my server to heroku, it gives me a 20 redis connection limit. If I use more than that my server errors out, so I try to lower the expiry time so inactive redis connections would close. BUT, if a redis connection expires and I try to use the channel name with self.channel_layer.send(), I'll get the error below, how do I recover from this error without having to have external calls to create another asyncConsumer instance?
ERROR Exception inside application: Reader at end of file
File "/app/.heroku/python/lib/python3.6/site-packages/channels/consumer.py", line 59, in __call__
[receive, self.channel_receive], self.dispatch
File "/app/.heroku/python/lib/python3.6/site-packages/channels/utils.py", line 51, in await_many_dispatch
await dispatch(result)
File "/app/.heroku/python/lib/python3.6/site-packages/channels/consumer.py", line 73, in dispatch
await handler(message)
File "./myapp/webhook.py", line 133, in http_request
await self.channel_layer.send(userDB.backEndChannelName,{"type": "device.query"})
File "/app/.heroku/python/lib/python3.6/site-packages/channels_redis/core.py", line 296, in send
if await connection.llen(channel_key) >= self.get_capacity(channel):
File "/app/.heroku/python/lib/python3.6/site-packages/aioredis/commands/list.py", line 70, in llen
return self.execute(b'LLEN', key)
File "/app/.heroku/python/lib/python3.6/site-packages/aioredis/commands/__init__.py", line 51, in execute
return self._pool_or_conn.execute(command, *args, **kwargs)
File "/app/.heroku/python/lib/python3.6/site-packages/aioredis/connection.py", line 322, in execute
raise ConnectionClosedError(msg)
Reader at end of file
I'm sure this is my fault, but my system has been working for months and now does not work. I've looked in Django docs and here on stackoverflow, and I understand the issue that 'connection refused' means nobody is listening on the socket; but I don't understand why, and don't know how to go forward.
I use Vagrant + VirtualBox on a MacPro, and for six months I have been able to create virtual machines and run my app with no problems. I have some VM's where everything works as expected.
On my newly-created Vagrant VM's, I install and configure my django apps and run them using 'python manage.py runserver --settings=mumble.settings.py 0.0.0.0:8000'. Note that this is Django 1.8.3, but again,it works fine on some VM's.
For mysterious reasons, when I create a new vagrant VM and install my django payload (using deploy scripts that haven't changed), and connect to the django server, I get Errno 111 Connection refused:
+ python ./manage.py runserver --settings=mydjango.settings 0.0.0.0:8000
.....
_d^^^^^^^^^b_
.d'' ``b.
.p' `q.
.d' `b.
.d' `b. * Mezzanine 4.0.1
:: :: * Django 1.8.3
:: M E Z Z A N I N E :: * Python 2.7.12
:: :: * MySQL 5.7.18
`p. .q' * Linux 4.4.0-75-generic
`p. .q'
`b. .d'
`q.. ..p'
^q........p^
''''
Performing system checks...
System check identified no issues (0 silenced).
May 31, 2017 - 09:50:50
Django version 1.8.3, using settings 'mydjango.settings'
Starting development server at http://0.0.0.0:8000/
Quit the server with CONTROL-C.
Traceback (most recent call last):
File "/usr/lib/python2.7/wsgiref/handlers.py", line 85, in run
self.result = application(self.environ, self.start_response)
File "/vagrant/repos_here/www.mydjango.org/venv/www/local/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 189, in __call__
response = self.get_response(request)
File "/vagrant/repos_here/www.mydjango.org/venv/www/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 204, in get_response
'request': request
File "/usr/lib/python2.7/logging/__init__.py", line 1185, in error
Log 'msg % args' with severity 'ERROR'.
File "/usr/lib/python2.7/logging/__init__.py", line 1193, in error
self._log(ERROR, msg, args, **kwargs)
File "/usr/lib/python2.7/logging/__init__.py", line 1286, in _log
self.handle(record)
File "/usr/lib/python2.7/logging/__init__.py", line 1296, in handle
self.callHandlers(record)
File "/usr/lib/python2.7/logging/__init__.py", line 1336, in callHandlers
hdlr.handle(record)
File "/usr/lib/python2.7/logging/__init__.py", line 759, in handle
self.emit(record)
File "/vagrant/repos_here/www.mydjango.org/venv/www/local/lib/python2.7/site-packages/django/utils/log.py", line 129, in emit
self.send_mail(subject, message, fail_silently=True, html_message=html_message)
File "/vagrant/repos_here/www.mydjango.org/venv/www/local/lib/python2.7/site-packages/django/utils/log.py", line 132, in send_mail
mail.mail_admins(subject, message, *args, connection=self.connection(), **kwargs)
File "/vagrant/repos_here/www.mydjango.org/venv/www/local/lib/python2.7/site-packages/django/core/mail/__init__.py", line 98, in mail_admins
mail.send(fail_silently=fail_silently)
File "/vagrant/repos_here/www.mydjango.org/venv/www/local/lib/python2.7/site-packages/django/core/mail/message.py", line 303, in send
return self.get_connection(fail_silently).send_messages([self])
File "/vagrant/repos_here/www.mydjango.org/venv/www/local/lib/python2.7/site-packages/django/core/mail/backends/smtp.py", line 100, in send_messages
new_conn_created = self.open()
File "/vagrant/repos_here/www.mydjango.org/venv/www/local/lib/python2.7/site-packages/django/core/mail/backends/smtp.py", line 58, in open
self.connection = connection_class(self.host, self.port, **connection_params)
File "/usr/lib/python2.7/smtplib.py", line 256, in __init__
(code, msg) = self.connect(host, port)
File "/usr/lib/python2.7/smtplib.py", line 316, in connect
self.sock = self._get_socket(host, port, self.timeout)
File "/usr/lib/python2.7/smtplib.py", line 291, in _get_socket
return socket.create_connection((host, port), timeout)
File "/usr/lib/python2.7/socket.py", line 575, in create_connection
raise err
error: [Errno 111] Connection refused
[31/May/2017 09:50:55]"GET /donate/ HTTP/1.1" 500 59
Things I know:
'python ./manage.py' with no arguments will run fine, and shows the help message.
All forms of addressing the web server will fail:
curl http://localhost:8000
curl http://127.0.0.1:8000
curl http://192.168.33.10:8000
from a browser: http://192.168.33.10:8000
nginx is serving on port 80, no problem
my database is a copy of the production database, and it looks fine through 'mysql'
And here is a weird thing: I brought down the Django tutorial, used the 'runserver' command, and it worked!
[vagrant][~/tmp/django_tutorial/mysite]
$ python manage.py runserver 0.0.0.0:8000
Performing system checks...
System check identified no issues (0 silenced).
May 30, 2017 - 20:36:44
Django version 1.8.3, using settings 'mysite.settings'
Starting development server at http://0.0.0.0:8000/
Quit the server with CONTROL-C.
[30/May/2017 20:37:05]"GET / HTTP/1.1" 200 1767
Any suggestions on how to debug this?
The answer has nothing to do with my environment. My co-worker added a name to settings.ADMINS, which had been an empty list before that. When ADMINS is a non-empty list, any error triggers an attempt to send email to all of the email addresses in ADMINS, and my VM was not configured to send email. This is documented but not emphasized. The problem did not happen to my coworker, because that associated changes did not trigger a server error.
I am using app engine and big query as the backend for my website. Whenever the user does some click, i log them into bigquery to do analytics later in the day. I get close to 75k clicks a day. It was working fine till last week. This is the code i use.
body = {"rows":[bodyFields]}
credentials = appengine.AppAssertionCredentials(scope=BIGQUERY_SCOPE)
http = credentials.authorize(httplib2.Http())
bigquery = discovery.build('bigquery', 'v2', http=http)
response = bigquery.tabledata().insertAll(
projectId=PROJECT_ID,
datasetId=BIGQUERY_DATASETID,
tableId=BIGQUERY_TABLEID,
body=body).execute()
Now all of a sudden i am getting over quota exception. My application is a paid app engine instance. Below is the stack-trace of my exception
Attempting refresh to obtain initial access_token
The API call app_identity_service.GetAccessToken() required more quota than is available.
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1102, in __call__
return handler.dispatch()
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "/base/data/home/apps/s~projectname/bigqueryapi.387952303347375306/filename.py", line 1611, in post
bigquery = discovery.build('bigquery', 'v2', http=http)
File "/base/data/home/apps/s~projectname/bigqueryapi.387952303347375306/oauth2client/util.py", line 129, in positional_wrapper
return wrapped(*args, **kwargs)
File "/base/data/home/apps/s~projectname/bigqueryapi.387952303347375306/apiclient/discovery.py", line 198, in build
resp, content = http.request(requested_url)
File "/base/data/home/apps/s~projectname/bigqueryapi.387952303347375306/oauth2client/util.py", line 129, in positional_wrapper
return wrapped(*args, **kwargs)
File "/base/data/home/apps/s~projectname/bigqueryapi.387952303347375306/oauth2client/client.py", line 516, in new_request
self._refresh(request_orig)
File "/base/data/home/apps/s~projectname/bigqueryapi.387952303347375306/oauth2client/appengine.py", line 194, in _refresh
scopes, service_account_id=self.service_account_id)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/app_identity/app_identity.py", line 589, in get_access_token
scopes, service_account_id=service_account_id)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/app_identity/app_identity.py", line 547, in get_access_token_uncached
return rpc.get_result()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result
return self.__get_result_hook(self)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/app_identity/app_identity.py", line 519, in get_access_token_result
rpc.check_success()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 579, in check_success
self.__rpc.CheckSuccess()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_rpc.py", line 134, in CheckSuccess
raise self.exception
OverQuotaError: The API call app_identity_service.GetAccessToken() required more quota than is available.
My traffic hasn't gone up by much also the number of time the handler is hit is almost the same as past 2 months data. So why am i getting this error.
In order to determine why you're hitting the quota error, you'll need to share more detail about your usage. The quota should reset every 24 hours. Do you know how long it takes for the error to appear and how much traffic you've successfully served to that point in time?
You mentioned that you "do analytics later in the day", which suggests that you might be using the TaskQueue API or Deferred Tasks. It's possible that those tasks are failing for other reasons and retrying, which could quickly eat up your quota. If you are using TaskQueues, you might try tuning the queue configuration and retry options.
Another way you might be able to conserve your quota would be to save the bigquery discovery service that you're building to something like the Memcache API, so that it can be reused for multiple requests to the BigQuery service.
I am having trouble setting up a connection from a Python script to Azure Service Bus. A similar code in C# (on windows) is running fine.
from azure.servicebus import *
AZURE_SERVICEBUS_NAMESPACE='blablabla.servicebus.windows.net'
AZURE_SERVICEBUS_ISSUER='RootManageSharedAccessKey'
AZURE_SERVICEBUS_ACCESS_KEY='sdkfjlkewjrlwerjlewrjlewrjfjflsdkfjew3='
bus_service = ServiceBusService(service_namespace=AZURE_SERVICEBUS_NAMESPACE, account_key=AZURE_SERVICEBUS_ACCESS_KEY, issuer=AZURE_SERVICEBUS_ISSUER)
bus_service.create_queue('taskqueue')
This results in the following output:
Traceback (most recent call last):
File "createqueue.py", line 9, in
bus_service.create_queue('taskqueue')
File "/usr/local/lib/python2.7/dist-packages/azure/servicebus/servicebusservice.py", line 143, in create_queue
request.headers = self._update_service_bus_header(request)
File "/usr/local/lib/python2.7/dist-packages/azure/servicebus/servicebusservice.py", line 860, in _update_service_bus_header
('Authorization', self._sign_service_bus_request(request)))
File "/usr/local/lib/python2.7/dist-packages/azure/servicebus/servicebusservice.py", line 868, in _sign_service_bus_request
self._get_token(request.host, request.path) + '"'
File "/usr/local/lib/python2.7/dist-packages/azure/servicebus/servicebusservice.py", line 908, in _get_token
resp = self._httpclient.perform_request(request)
File "/usr/local/lib/python2.7/dist-packages/azure/http/httpclient.py", line 191, in perform_request
self.send_request_headers(connection, request.headers)
File "/usr/local/lib/python2.7/dist-packages/azure/http/httpclient.py", line 170, in send_request_headers
connection.endheaders()
File "/usr/lib/python2.7/httplib.py", line 958, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 818, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 780, in send
self.connect()
File "/usr/lib/python2.7/httplib.py", line 1161, in connect
self.timeout, self.source_address)
File "/usr/lib/python2.7/socket.py", line 553, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
socket.gaierror: [Errno -2] Name or service not known
What is going wrong here?
Thanks
AZURE_SERVICEBUS_NAMESPACE='blablabla'
You don't need to specify the entire URL for the namespace.
Check your connectivity with servicebus service on microsift.
I had the same problem and the issue was with wlan0 connectivity - I've lost my IP assigned by DHCP server.
Or your DNS is not working as should.