Currently, I have tried both plain-old Django SMTP and a few different api-based Django libraries for my transactional email provider (Postmark).
When I run my development server, everything works perfectly. Emails send via the Postmark API with no problem.
When I deploy to AWS with Zappa, visit my website, and do a task that is supposed to send an email (Ex. Resetting a user's password) the page continually loads until it says Endpoint request timed out.
I have tried setting the timeout of my AWS Lambda function to a longer duration in case Django decides to throw an error.
Here is the error that was thrown. Just keep in mind this error only happens in production. I created a custom management command in able to retrieve this error.
HTTPSConnectionPool(host='api.postmarkapp.com', port=443): Max retries exceeded with url: /email/batch (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f6cfbd5dd30>: Failed to establish a new connection: [Errno 110] Connection timed out',)): ConnectionError
Traceback (most recent call last):
File "/var/task/handler.py", line 509, in lambda_handler
return LambdaHandler.lambda_handler(event, context)
File "/var/task/handler.py", line 240, in lambda_handler
return handler.handler(event, context)
File "/var/task/handler.py", line 376, in handler
management.call_command(*event['manage'].split(' '))
File "/var/task/django/core/management/__init__.py", line 131, in call_command
return command.execute(*args, **defaults)
File "/var/task/django/core/management/base.py", line 330, in execute
output = self.handle(*args, **options)
File "/var/task/users/management/commands/sendemail.py", line 13, in handle
fail_silently=False,
File "/var/task/django/core/mail/__init__.py", line 62, in send_mail
return mail.send()
File "/var/task/django/core/mail/message.py", line 348, in send
return self.get_connection(fail_silently).send_messages([self])
File "/var/task/postmarker/django/backend.py", line 66, in send_messages
responses = self.client.emails.send_batch(*prepared_messages, TrackOpens=self.get_option('TRACK_OPENS'))
File "/var/task/postmarker/models/emails.py", line 332, in send_batch
return self.EmailBatch(*emails).send(**extra)
File "/var/task/postmarker/models/emails.py", line 247, in send
responses = [self._manager._send_batch(*batch) for batch in chunks(emails, self.MAX_SIZE)]
File "/var/task/postmarker/models/emails.py", line 247, in <listcomp>
responses = [self._manager._send_batch(*batch) for batch in chunks(emails, self.MAX_SIZE)]
File "/var/task/postmarker/models/emails.py", line 276, in _send_batch
return self.call('POST', '/email/batch', data=emails)
File "/var/task/postmarker/models/base.py", line 72, in call
return self.client.call(*args, **kwargs)
File "/var/task/postmarker/core.py", line 106, in call
**kwargs
File "/var/task/postmarker/core.py", line 129, in _call
method, url, json=data, params=kwargs, headers=default_headers, timeout=self.timeout
File "/var/task/requests/sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "/var/task/requests/sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "/var/task/requests/adapters.py", line 508, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.postmarkapp.com', port=443): Max retries exceeded with url: /email/batch (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f6cfbd5dd30>: Failed to establish a new connection: [Errno 110] Connection timed out',))
I have allowed all incoming and outgoing traffic to my AWS security group in an attempt to fix this. Still to no avail.
Any help would be greatly, greatly appreciated. Cheers.
The explanation is simple: A Lambda instance running in a VPC cannot access the internet:
When you add VPC configuration to a Lambda function, it can only access resources in that VPC. If a Lambda function needs to access both VPC resources and the public Internet, the VPC needs to have a Network Address Translation (NAT) instance inside the VPC.
The solution is also simple, if annoying: run a NAT Instance or NAT Gateway in the VPC. (An alternate solution is to take your Lambda out of the VPC, but that is a much bigger change.)
I am running Django / Zappa in Lambda with a NAT instance for connecting to Amazon Simple Email Service and it works fine.
Related
I have a custom container that takes a request, does some feature extraction and then passes on the enhanced request to a classifier endpoint. During feature extraction another endpoint is being called for generating text embeddings. I am using the HuggingFace estimator for my embedding model.
It has been working fine, but there was an increase in requests and looks like the embedding endpoint timed out somehow.
I am looking at adding automatic scaling to the endpoint, but I want to make sure I understand what is happening and that it properly addresses the issue. Unfortunately searching for this error message does not pull up much. The instance metrics is not showing the endpoint to be overloaded - cpu utilization was max ~30%. Would auto scaling address the no worker issue or is this something different? I was receiving a few hundred requests per minute at the time.
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2073, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1518, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1516, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1502, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
File "/opt/program/predictor.py", line 56, in transformation
result = preprocessor.transform(data)
File "/opt/program/preprocessor.py", line 189, in transform
response = embed_predictor.predict(data=json.dumps(payload))
File "/usr/local/lib/python3.7/site-packages/sagemaker/predictor.py", line 136, in predict
response = self.sagemaker_session.sagemaker_runtime_client.invoke_endpoint(**request_args)
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 386, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 705, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received server error (503) from primary with message "{
"code": 503,
"type": "ServiceUnavailableException",
"message": "No worker is available to serve request: model"
}
I would suggest confirming MemoryUtilization is not being overwhelmed and there is no specifc error in CloudWatch Logs as well.
If MemoryUtilization is overwhelmed, you can test configuring Auto Scaling in order to distribute the load of request to multiple instances. That being said, while I am not sure of the details of your custom container, I also recommend confirming the container itself can handle multiple concurrent requests (i.e have multiple workers available to serve requests).
I'm running a server on heroku where I have intermittent channel layer communications between instances of asyncConsumers. Running heroku locally, I have no problem with redis channel connection since I can have as many of them as I want, but once I upload my server to heroku, it gives me a 20 redis connection limit. If I use more than that my server errors out, so I try to lower the expiry time so inactive redis connections would close. BUT, if a redis connection expires and I try to use the channel name with self.channel_layer.send(), I'll get the error below, how do I recover from this error without having to have external calls to create another asyncConsumer instance?
ERROR Exception inside application: Reader at end of file
File "/app/.heroku/python/lib/python3.6/site-packages/channels/consumer.py", line 59, in __call__
[receive, self.channel_receive], self.dispatch
File "/app/.heroku/python/lib/python3.6/site-packages/channels/utils.py", line 51, in await_many_dispatch
await dispatch(result)
File "/app/.heroku/python/lib/python3.6/site-packages/channels/consumer.py", line 73, in dispatch
await handler(message)
File "./myapp/webhook.py", line 133, in http_request
await self.channel_layer.send(userDB.backEndChannelName,{"type": "device.query"})
File "/app/.heroku/python/lib/python3.6/site-packages/channels_redis/core.py", line 296, in send
if await connection.llen(channel_key) >= self.get_capacity(channel):
File "/app/.heroku/python/lib/python3.6/site-packages/aioredis/commands/list.py", line 70, in llen
return self.execute(b'LLEN', key)
File "/app/.heroku/python/lib/python3.6/site-packages/aioredis/commands/__init__.py", line 51, in execute
return self._pool_or_conn.execute(command, *args, **kwargs)
File "/app/.heroku/python/lib/python3.6/site-packages/aioredis/connection.py", line 322, in execute
raise ConnectionClosedError(msg)
Reader at end of file
First Time I am trying AWS services. I have to integrate AWS polly with asterisk for text to speech.
here is example code i written to convert text to speech
from boto3 import client
import boto3
import StringIO
from contextlib import closing
polly = client("polly", 'us-east-1' )
response = polly.synthesize_speech(
Text="Good Morning. My Name is Rajesh. I am Testing Polly AWS Service For Voice Application.",
OutputFormat="mp3",
VoiceId="Raveena")
print(response)
if "AudioStream" in response:
with closing(response["AudioStream"]) as stream:
data = stream.read()
fo = open("pollytest.mp3", "w+")
fo.write( data )
fo.close()
I am getting following error.
Traceback (most recent call last):
File "pollytest.py", line 11, in <module>
VoiceId="Raveena")
File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 253, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 530, in _make_api_call
operation_model, request_dict)
File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 141, in make_request
return self._send_request(request_dict, operation_model)
File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 166, in _send_request
request = self.create_request(request_dict, operation_model)
File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 150, in create_request
operation_name=operation_model.name)
File "/usr/local/lib/python2.7/dist-packages/botocore/hooks.py", line 227, in emit
return self._emit(event_name, kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/hooks.py", line 210, in _emit
response = handler(**kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/signers.py", line 90, in handler
return self.sign(operation_name, request)
File "/usr/local/lib/python2.7/dist-packages/botocore/signers.py", line 147, in sign
auth.add_auth(request)
File "/usr/local/lib/python2.7/dist-packages/botocore/auth.py", line 316, in add_auth
raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials
I want to provide credentials directly in this script so that i can use this in asterisk system application.
UPDATE:
created a file ~/.aws/credentials with below content
[default]
aws_access_key_id=XXXXXXXX
aws_secret_access_key=YYYYYYYYYYY
now for my current login user its working fine, but for asterisk PBX it is not working.
Your code runs perfectly fine for me!
The last line is saying:
botocore.exceptions.NoCredentialsError: Unable to locate credentials
So, it is unable to authenticate against AWS.
If you are running this code on an Amazon EC2 instance, the simplest method is to assign an IAM Role to the instance when it is launched (it can't be added later). This will automatically assign credentials that can be used by application running on the instance -- no code changes required.
Alternatively, you could obtain an Access Key and Secret Key from IAM for your IAM User and store those credentials in a local file via the aws configure command.
It is bad practice to put credentials in source code, since they may become compromised.
See:
IAM Roles for Amazon EC2
Best Practices for Managing AWS Access Keys
Please note,asterisk pbx usually run under asterisk user.
So you have put authentification for that user, not root.
I am using app engine and big query as the backend for my website. Whenever the user does some click, i log them into bigquery to do analytics later in the day. I get close to 75k clicks a day. It was working fine till last week. This is the code i use.
body = {"rows":[bodyFields]}
credentials = appengine.AppAssertionCredentials(scope=BIGQUERY_SCOPE)
http = credentials.authorize(httplib2.Http())
bigquery = discovery.build('bigquery', 'v2', http=http)
response = bigquery.tabledata().insertAll(
projectId=PROJECT_ID,
datasetId=BIGQUERY_DATASETID,
tableId=BIGQUERY_TABLEID,
body=body).execute()
Now all of a sudden i am getting over quota exception. My application is a paid app engine instance. Below is the stack-trace of my exception
Attempting refresh to obtain initial access_token
The API call app_identity_service.GetAccessToken() required more quota than is available.
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1102, in __call__
return handler.dispatch()
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "/base/data/home/apps/s~projectname/bigqueryapi.387952303347375306/filename.py", line 1611, in post
bigquery = discovery.build('bigquery', 'v2', http=http)
File "/base/data/home/apps/s~projectname/bigqueryapi.387952303347375306/oauth2client/util.py", line 129, in positional_wrapper
return wrapped(*args, **kwargs)
File "/base/data/home/apps/s~projectname/bigqueryapi.387952303347375306/apiclient/discovery.py", line 198, in build
resp, content = http.request(requested_url)
File "/base/data/home/apps/s~projectname/bigqueryapi.387952303347375306/oauth2client/util.py", line 129, in positional_wrapper
return wrapped(*args, **kwargs)
File "/base/data/home/apps/s~projectname/bigqueryapi.387952303347375306/oauth2client/client.py", line 516, in new_request
self._refresh(request_orig)
File "/base/data/home/apps/s~projectname/bigqueryapi.387952303347375306/oauth2client/appengine.py", line 194, in _refresh
scopes, service_account_id=self.service_account_id)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/app_identity/app_identity.py", line 589, in get_access_token
scopes, service_account_id=service_account_id)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/app_identity/app_identity.py", line 547, in get_access_token_uncached
return rpc.get_result()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result
return self.__get_result_hook(self)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/app_identity/app_identity.py", line 519, in get_access_token_result
rpc.check_success()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 579, in check_success
self.__rpc.CheckSuccess()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_rpc.py", line 134, in CheckSuccess
raise self.exception
OverQuotaError: The API call app_identity_service.GetAccessToken() required more quota than is available.
My traffic hasn't gone up by much also the number of time the handler is hit is almost the same as past 2 months data. So why am i getting this error.
In order to determine why you're hitting the quota error, you'll need to share more detail about your usage. The quota should reset every 24 hours. Do you know how long it takes for the error to appear and how much traffic you've successfully served to that point in time?
You mentioned that you "do analytics later in the day", which suggests that you might be using the TaskQueue API or Deferred Tasks. It's possible that those tasks are failing for other reasons and retrying, which could quickly eat up your quota. If you are using TaskQueues, you might try tuning the queue configuration and retry options.
Another way you might be able to conserve your quota would be to save the bigquery discovery service that you're building to something like the Memcache API, so that it can be reused for multiple requests to the BigQuery service.
Why google adwords api stops on call this link:
https://adwords.google.com/api/adwords/mcm/v201502/CustomerService?wsdl
With this error - should I load some certificate before and how?
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>
Using Python 2.7.10.
Full source code:
create_adwords_client_without_yaml.py
Full error code:
Traceback (most recent call last):
File "C:/Users/Crezary Wagner/PycharmProjects/learn-adwords/src/examples/create_adwords_client_without_yaml.py", line 56, in <module>
CLIENT_CUSTOMER_ID)
File "C:/Users/Crezary Wagner/PycharmProjects/learn-adwords/src/examples/create_adwords_client_without_yaml.py", line 50, in main
customer = adwords_client.GetService('CustomerService').get()
File "C:\root\Python27\lib\site-packages\googleads\adwords.py", line 256, in GetService
proxy=proxy_option, cache=self.cache, timeout=3600)
File "C:\root\Python27\lib\site-packages\suds\client.py", line 115, in __init__
self.wsdl = reader.open(url)
File "C:\root\Python27\lib\site-packages\suds\reader.py", line 150, in open
d = self.fn(url, self.options)
File "C:\root\Python27\lib\site-packages\suds\wsdl.py", line 136, in __init__
d = reader.open(url)
File "C:\root\Python27\lib\site-packages\suds\reader.py", line 74, in open
d = self.download(url)
File "C:\root\Python27\lib\site-packages\suds\reader.py", line 92, in download
fp = self.options.transport.open(Request(url))
File "C:\root\Python27\lib\site-packages\suds\transport\https.py", line 62, in open
return HttpTransport.open(self, request)
File "C:\root\Python27\lib\site-packages\suds\transport\http.py", line 67, in open
return self.u2open(u2request)
File "C:\root\Python27\lib\site-packages\suds\transport\http.py", line 132, in u2open
return url.open(u2request, timeout=tm)
File "C:\root\Python27\lib\urllib2.py", line 431, in open
response = self._open(req, data)
File "C:\root\Python27\lib\urllib2.py", line 449, in _open
'_open', req)
File "C:\root\Python27\lib\urllib2.py", line 409, in _call_chain
result = func(*args)
File "C:\root\Python27\lib\urllib2.py", line 1240, in https_open
context=self._context)
File "C:\root\Python27\lib\urllib2.py", line 1197, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>
Python uses certificates from 'system ssl certificate store' to certify https connection, if there is not any appropriate ssl cert in the store error occurs like this.
Download ssl certificate (Open your https link in browser and click lock icon in address bar > More Information > View Certificate > Details > Export) and install it on your system as stated this link http://windows.microsoft.com/en-us/windows/import-export-certificates-private-keys#1TC=windows-7
Not sure if that's the problem here, but worth checking it.
Python 2.7.9 enabled certificate validation by default for HTTP connections.
The server you're connecting to does not have a certificate that is trusted by your client. pysphere should configure SSL appropriately for this use case.
Try making your request like:
requests.get('https://adwords.google.com/api/adwords/mcm/v201502/CustomerService?wsdl', verify=False)
Try this, it helped me:
import ssl
ssl._create_default_https_context = ssl._create_unverified_context
I encountered this issue. I had my phone setup using the same DNS block list and it wasn't immediately apparent after I'd enabled the tool and resumed work on this particular project. I suggest scrutinizing your setup and verify that there aren't any adblockers (DNS level in my case ala NextDNS/hosted PiHole) enabled. Hours upon hours spent trying out python versions, certificates, reinstalling things. Hope this helps someone!