I'm trying to put to work AWS's Textract export table suggestion in this link
I'm a complete newbie in AWS's solutions and in command prompt so I'm trying to do exactly as they suggest. I'm running that in python so I'm using this piece of code:
import os
k=os.system("python textract_python_table_parser.py my_pdf_file_path.pdf")
print(k)
The code runs, I get an Image loaded my_pdf_file_path.pdf however at some point it bugs on credential matters:
Traceback (most recent call last):
File "/Users/santanna_santanna/PycharmProjects/KlooksExplore/PDFWork/textract_python_table_parser.py", line 108, in <module>
main(file_name)
File "/Users/santanna_santanna/PycharmProjects/KlooksExplore/PDFWork/textract_python_table_parser.py", line 94, in main
table_csv = get_table_csv_results(file_name)
File "/Users/santanna_santanna/PycharmProjects/KlooksExplore/PDFWork/textract_python_table_parser.py", line 53, in get_table_csv_results
response = client.analyze_document(Document={'Bytes': bytes_test}, FeatureTypes=['TABLES'])
File "/Users/santanna_santanna/anaconda3/lib/python3.6/site-packages/botocore/client.py", line 316, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/Users/santanna_santanna/anaconda3/lib/python3.6/site-packages/botocore/client.py", line 622, in _make_api_call
operation_model, request_dict, request_context)
File "/Users/santanna_santanna/anaconda3/lib/python3.6/site-packages/botocore/client.py", line 641, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/Users/santanna_santanna/anaconda3/lib/python3.6/site-packages/botocore/endpoint.py", line 102, in make_request
return self._send_request(request_dict, operation_model)
File "/Users/santanna_santanna/anaconda3/lib/python3.6/site-packages/botocore/endpoint.py", line 132, in _send_request
request = self.create_request(request_dict, operation_model)
File "/Users/santanna_santanna/anaconda3/lib/python3.6/site-packages/botocore/endpoint.py", line 116, in create_request
operation_name=operation_model.name)
File "/Users/santanna_santanna/anaconda3/lib/python3.6/site-packages/botocore/hooks.py", line 356, in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "/Users/santanna_santanna/anaconda3/lib/python3.6/site-packages/botocore/hooks.py", line 228, in emit
return self._emit(event_name, kwargs)
File "/Users/santanna_santanna/anaconda3/lib/python3.6/site-packages/botocore/hooks.py", line 211, in _emit
response = handler(**kwargs)
File "/Users/santanna_santanna/anaconda3/lib/python3.6/site-packages/botocore/signers.py", line 90, in handler
return self.sign(operation_name, request)
File "/Users/santanna_santanna/anaconda3/lib/python3.6/site-packages/botocore/signers.py", line 160, in sign
auth.add_auth(request)
File "/Users/santanna_santanna/anaconda3/lib/python3.6/site-packages/botocore/auth.py", line 357, in add_auth
raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials
I'm aware I didn't pass any credentials and that's natural to happen but where should I pass it and what would be the right syntax for that using python os? Amazon's example doesn't say anything about that.
It depends where you run your code, for example:
local computer - can use aws configure CLI to set your credetnails
EC2 instance - use instance role
lambda function - use lambda execution role
Related
We have a Glue Python Shell Pipeline Job trying to invoke AWS AppFlow through boto3 APIs. The challenge we have is that we need to design our solution without the need for internet access within Glue
Previously we had challenges in upgrading the boto3 APIs in AWS Glue Shell to the latest version without internet. But then we have found the solution and documented in the following link. In our architecture, we are using AWS Python Shell as our lightweight Datapipeline Engine leveraging boto3 APIs
Git Glue Boto3 Bug & Solution
The following Appflow API python code is working perfectly fine in our local Jupyter Notebooks, as AWS App flow API is invoked over the internet.
##Extra code as per above link to update boto3 version as well, which is omitted
import boto3
print(boto3.__version__)
client=boto3.client('appflow')
response=client.describe_flow(
flowName='sf_non_pii'
)
But while we use the same code in AWS Glue, it is failing as it is not able to access the internet.
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "https://appflow.us-east-2.amazonaws.com/describe-flow"
Please find the logs below
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 160, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 84, in create_connection
raise err
File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 74, in create_connection
sock.connect(sa)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/botocore/httpsession.py", line 263, in send
chunked=self._chunked(request.headers),
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 727, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 386, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/local/lib/python3.6/site-packages/urllib3/packages/six.py", line 735, in reraise
raise value
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
chunked=chunked,
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 381, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 309, in connect
conn = self._new_conn()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 167, in _new_conn
% (self.host, self.timeout),
urllib3.exceptions.ConnectTimeoutError: (<botocore.awsrequest.AWSHTTPSConnection object at 0x7f5f8802c0f0>, 'Connection to appflow.us-east-2.amazonaws.com timed out. (connect timeout=60)')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/runscript.py", line 123, in <module>
runpy.run_path(temp_file_path, run_name='__main__')
File "/usr/local/lib/python3.6/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/usr/local/lib/python3.6/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/tmp/glue-python-scripts-a3cgmro3/ref_appflow.py", line 44, in <module>
File "/tmp/botocore/client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/tmp/botocore/client.py", line 663, in _make_api_call
operation_model, request_dict, request_context)
File "/tmp/botocore/client.py", line 682, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/tmp/botocore/endpoint.py", line 102, in make_request
return self._send_request(request_dict, operation_model)
File "/tmp/botocore/endpoint.py", line 137, in _send_request
success_response, exception):
File "/tmp/botocore/endpoint.py", line 256, in _needs_retry
caught_exception=caught_exception, request_dict=request_dict)
File "/tmp/botocore/hooks.py", line 356, in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "/tmp/botocore/hooks.py", line 228, in emit
return self._emit(event_name, kwargs)
File "/tmp/botocore/hooks.py", line 211, in _emit
response = handler(**kwargs)
File "/tmp/botocore/retryhandler.py", line 183, in __call__
if self._checker(attempts, response, caught_exception):
File "/tmp/botocore/retryhandler.py", line 251, in __call__
caught_exception)
File "/tmp/botocore/retryhandler.py", line 277, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/tmp/botocore/retryhandler.py", line 317, in __call__
caught_exception)
File "/tmp/botocore/retryhandler.py", line 223, in __call__
attempt_number, caught_exception)
File "/tmp/botocore/retryhandler.py", line 359, in _check_caught_exception
raise caught_exception
File "/tmp/botocore/endpoint.py", line 200, in _do_get_response
http_response = self._send(request)
File "/tmp/botocore/endpoint.py", line 269, in _send
return self.http_session.send(request)
File "/tmp/botocore/httpsession.py", line 287, in send
raise ConnectTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "https://appflow.us-east-2.amazonaws.com/describe-flow"
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/runscript.py", line 142, in <module>
raise e_type(e_value).with_traceback(new_stack)
TypeError: __init__() takes 1 positional argument but 2 were given
We had faced similar issues for Redshift, AWS Athena, SSM, etc. But for these services, we have VPC endpoints available. Creating the endpoints enabled our glue job to connect privately within the VPC. Please refer to the link below for supported products
AWS services that you can use with AWS PrivateLink
But since Appflow is a newer product, we do not have endpoint, even though some unclear documentation regarding a throwaway create and delete VPC endpoint as below
Private Amazon AppFlow flows
Please let us know how we can invoke boto3 APIs for Appflow without going through the internet.
Hi I'm trying to access serverless API. I got as far as creating virtual environments, activating it and puting my credentials in. Though when I try to deploy aws chalice, this is what i get:
Creating deployment package.
Traceback (most recent call last):
File "c:\users\jerom\desktop\venv\lib\site-packages\chalice\cli\__init__.py", line 599, in main
return cli(obj={})
File "c:\users\jerom\desktop\venv\lib\site-packages\click\core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "c:\users\jerom\desktop\venv\lib\site-packages\click\core.py", line 782, in main
rv = self.invoke(ctx)
File "c:\users\jerom\desktop\venv\lib\site-packages\click\core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "c:\users\jerom\desktop\venv\lib\site-packages\click\core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "c:\users\jerom\desktop\venv\lib\site-packages\click\core.py", line 610, in invoke
return callback(*args, **kwargs)
File "c:\users\jerom\desktop\venv\lib\site-packages\click\decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "c:\users\jerom\desktop\venv\lib\site-packages\chalice\cli\__init__.py", line 206, in deploy
deployed_values = d.deploy(config, chalice_stage_name=stage)
File "c:\users\jerom\desktop\venv\lib\site-packages\chalice\deploy\deployer.py", line 353, in deploy
return self._deploy(config, chalice_stage_name)
File "c:\users\jerom\desktop\venv\lib\site-packages\chalice\deploy\deployer.py", line 364, in _deploy
plan = self._plan_stage.execute(resources)
File "c:\users\jerom\desktop\venv\lib\site-packages\chalice\deploy\planner.py", line 139, in execute
result = handler(resource)
File "c:\users\jerom\desktop\venv\lib\site-packages\chalice\deploy\planner.py", line 195, in _plan_lambdafunction
if not self._remote_state.resource_exists(resource):
File "c:\users\jerom\desktop\venv\lib\site-packages\chalice\deploy\planner.py", line 61, in resource_exists
result = handler(resource)
File "c:\users\jerom\desktop\venv\lib\site-packages\chalice\deploy\planner.py", line 94, in _resource_exists_lambdafunction
return self._client.lambda_function_exists(resource.function_name)
File "c:\users\jerom\desktop\venv\lib\site-packages\chalice\awsclient.py", line 103, in lambda_function_exists
client = self._client('lambda')
File "c:\users\jerom\desktop\venv\lib\site-packages\chalice\awsclient.py", line 708, in _client
self._client_cache[service_name] = self._session.create_client(
File "c:\users\jerom\desktop\venv\lib\site-packages\botocore\session.py", line 831, in create_client
client = client_creator.create_client(
File "c:\users\jerom\desktop\venv\lib\site-packages\botocore\client.py", line 83, in create_client
client_args = self._get_client_args(
File "c:\users\jerom\desktop\venv\lib\site-packages\botocore\client.py", line 285, in _get_client_args
return args_creator.get_client_args(
File "c:\users\jerom\desktop\venv\lib\site-packages\botocore\args.py", line 99, in get_client_args
endpoint = endpoint_creator.create_endpoint(
File "c:\users\jerom\desktop\venv\lib\site-packages\botocore\endpoint.py", line 286, in create_endpoint
raise ValueError("Invalid endpoint: %s" % endpoint_url)
ValueError: Invalid endpoint: https://lambda.New Jersey.amazonaws.com
does anyone have any idea how to solve this?
It would appear that you provided an invalid valid value for "Region" when storing your credentials.
The region name forms part of the URL when connecting to AWS services, which is why your code is trying to access https://lambda.New Jersey.amazonaws.com. (New Jersey is not a valid Region.)
To fix:
Use the AWS CLI aws configure command to update your credentials.
In the Region field, provided a region code from the list of AWS Endpoints, such as us-west-2 or eu-west-2.
I'm having trouble running the install script for Google Cloud SDK - this question has been asked but without clear answers so far so any help would be welcome! I'm running Python 2.7.14.
This is the error I'm getting:
httplib2.python2.httplib2.SSLHandshakeError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)
The full stack trace is:
Welcome to the Google Cloud SDK!
To help improve the quality of this product, we collect anonymized usage data
and anonymized stacktraces when crashes are encountered; additional information
is available at <https://cloud.google.com/sdk/usage-statistics>. You may choose
to opt out of this collection now (by choosing 'N' at the below prompt), or at
any time in the future by running the following command:
gcloud config set disable_usage_reporting true
Do you want to help improve the Google Cloud SDK (Y/n)? Y
Traceback (most recent call last):
File "/Users/<user>/Downloads/google-cloud-sdk/bin/bootstrapping/install.py", line 218, in <module>
main()
File "/Users/<user>/Downloads/google-cloud-sdk/bin/bootstrapping/install.py", line 196, in main
Install(pargs.override_components, pargs.additional_components)
File "/Users/<user>/Downloads/google-cloud-sdk/bin/bootstrapping/install.py", line 141, in Install
_CLI.Execute(['--quiet', 'components', 'list'])
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 870, in Execute
self._HandleAllErrors(exc, command_path_string, specified_arg_names)
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 908, in _HandleAllErrors
exceptions.HandleError(exc, command_path_string, self.__known_error_handler)
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/calliope/exceptions.py", line 526, in HandleError
core_exceptions.reraise(exc)
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/core/exceptions.py", line 111, in reraise
six.reraise(type(exc_value), exc_value, tb)
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 844, in Execute
resources = calliope_command.Run(cli=self, args=args)
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 756, in Run
resources = command_instance.Run(args)
File "/Users/<user>/Downloads/google-cloud-sdk/lib/surface/components/list.py", line 98, in Run
only_local_state=args.only_local_state)
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/core/updater/update_manager.py", line 674, in List
to_print, current_version, latest_version = self._GetPrintListWithDiff()
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/core/updater/update_manager.py", line 704, in _GetPrintListWithDiff
_, diff = self._GetStateAndDiff(command_path='components.list')
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/core/updater/update_manager.py", line 600, in _GetStateAndDiff
command_path=command_path)
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/core/updater/update_manager.py", line 583, in _GetLatestSnapshot
*effective_url.split(','), command_path=command_path)
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/core/updater/snapshots.py", line 178, in FromURLs
for url in urls]
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/core/updater/snapshots.py", line 199, in _DictFromURL
response = installers.ComponentInstaller.MakeRequest(url, command_path)
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/core/updater/installers.py", line 293, in MakeRequest
return ComponentInstaller._RawRequest(req, timeout=timeout)
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/core/updater/installers.py", line 337, in _RawRequest
should_retry_if=RetryIf, sleep_ms=500)
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/core/util/retry.py", line 178, in RetryOnException
exceptions.reraise(exc_info[1], tb=exc_info[2])
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/core/exceptions.py", line 111, in reraise
six.reraise(type(exc_value), exc_value, tb)
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/core/util/retry.py", line 159, in TryFunc
return func(*args, **kwargs), None
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/core/url_opener.py", line 78, in urlopen
return opener.open(req, data, timeout)
File "/usr/local/Cellar/python#2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 429, in open
response = self._open(req, data)
File "/usr/local/Cellar/python#2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 447, in _open
'_open', req)
File "/usr/local/Cellar/python#2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/core/url_opener.py", line 63, in https_open
return self.do_open(build, req)
File "/usr/local/Cellar/python#2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1195, in do_open
h.request(req.get_method(), req.get_selector(), req.data, headers)
File "/usr/local/Cellar/python#2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1042, in request
self._send_request(method, url, body, headers)
File "/usr/local/Cellar/python#2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1082, in _send_request
self.endheaders(body)
File "/usr/local/Cellar/python#2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1038, in endheaders
self._send_output(message_body)
File "/usr/local/Cellar/python#2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 882, in _send_output
self.send(msg)
File "/usr/local/Cellar/python#2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 844, in send
self.connect()
File "/Users/<user>/Downloads/google-cloud-sdk/lib/third_party/httplib2/python2/httplib2/__init__.py", line 1120, in connect
raise SSLHandshakeError(e)
httplib2.python2.httplib2.SSLHandshakeError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)
This type of error is usually encountered due to a number of reasons, including the following:
a. Networking configurations and any proxy that can cause the handshake to fail.
b. Antivirus, firewalls, or any other software that are preventing the traffic.
C. Corporate certificate is being used.
Depending on the situation, you can check your proxy and ensure that traffic/connections to accounts.google.com are allowed for authorization. To use corporate certificates, it is necessary to append your corporate certificate to google-cloud-sdk/lib/third_party/httplib2/cacerts.txt to allow connections from Google cloud.
I am new to dealing with the Parse-Server and hosting on AWS. But I have noticed that the "eb labs download" command in terminal works when my Parse Server environment is in N.Virginia but comes back with a whole list of errors when the the server environment was initially kept in Oregon. The errors are pertaining to "HTTP header errors". Anyone know why this is happening? Thanks in advance! Error is below:
Downloading application version...
Traceback (most recent call last):
File "/usr/local/bin/eb", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/site-packages/ebcli/core/ebcore.py", line 150, in main
app.run()
File "/usr/local/lib/python2.7/site-packages/cement/core/foundation.py", line 797, in run
return_val = self.controller._dispatch()
File "/usr/local/lib/python2.7/site-packages/cement/core/controller.py", line 472, in _dispatch
return func()
File "/usr/local/lib/python2.7/site-packages/cement/core/controller.py", line 472, in _dispatch
return func()
File "/usr/local/lib/python2.7/site-packages/cement/core/controller.py", line 478, in _dispatch
return func()
File "/usr/local/lib/python2.7/site-packages/ebcli/core/abstractcontroller.py", line 57, in default
self.do_command()
File "/usr/local/lib/python2.7/site-packages/ebcli/labs/download.py", line 36, in do_command
download_source_bundle(app_name, env_name)
File "/usr/local/lib/python2.7/site-packages/ebcli/labs/download.py", line 49, in download_source_bundle
data = s3.get_object(bucket_name, key_name)
File "/usr/local/lib/python2.7/site-packages/ebcli/lib/s3.py", line 68, in get_object
Key=key)
File "/usr/local/lib/python2.7/site-packages/ebcli/lib/s3.py", line 34, in _make_api_call
return aws.make_api_call('s3', operation_name, **operation_options)
File "/usr/local/lib/python2.7/site-packages/ebcli/lib/aws.py", line 218, in make_api_call
response_data = operation(**operation_options)
File "/Users/Home/Library/Python/2.7/lib/python/site-packages/botocore/client.py", line 251, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/Users/Home/Library/Python/2.7/lib/python/site-packages/botocore/client.py", line 526, in _make_api_call
operation_model, request_dict)
File "/Users/Home/Library/Python/2.7/lib/python/site-packages/botocore/endpoint.py", line 141, in make_request
return self._send_request(request_dict, operation_model)
File "/Users/Home/Library/Python/2.7/lib/python/site-packages/botocore/endpoint.py", line 170, in _send_request
success_response, exception):
File "/Users/Home/Library/Python/2.7/lib/python/site-packages/botocore/endpoint.py", line 249, in _needs_retry
caught_exception=caught_exception, request_dict=request_dict)
File "/Users/Home/Library/Python/2.7/lib/python/site-packages/botocore/hooks.py", line 227, in emit
return self._emit(event_name, kwargs)
File "/Users/Home/Library/Python/2.7/lib/python/site-packages/botocore/hooks.py", line 210, in _emit
response = handler(**kwargs)
File "/Users/Home/Library/Python/2.7/lib/python/site-packages/botocore/utils.py", line 868, in redirect_from_error
new_region = self.get_bucket_region(bucket, response)
File "/Users/Home/Library/Python/2.7/lib/python/site-packages/botocore/utils.py", line 913, in get_bucket_region
response_headers = service_response['ResponseMetadata']['HTTPHeaders']
KeyError: 'HTTPHeaders'
Generally the EB CLI works in a single region at a time. If you have a specific region in which you want to use you can specify it using the --region flag.
eb labs download --region us-west-2
Otherwise it is usually best practice to keep your AWS stack resources in a single region.
I am trying to setup dynamic thumbnail service thumbor and to support s3 as storage, I need to setup this community powered pip library for aws.
Its working well on my local environment but when I am trying to host it on one of our servers, I am getting NoCredentialsError. I am assuming this is because of difference versions of botocore (latest one and one installed by pip library). Here is error log:
File "/usr/local/lib/python2.7/dist-packages/botocore/session.py", line 774, in get_component
# client config from the session
File "/usr/local/lib/python2.7/dist-packages/botocore/session.py", line 174, in <lambda>
self._components.lazy_register_component(
File "/usr/local/lib/python2.7/dist-packages/botocore/session.py", line 453, in get_data
- agent_version is the value of the `user_agent_version`
File "/usr/local/lib/python2.7/dist-packages/botocore/loaders.py", line 119, in _wrapper
data = func(self, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/loaders.py", line 364, in load_data
DataNotFoundError: Unable to load data for: _endpoints
2016-04-24 12:14:34 tornado.application:ERROR Future exception was never retrieved: Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 230, in wrapper
yielded = next(result)
File "/usr/local/lib/python2.7/dist-packages/thumbor/handlers/imaging.py", line 31, in check_image
exists = yield gen.maybe_future(self.context.modules.storage.exists(kw['image'][:self.context.config.MAX_ID_LENGTH]))
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 455, in wrapper
future.result()
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 215, in result
raise_exc_info(self._exc_info)
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 443, in wrapper
result = f(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tc_aws/aws/storage.py", line 107, in exists
self.storage.get(file_abspath, callback=return_data)
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 455, in wrapper
future.result()
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 215, in result
raise_exc_info(self._exc_info)
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 443, in wrapper
result = f(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tc_aws/aws/bucket.py", line 44, in get
Key=self._clean_key(path),
File "/usr/local/lib/python2.7/dist-packages/tornado_botocore/base.py", line 97, in call
return self._make_api_call(operation_name=self.operation, api_params=kwargs, callback=callback)
File "/usr/local/lib/python2.7/dist-packages/tornado_botocore/base.py", line 60, in _make_api_call
operation_model=operation_model, request_dict=request_dict, callback=callback)
File "/usr/local/lib/python2.7/dist-packages/tornado_botocore/base.py", line 54, in _make_request
request_dict=request_dict, operation_model=operation_model, callback=callback)
File "/usr/local/lib/python2.7/dist-packages/tornado_botocore/base.py", line 32, in _send_request
request = self.endpoint.create_request(request_dict, operation_model)
File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 126, in create_request
operation_name=operation_model.name)
File "/usr/local/lib/python2.7/dist-packages/botocore/hooks.py", line 226, in emit
return self._emit(event_name, kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/hooks.py", line 209, in _emit
response = handler(**kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/signers.py", line 90, in handler
return self.sign(operation_name, request)
File "/usr/local/lib/python2.7/dist-packages/botocore/signers.py", line 124, in sign
signer.add_auth(request=request)
File "/usr/local/lib/python2.7/dist-packages/botocore/auth.py", line 626, in add_auth
raise NoCredentialsError
NoCredentialsError: Unable to locate credentials
Could it be fixed with proper ordering in which I install libraries? Because the pip library removes existing newer version of botocore and installs an older version.
EDIT:
I am running processes with supervisor and it seems process cant access aws credentials
EDIT 2:
The issue got resolved with proper configuration of supervisor. The user for process started by supervisor did not have access to config file
The issue got resolved with proper configuration of supervisor. The user for subprocess started by supervisor did not have access to aws config file. So it was working with local environment or creating process separately but not with supervisor.