eb labs download not working in some AWS regions - amazon-web-services

I am new to dealing with the Parse-Server and hosting on AWS. But I have noticed that the "eb labs download" command in terminal works when my Parse Server environment is in N.Virginia but comes back with a whole list of errors when the the server environment was initially kept in Oregon. The errors are pertaining to "HTTP header errors". Anyone know why this is happening? Thanks in advance! Error is below:
Downloading application version...
Traceback (most recent call last):
File "/usr/local/bin/eb", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/site-packages/ebcli/core/ebcore.py", line 150, in main
app.run()
File "/usr/local/lib/python2.7/site-packages/cement/core/foundation.py", line 797, in run
return_val = self.controller._dispatch()
File "/usr/local/lib/python2.7/site-packages/cement/core/controller.py", line 472, in _dispatch
return func()
File "/usr/local/lib/python2.7/site-packages/cement/core/controller.py", line 472, in _dispatch
return func()
File "/usr/local/lib/python2.7/site-packages/cement/core/controller.py", line 478, in _dispatch
return func()
File "/usr/local/lib/python2.7/site-packages/ebcli/core/abstractcontroller.py", line 57, in default
self.do_command()
File "/usr/local/lib/python2.7/site-packages/ebcli/labs/download.py", line 36, in do_command
download_source_bundle(app_name, env_name)
File "/usr/local/lib/python2.7/site-packages/ebcli/labs/download.py", line 49, in download_source_bundle
data = s3.get_object(bucket_name, key_name)
File "/usr/local/lib/python2.7/site-packages/ebcli/lib/s3.py", line 68, in get_object
Key=key)
File "/usr/local/lib/python2.7/site-packages/ebcli/lib/s3.py", line 34, in _make_api_call
return aws.make_api_call('s3', operation_name, **operation_options)
File "/usr/local/lib/python2.7/site-packages/ebcli/lib/aws.py", line 218, in make_api_call
response_data = operation(**operation_options)
File "/Users/Home/Library/Python/2.7/lib/python/site-packages/botocore/client.py", line 251, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/Users/Home/Library/Python/2.7/lib/python/site-packages/botocore/client.py", line 526, in _make_api_call
operation_model, request_dict)
File "/Users/Home/Library/Python/2.7/lib/python/site-packages/botocore/endpoint.py", line 141, in make_request
return self._send_request(request_dict, operation_model)
File "/Users/Home/Library/Python/2.7/lib/python/site-packages/botocore/endpoint.py", line 170, in _send_request
success_response, exception):
File "/Users/Home/Library/Python/2.7/lib/python/site-packages/botocore/endpoint.py", line 249, in _needs_retry
caught_exception=caught_exception, request_dict=request_dict)
File "/Users/Home/Library/Python/2.7/lib/python/site-packages/botocore/hooks.py", line 227, in emit
return self._emit(event_name, kwargs)
File "/Users/Home/Library/Python/2.7/lib/python/site-packages/botocore/hooks.py", line 210, in _emit
response = handler(**kwargs)
File "/Users/Home/Library/Python/2.7/lib/python/site-packages/botocore/utils.py", line 868, in redirect_from_error
new_region = self.get_bucket_region(bucket, response)
File "/Users/Home/Library/Python/2.7/lib/python/site-packages/botocore/utils.py", line 913, in get_bucket_region
response_headers = service_response['ResponseMetadata']['HTTPHeaders']
KeyError: 'HTTPHeaders'

Generally the EB CLI works in a single region at a time. If you have a specific region in which you want to use you can specify it using the --region flag.
eb labs download --region us-west-2
Otherwise it is usually best practice to keep your AWS stack resources in a single region.

Related

Localstack : Creating stack from cloudformation file which refer to some existing reources

I am trying to deploy a stack using localstack.
My cloudformation file refer to few existing resources which are already created on the dev environment.
While deploying my cloudformation I am getting below errors:
2022-10-03T10:17:26.431 DEBUG --- [ Thread-256] l.u.c.template_deployer : Error applying changes for CloudFormation stack "ashford1": An error occurred (InternalError) when calling the AttachRolePolicy operation (reached max retries: 4): exception while calling iam.AttachRolePolicy: Traceback (most recent call last):
File "/opt/code/localstack/localstack/aws/chain.py", line 90, in handle
handler(self, self.context, response)
File "/opt/code/localstack/localstack/aws/handlers/service.py", line 122, in __call__
handler(chain, context, response)
File "/opt/code/localstack/localstack/aws/handlers/service.py", line 92, in __call__
skeleton_response = self.skeleton.invoke(context)
File "/opt/code/localstack/localstack/aws/skeleton.py", line 153, in invoke
return self.dispatch_request(context, instance)
File "/opt/code/localstack/localstack/aws/skeleton.py", line 165, in dispatch_request
result = handler(context, instance) or {}
File "/opt/code/localstack/localstack/aws/forwarder.py", line 60, in _call
return fallthrough_handler(context, req)
File "/opt/code/localstack/localstack/services/moto.py", line 83, in _proxy_moto
return call_moto(context)
File "/opt/code/localstack/localstack/services/moto.py", line 46, in call_moto
return dispatch_to_backend(context, dispatch_to_moto, include_response_metadata)
File "/opt/code/localstack/localstack/aws/forwarder.py", line 113, in dispatch_to_backend
http_response = http_request_dispatcher(context)
File "/opt/code/localstack/localstack/services/moto.py", line 111, in dispatch_to_moto
status, headers, content = dispatch(request, request.url, request.headers)
File "/opt/code/localstack/.venv/lib/python3.10/site-packages/moto/core/responses.py", line 217, in dispatch
return cls()._dispatch(*args, **kwargs)
File "/opt/code/localstack/.venv/lib/python3.10/site-packages/moto/core/responses.py", line 356, in _dispatch
return self.call_action()
File "/opt/code/localstack/.venv/lib/python3.10/site-packages/moto/core/responses.py", line 443, in call_action
response = method()
File "/opt/code/localstack/.venv/lib/python3.10/site-packages/moto/iam/responses.py", line 17, in attach_role_policy
self.backend.attach_role_policy(policy_arn, role_name)
File "/opt/code/localstack/.venv/lib/python3.10/site-packages/moto/iam/models.py", line 1689, in attach_role_policy
policy = arns[policy_arn]
KeyError: 'arn:aws:iam::XXXXX:policy/BasePolicy-Default-XXX'
Traceback (most recent call last):
File "/opt/code/localstack/localstack/utils/cloudformation/template_deployer.py", line 1569, in _run
self.do_apply_changes_in_loop(changes, stack)
File "/opt/code/localstack/localstack/utils/cloudformation/template_deployer.py", line 1646, in do_apply_changes_in_loop
self.apply_change(change, stack=stack)
File "/opt/code/localstack/localstack/utils/cloudformation/template_deployer.py", line 1723, in apply_change
result = deploy_resource(self, resource_id)
File "/opt/code/localstack/localstack/utils/cloudformation/template_deployer.py", line 853, in deploy_resource
result = execute_resource_action(resource_id, stack, ACTION_CREATE)
File "/opt/code/localstack/localstack/utils/cloudformation/template_deployer.py", line 891, in execute_resource_action
result = func["function"](resource_id, resources, resource_type, func, stack_name)
File "/opt/code/localstack/localstack/services/cloudformation/models/iam.py", line 294, in _post_create
iam.attach_role_policy(RoleName=role_name, PolicyArn=arn)
File "/opt/code/localstack/.venv/lib/python3.10/site-packages/botocore/client.py", line 514, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/opt/code/localstack/.venv/lib/python3.10/site-packages/botocore/client.py", line 938, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InternalError) when calling the AttachRolePolicy operation (reached max retries: 4): exception while calling iam.AttachRolePolicy: Traceback (most recent call last):
File "/opt/code/localstack/localstack/aws/chain.py", line 90, in handle
handler(self, self.context, response)
File "/opt/code/localstack/localstack/aws/handlers/service.py", line 122, in __call__
handler(chain, context, response)
File "/opt/code/localstack/localstack/aws/handlers/service.py", line 92, in __call__
skeleton_response = self.skeleton.invoke(context)
File "/opt/code/localstack/localstack/aws/skeleton.py", line 153, in invoke
return self.dispatch_request(context, instance)
File "/opt/code/localstack/localstack/aws/skeleton.py", line 165, in dispatch_request
result = handler(context, instance) or {}
File "/opt/code/localstack/localstack/aws/forwarder.py", line 60, in _call
return fallthrough_handler(context, req)
File "/opt/code/localstack/localstack/services/moto.py", line 83, in _proxy_moto
return call_moto(context)
File "/opt/code/localstack/localstack/services/moto.py", line 46, in call_moto
return dispatch_to_backend(context, dispatch_to_moto, include_response_metadata)
File "/opt/code/localstack/localstack/aws/forwarder.py", line 113, in dispatch_to_backend
http_response = http_request_dispatcher(context)
File "/opt/code/localstack/localstack/services/moto.py", line 111, in dispatch_to_moto
status, headers, content = dispatch(request, request.url, request.headers)
File "/opt/code/localstack/.venv/lib/python3.10/site-packages/moto/core/responses.py", line 217, in dispatch
return cls()._dispatch(*args, **kwargs)
File "/opt/code/localstack/.venv/lib/python3.10/site-packages/moto/core/responses.py", line 356, in _dispatch
return self.call_action()
File "/opt/code/localstack/.venv/lib/python3.10/site-packages/moto/core/responses.py", line 443, in call_action
response = method()
File "/opt/code/localstack/.venv/lib/python3.10/site-packages/moto/iam/responses.py", line 17, in attach_role_policy
self.backend.attach_role_policy(policy_arn, role_name)
File "/opt/code/localstack/.venv/lib/python3.10/site-packages/moto/iam/models.py", line 1689, in attach_role_policy
policy = arns[policy_arn]
KeyError: 'arn:aws:iam::XXXXXXXXXXXX:policy/Policy-Default-XXXX'
My questions:
Should I create the existing resources manually before deploying the stack?
Can I create the stack with multiple resource files places in cloudformation.d folder? As of now, I need to create a single cloudformation.yml file and use that to deploy stack.
Thanks for your time and help.

textract_python_table_parser.py command prompt lacking credentials

I'm trying to put to work AWS's Textract export table suggestion in this link
I'm a complete newbie in AWS's solutions and in command prompt so I'm trying to do exactly as they suggest. I'm running that in python so I'm using this piece of code:
import os
k=os.system("python textract_python_table_parser.py my_pdf_file_path.pdf")
print(k)
The code runs, I get an Image loaded my_pdf_file_path.pdf however at some point it bugs on credential matters:
Traceback (most recent call last):
File "/Users/santanna_santanna/PycharmProjects/KlooksExplore/PDFWork/textract_python_table_parser.py", line 108, in <module>
main(file_name)
File "/Users/santanna_santanna/PycharmProjects/KlooksExplore/PDFWork/textract_python_table_parser.py", line 94, in main
table_csv = get_table_csv_results(file_name)
File "/Users/santanna_santanna/PycharmProjects/KlooksExplore/PDFWork/textract_python_table_parser.py", line 53, in get_table_csv_results
response = client.analyze_document(Document={'Bytes': bytes_test}, FeatureTypes=['TABLES'])
File "/Users/santanna_santanna/anaconda3/lib/python3.6/site-packages/botocore/client.py", line 316, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/Users/santanna_santanna/anaconda3/lib/python3.6/site-packages/botocore/client.py", line 622, in _make_api_call
operation_model, request_dict, request_context)
File "/Users/santanna_santanna/anaconda3/lib/python3.6/site-packages/botocore/client.py", line 641, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/Users/santanna_santanna/anaconda3/lib/python3.6/site-packages/botocore/endpoint.py", line 102, in make_request
return self._send_request(request_dict, operation_model)
File "/Users/santanna_santanna/anaconda3/lib/python3.6/site-packages/botocore/endpoint.py", line 132, in _send_request
request = self.create_request(request_dict, operation_model)
File "/Users/santanna_santanna/anaconda3/lib/python3.6/site-packages/botocore/endpoint.py", line 116, in create_request
operation_name=operation_model.name)
File "/Users/santanna_santanna/anaconda3/lib/python3.6/site-packages/botocore/hooks.py", line 356, in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "/Users/santanna_santanna/anaconda3/lib/python3.6/site-packages/botocore/hooks.py", line 228, in emit
return self._emit(event_name, kwargs)
File "/Users/santanna_santanna/anaconda3/lib/python3.6/site-packages/botocore/hooks.py", line 211, in _emit
response = handler(**kwargs)
File "/Users/santanna_santanna/anaconda3/lib/python3.6/site-packages/botocore/signers.py", line 90, in handler
return self.sign(operation_name, request)
File "/Users/santanna_santanna/anaconda3/lib/python3.6/site-packages/botocore/signers.py", line 160, in sign
auth.add_auth(request)
File "/Users/santanna_santanna/anaconda3/lib/python3.6/site-packages/botocore/auth.py", line 357, in add_auth
raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials
I'm aware I didn't pass any credentials and that's natural to happen but where should I pass it and what would be the right syntax for that using python os? Amazon's example doesn't say anything about that.
It depends where you run your code, for example:
local computer - can use aws configure CLI to set your credetnails
EC2 instance - use instance role
lambda function - use lambda execution role

Adjust node configuration breaks sqlproxy and scheduler

Today I tried to change the node type of the cluster backing a cloud composer environment and switch to the Ubuntu image instead of COS. I did so by adding a second node pool to the GKE cluster, then deleting the first one and have all workloads migrated.
This spawns following errors in the airflow-sqlproxy logs:
couldn't connect to "XXXXX:europe-west1:XXXXX": ensure that the Cloud SQL API is enabled for your project (https://console.cloud.google.com/flows/enableapi?apiid=sqladmin). Error during createEphemeral for XXXXX:europe-west1:XXXXX: googleapi: Error 403: Insufficient Permission, insufficientPermissions
The scheduler fails to start completely and emits following stacktraces:
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 27, in <module>
args.func(args)
File "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 826, in scheduler
job.run()
File "/usr/local/lib/python2.7/site-packages/airflow/jobs.py", line 192, in run
session.commit()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 943, in commit
self.transaction.commit()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 467, in commit
self._prepare_impl()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 447, in _prepare_impl
self.session.flush()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2254, in flush
self._flush(objects)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2380, in _flush
transaction.rollback(_capture_exception=True)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2344, in _flush
flush_context.execute()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 391, in execute
rec.execute(self)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 556, in execute
uow
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 156, in save_obj
base_mapper, states, uowtransaction
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 286, in _organize_states_for_save(states):
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 1252, in _connections_for_states
connection = uowtransaction.transaction.connection(base_mapper)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 294, in connection
return self._connection_for_bind(bind, execution_options)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind
conn = self._parent._connection_for_bind(bind, execution_options)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind
conn = bind.contextual_connect()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect
e, dialect, self)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception_noconnection
exc_info
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect
return fn()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect
return _ConnectionFairy._checkout(self)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 788, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 532, in checkout
rec = pool._do_get()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 1193, in _do_get
self._dec_overflow()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 1190, in _do_get
return self._create_connection()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 350, in _create_connection
return _ConnectionRecord(self)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 477, in __init__
self.__connect(first_connect_check=True)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 671, in __connect
connection = pool._invoke_creator(self)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 106, in connect
return dialect.connect(*cargs, **cparams)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/usr/local/lib/python2.7/site-packages/MySQLdb/__init__.py", line 86, in Connect
return Connection(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/MySQLdb/connections.py", line 204, in __init__
super(Connection, self).__init__(*args, **kwargs2)
sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (2013, "Lost connection to MySQL server at 'reading initial communication packet', system error: 0") (Background on this error at: http://sqlalche.me/e/e3q8)
It seems that the connection to the backing SQL database is now broken. Its still the same cluster but the nodes are different. Are there any additional configurations I must update?
The SQLProxy relies on the credentials of the service account used to create the Composer environment. If you do not change any settings, this should be the Compute engine default service account.
You should verify that the new node pool and your previous Composer node pool share the same service account.
Additionally, you should verify the new pool has sufficient scopes-- you are likely missing the sql admin scope. See https://cloud.google.com/sdk/gcloud/reference/container/node-pools/create?hl=en_US&_ga=2.222157720.-1458722175.1530287449

SSLHandshakeError with httplib2.python2 while installing Google Cloud SDK

I'm having trouble running the install script for Google Cloud SDK - this question has been asked but without clear answers so far so any help would be welcome! I'm running Python 2.7.14.
This is the error I'm getting:
httplib2.python2.httplib2.SSLHandshakeError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)
The full stack trace is:
Welcome to the Google Cloud SDK!
To help improve the quality of this product, we collect anonymized usage data
and anonymized stacktraces when crashes are encountered; additional information
is available at <https://cloud.google.com/sdk/usage-statistics>. You may choose
to opt out of this collection now (by choosing 'N' at the below prompt), or at
any time in the future by running the following command:
gcloud config set disable_usage_reporting true
Do you want to help improve the Google Cloud SDK (Y/n)? Y
Traceback (most recent call last):
File "/Users/<user>/Downloads/google-cloud-sdk/bin/bootstrapping/install.py", line 218, in <module>
main()
File "/Users/<user>/Downloads/google-cloud-sdk/bin/bootstrapping/install.py", line 196, in main
Install(pargs.override_components, pargs.additional_components)
File "/Users/<user>/Downloads/google-cloud-sdk/bin/bootstrapping/install.py", line 141, in Install
_CLI.Execute(['--quiet', 'components', 'list'])
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 870, in Execute
self._HandleAllErrors(exc, command_path_string, specified_arg_names)
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 908, in _HandleAllErrors
exceptions.HandleError(exc, command_path_string, self.__known_error_handler)
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/calliope/exceptions.py", line 526, in HandleError
core_exceptions.reraise(exc)
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/core/exceptions.py", line 111, in reraise
six.reraise(type(exc_value), exc_value, tb)
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 844, in Execute
resources = calliope_command.Run(cli=self, args=args)
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 756, in Run
resources = command_instance.Run(args)
File "/Users/<user>/Downloads/google-cloud-sdk/lib/surface/components/list.py", line 98, in Run
only_local_state=args.only_local_state)
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/core/updater/update_manager.py", line 674, in List
to_print, current_version, latest_version = self._GetPrintListWithDiff()
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/core/updater/update_manager.py", line 704, in _GetPrintListWithDiff
_, diff = self._GetStateAndDiff(command_path='components.list')
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/core/updater/update_manager.py", line 600, in _GetStateAndDiff
command_path=command_path)
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/core/updater/update_manager.py", line 583, in _GetLatestSnapshot
*effective_url.split(','), command_path=command_path)
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/core/updater/snapshots.py", line 178, in FromURLs
for url in urls]
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/core/updater/snapshots.py", line 199, in _DictFromURL
response = installers.ComponentInstaller.MakeRequest(url, command_path)
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/core/updater/installers.py", line 293, in MakeRequest
return ComponentInstaller._RawRequest(req, timeout=timeout)
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/core/updater/installers.py", line 337, in _RawRequest
should_retry_if=RetryIf, sleep_ms=500)
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/core/util/retry.py", line 178, in RetryOnException
exceptions.reraise(exc_info[1], tb=exc_info[2])
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/core/exceptions.py", line 111, in reraise
six.reraise(type(exc_value), exc_value, tb)
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/core/util/retry.py", line 159, in TryFunc
return func(*args, **kwargs), None
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/core/url_opener.py", line 78, in urlopen
return opener.open(req, data, timeout)
File "/usr/local/Cellar/python#2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 429, in open
response = self._open(req, data)
File "/usr/local/Cellar/python#2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 447, in _open
'_open', req)
File "/usr/local/Cellar/python#2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/Users/<user>/Downloads/google-cloud-sdk/lib/googlecloudsdk/core/url_opener.py", line 63, in https_open
return self.do_open(build, req)
File "/usr/local/Cellar/python#2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1195, in do_open
h.request(req.get_method(), req.get_selector(), req.data, headers)
File "/usr/local/Cellar/python#2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1042, in request
self._send_request(method, url, body, headers)
File "/usr/local/Cellar/python#2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1082, in _send_request
self.endheaders(body)
File "/usr/local/Cellar/python#2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1038, in endheaders
self._send_output(message_body)
File "/usr/local/Cellar/python#2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 882, in _send_output
self.send(msg)
File "/usr/local/Cellar/python#2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 844, in send
self.connect()
File "/Users/<user>/Downloads/google-cloud-sdk/lib/third_party/httplib2/python2/httplib2/__init__.py", line 1120, in connect
raise SSLHandshakeError(e)
httplib2.python2.httplib2.SSLHandshakeError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)
This type of error is usually encountered due to a number of reasons, including the following:
a. Networking configurations and any proxy that can cause the handshake to fail.
b. Antivirus, firewalls, or any other software that are preventing the traffic.
C. Corporate certificate is being used.
Depending on the situation, you can check your proxy and ensure that traffic/connections to accounts.google.com are allowed for authorization. To use corporate certificates, it is necessary to append your corporate certificate to google-cloud-sdk/lib/third_party/httplib2/cacerts.txt to allow connections from Google cloud.

aws no credentials error

I am trying to setup dynamic thumbnail service thumbor and to support s3 as storage, I need to setup this community powered pip library for aws.
Its working well on my local environment but when I am trying to host it on one of our servers, I am getting NoCredentialsError. I am assuming this is because of difference versions of botocore (latest one and one installed by pip library). Here is error log:
File "/usr/local/lib/python2.7/dist-packages/botocore/session.py", line 774, in get_component
# client config from the session
File "/usr/local/lib/python2.7/dist-packages/botocore/session.py", line 174, in <lambda>
self._components.lazy_register_component(
File "/usr/local/lib/python2.7/dist-packages/botocore/session.py", line 453, in get_data
- agent_version is the value of the `user_agent_version`
File "/usr/local/lib/python2.7/dist-packages/botocore/loaders.py", line 119, in _wrapper
data = func(self, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/loaders.py", line 364, in load_data
DataNotFoundError: Unable to load data for: _endpoints
2016-04-24 12:14:34 tornado.application:ERROR Future exception was never retrieved: Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 230, in wrapper
yielded = next(result)
File "/usr/local/lib/python2.7/dist-packages/thumbor/handlers/imaging.py", line 31, in check_image
exists = yield gen.maybe_future(self.context.modules.storage.exists(kw['image'][:self.context.config.MAX_ID_LENGTH]))
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 455, in wrapper
future.result()
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 215, in result
raise_exc_info(self._exc_info)
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 443, in wrapper
result = f(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tc_aws/aws/storage.py", line 107, in exists
self.storage.get(file_abspath, callback=return_data)
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 455, in wrapper
future.result()
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 215, in result
raise_exc_info(self._exc_info)
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 443, in wrapper
result = f(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tc_aws/aws/bucket.py", line 44, in get
Key=self._clean_key(path),
File "/usr/local/lib/python2.7/dist-packages/tornado_botocore/base.py", line 97, in call
return self._make_api_call(operation_name=self.operation, api_params=kwargs, callback=callback)
File "/usr/local/lib/python2.7/dist-packages/tornado_botocore/base.py", line 60, in _make_api_call
operation_model=operation_model, request_dict=request_dict, callback=callback)
File "/usr/local/lib/python2.7/dist-packages/tornado_botocore/base.py", line 54, in _make_request
request_dict=request_dict, operation_model=operation_model, callback=callback)
File "/usr/local/lib/python2.7/dist-packages/tornado_botocore/base.py", line 32, in _send_request
request = self.endpoint.create_request(request_dict, operation_model)
File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 126, in create_request
operation_name=operation_model.name)
File "/usr/local/lib/python2.7/dist-packages/botocore/hooks.py", line 226, in emit
return self._emit(event_name, kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/hooks.py", line 209, in _emit
response = handler(**kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/signers.py", line 90, in handler
return self.sign(operation_name, request)
File "/usr/local/lib/python2.7/dist-packages/botocore/signers.py", line 124, in sign
signer.add_auth(request=request)
File "/usr/local/lib/python2.7/dist-packages/botocore/auth.py", line 626, in add_auth
raise NoCredentialsError
NoCredentialsError: Unable to locate credentials
Could it be fixed with proper ordering in which I install libraries? Because the pip library removes existing newer version of botocore and installs an older version.
EDIT:
I am running processes with supervisor and it seems process cant access aws credentials
EDIT 2:
The issue got resolved with proper configuration of supervisor. The user for process started by supervisor did not have access to config file
The issue got resolved with proper configuration of supervisor. The user for subprocess started by supervisor did not have access to aws config file. So it was working with local environment or creating process separately but not with supervisor.