boto3 can not find the config profile - amazon-web-services

Question

my boto3 setup works with providing the credentials through /.aws/credentials - how ever I would like to pass that as environmental variable to work through docker.
this is the content of /.aws/credentials
[default]
aws_access_key_id = ABCD123
aws_secret_access_key = BCDSA123
[testing]
source_profile = default
role_arn = arn:aws:iam::123:role/access-db
everything works - but if I make a setup env variable in a aws.env file
export AWS_ACCESS_KEY_ID="ABCD123"
export AWS_SECRET_ACCESS_KEY="BCDSA123"
export AWS_ROLE_SESSION_NAME="default"
export AWS_PROFILE="testing"
export AWS_DEFAULT_PROFILE="testing"
export AWS_ROLE_ARN="arn:aws:iam::123:role/access-db"
I get the following error
root#64813e0cc755:/rate_prediction# python test.py
Traceback (most recent call last):
File "test.py", line 4, in <module>
boto3.setup_default_session(profile_name = 'testing')
File "/usr/local/lib/python3.7/site-packages/boto3/__init__.py", line 34, in setup_default_session
DEFAULT_SESSION = Session(**kwargs)
File "/usr/local/lib/python3.7/site-packages/boto3/session.py", line 80, in __init__
self._setup_loader()
File "/usr/local/lib/python3.7/site-packages/boto3/session.py", line 120, in _setup_loader
self._loader = self._session.get_component('data_loader')
File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 685, in get_component
return self._components.get_component(name)
File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 924, in get_component
self._components[name] = factory()
File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 158, in <lambda>
lambda: create_loader(self.get_config_variable('data_path')))
File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 241, in get_config_variable
logical_name)
File "/usr/local/lib/python3.7/site-packages/botocore/configprovider.py", line 293, in get_config_variable
return provider.provide()
File "/usr/local/lib/python3.7/site-packages/botocore/configprovider.py", line 390, in provide
value = provider.provide()
File "/usr/local/lib/python3.7/site-packages/botocore/configprovider.py", line 451, in provide
scoped_config = self._session.get_scoped_config()
File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 340, in get_scoped_config
raise ProfileNotFound(profile=profile_name)
botocore.exceptions.ProfileNotFound: The config profile (testing) could not be found
and here is my python script
import boto3
import os
boto3.setup_default_session(profile_name = 'testing')
s3 = boto3.resource('s3')
s3_client = boto3.client('s3')
my_bucket = s3.Bucket('ds-models-testing')
which I am expecting to work with - again, seem there is sth missing from my environmental variable and boto is searching for a config file botocore.exceptions.ProfileNotFound: The config profile (testing) could not be foun

Answer 1

You need to unset the AWS_PROFILE env var.

Answer 2

You don't have config file. config and credentials file should be in ~/.aws folder.
.aws/credentials
[default]
aws_access_key_id = ABCD123
aws_secret_access_key = BCDSA123
and in config file .aws/config
[profile testing]
source_profile = default
role_arn = arn:aws:iam::123:role/access-db

Answer 3

First, I think you should remove this line if you are trying to read from environment variables:
boto3.setup_default_session(profile_name = 'testing')
Documentation says
The order in which Boto3 searches for credentials is:
1. Passing credentials as parameters in the boto.client() method
2. Passing credentials as parameters when creating a Session object
3. Environment variables
4. Shared credential file (~/.aws/credentials)
5. AWS config file (~/.aws/config)
6. Assume Role provider
7. Boto2 config file (/etc/boto.cfg and ~/.boto)
8. Instance metadata service on an Amazon EC2 instance that has an IAM role configured.
Reference: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html
So as you suspect, most likely the environment variables are not getting set as you expect. Try adding code to log the value of an environment variable that you are setting via the app.env file just to be sure you actually see it set and can read it as an environment variable:
os.environ.get('AWS_ACCESS_KEY_ID', 'It is not set!')
Assuming it is actually reading the environment variables you are setting in aws.env, then try exporting only these 2 environment variables in your aws.env and removing the others:
export AWS_ACCESS_KEY_ID="ABCD123"
export AWS_SECRET_ACCESS_KEY="BCDSA123"