Deepracer log_analysis tool - sagemaker role errors - amazon-web-services

I'm trying to run the Deepracer log analysis tool from https://github.com/aws-samples/aws-deepracer-workshops/blob/master/log-analysis/DeepRacer%20Log%20Analysis.ipynb on my local laptop. However I get below error while trying to run step [5] "Create an IAM role".
try:
sagemaker_role = sagemaker.get_execution_role()
except:
sagemaker_role = get_execution_role('sagemaker')
print("Using Sagemaker IAM role arn: \n{}".format(sagemaker_role))
Couldn't call 'get_role' to get Role ARN from role name arn:aws:iam::26********:root to get Role path.
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-5-3bea8175b8c7> in <module>
1 try:
----> 2 sagemaker_role = sagemaker.get_execution_role()
3 except:
/opt/conda/lib/python3.7/site-packages/sagemaker/session.py in get_execution_role(sagemaker_session)
3302 )
-> 3303 raise ValueError(message.format(arn))
3304
ValueError: The current AWS identity is not a role: arn:aws:iam::26********:root, therefore it cannot be used as a SageMaker execution role
During handling of the above exception, another exception occurred:
NameError Traceback (most recent call last)
<ipython-input-5-3bea8175b8c7> in <module>
2 sagemaker_role = sagemaker.get_execution_role()
3 except:
----> 4 sagemaker_role = get_execution_role('sagemaker')
5
6 print("Using Sagemaker IAM role arn: \n{}".format(sagemaker_role))
NameError: name 'get_execution_role' is not defined
Does anybody know what needs to be done to execute above code without errors?

AWS support recommended below solution:
This seems to be a known issue when executing the code locally, as mentioned in the following Github issue [3]. A work-around to fix the issue is also defined in that issue [3] and can be referred to using the following link: aws/sagemaker-python-sdk#300 (comment)
The steps in the work-around given in the above link are:
Login to the AWS console -> IAM -> Roles -> Create Role
Create an IAM role and select the "SageMaker" service
Give the role "AmazonSageMakerFullAccess" permission
Review and create the role
Next, also attach the "AWSRoboMakerFullAccess" permission policy to the above created role (as required in the Github notebook [1]).
The original code would then need to be modified to fetch the IAM role directly when the code is executed on a local machine. The code snippet to be used is given below:
try:
sagemaker_role = sagemaker.get_execution_role()
except ValueError:
iam = boto3.client('iam')
sagemaker_role = iam.get_role(RoleName='<sagemaker-IAM-role-name>')['Role']['Arn']
In the above snippet, replace the "" text with the IAM role name created in Step 4.
References:
[1] https://github.com/aws-samples/aws-deepracer-workshops/blob/master/log-analysis/DeepRacer%20Log%20Analysis.ipynb
[2] https://docs.aws.amazon.com/sagemaker/latest/dg/automatic-model-tuning-ex-role.html
[3] aws/sagemaker-python-sdk#300

Related

how to use pageSize or pageToken in python code for GCP

this is the python code I got from github. running it, I got 300. But when I use gcloud to get role number, I got a total of 479 roles. I was told by the GCP support that pageSize needs to be used. where can I find documentation of how and pageSize can be used? so in my code below, where should pageSize go? or perhaps pageToken needs to be used?
(gcptest):$ gcloud iam roles list |grep name |wc -l
479
(gcptest) : $ python quickstart.py
300
def quickstart():
# [START iam_quickstart]
import os
from google.oauth2 import service_account
import googleapiclient.discovery
import pprint
# Get credentials
credentials = service_account.Credentials.from_service_account_file(
filename=os.environ['GOOGLE_APPLICATION_CREDENTIALS'],
scopes=['https://www.googleapis.com/auth/cloud-platform'])
# Create the Cloud IAM service object
service = googleapiclient.discovery.build(
'iam', 'v1', credentials=credentials)
# Call the Cloud IAM Roles API
# If using pylint, disable weak-typing warnings
# pylint: disable=no-member
response = service.roles().list().execute()
roles = response['roles']
print(type(roles))
print(len(roles))
if name == 'main':
quickstart()
You will need to write code similar to this:
roles = service.roles()
request = roles.list()
while request is not None:
role_list = request.execute()
# process each role here
for role in role_list:
print(role)
# Get next page of results
request = roles.list_next(request, role_list)
Documentation link for the list_next method
In addition of the solution of #JohnHanley, you can also add the queries parameters in parameter of your method. Like this
# Page size of 10
response = service.roles().list(pageSize=10).execute()
Here the definition of this list method

Permission denied for Cloud Function to decrypt with KMS key

I have a Python Cloud Function which uses a KMS key to decrypt some authentication tokens for other services from the environment, as in https://dev.to/googlecloud/using-secrets-in-google-cloud-functions-5aem
I keep getting a 403 Permission Denied whenever I run my function. When I call the function locally on my computer it works fine. I've tried adding the "Cloud KMS CryptoKey Decrypter" role to the default Compute Engine service account but that didn't work.
Any other ideas?
Edit: here's some code that shows what I'm doing. The environment variables are stored in an environment.yaml file which I point to when I gcloud functions deploy
def decrypt_secret(key: str, secret: str):
kms_client = kms.KeyManagementServiceClient()
decrypted = kms_client.decrypt(key, base64.b64decode(secret))
return decrypted.plaintext.decode("ascii")
def do_kms_stuff():
key = os.environ["KMS_RESOURCE_NAME"]
session = boto3.Session(
profile_name="my-profile",
aws_access_key_id=decrypt_secret(
key, os.environ["AWS_ACCESS_KEY_ID_ENCRYPTED"]
),
aws_secret_access_key=decrypt_secret(
key, os.environ["AWS_SECRET_ACCESS_KEY_ENCRYPTED"]
),
)
# ...
And here's the error from the Cloud Functions console:
File "<string>", line 3, in raise_from: google.api_core.exceptions.PermissionDenied: 403 Permission 'cloudkms.cryptoKeyVersions.useToDecrypt' denied on resource 'projects/my-project/locations/my-location1/keyRings/my-keyring/cryptoKeys/my-key' (or it may not exist). at error_remapped_callable (/env/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py:59) at func_with_timeout (/env/local/lib/python3.7/site-packages/google/api_core/timeout.py:214) at retry_target (/env/local/lib/python3.7/site-packages/google/api_core/retry.py:182) at retry_wrapped_func (/env/local/lib/python3.7/site-packages/google/api_core/retry.py:277) at
__call__ (/env/local/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py:143) at decrypt (/env/local/lib/python3.7/site-packages/google/cloud/kms_v1/gapic/key_management_service_client.py:1816) at decrypt_secret (/user_code/kms_stuff.py:17) at do_kms_stuff (/user_code/kms_stuff.py:48) at my_cloud_function (/user_code/main.py:46) at call_user_function (/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py:214) at invoke_user_function (/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py:217) at run_background_function (/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py:383)
As #DazWilkin and #pessolato mentioned, the issue was that I was using the wrong service account. Once I changed to use the default AppSpot account everything worked smoothly.

What I am doing wrong by allowing user to download file from AWS with django boto3?

Hello Awesome People!
I will try to be clear with my question.
All of my media files are uploaded to AWS, I created a view that allows each user to download images. Before, I did it without the boto that summed up
"this back-end doesn't support absolute path."
And now, after some research, I'm using the s3 boto3 connection.
Model
class Photo(models.Model):
file = models.ImageField(upload_to="uploaded_documents/")
total_download = models.PositiveIntegerField()
View
def download_file(request,id):
photo = get_object_or_404(Photo,id=id)
photo.total_download += 1
photo.save()
path = os.path.basename(photo.file.name)
# path = '/media/public/uploaded_documents/museum.jpg'
client = boto3.client('s3',aws_access_key_id=settings.AWS_ACCESS_KEY_ID,
aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY)
resource = boto3.resource('s3')
bucket = resource.Bucket(settings.AWS_STORAGE_BUCKET_NAME)
bucket.download_file(path, 'my_local_image.jpg')
Here I don't know what to do to trigger it. when I run it, I get the following error:
NoCredentialsError at /api/download-file/75
Exception Type: NoCredentialsError
Exception Value: Unable to locate credentials
UPDATE
I use the credentials in resources instead of client
client = boto3.client('s3')
resource = boto3.resource('s3',aws_access_key_id=settings.AWS_ACCESS_KEY_ID,
aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY)
and it seems to be authenticated. But now I got the error:
Exception Type: ClientError
Exception Value: An error occurred (404) when calling the HeadObject operation: Not Found
please try looking at the answers in:
Boto3 Error: botocore.exceptions.NoCredentialsError: Unable to locate credentials
also check that the file path is correct in S3

Can't read from tfrecords in S3 from notebook instance

I try to read from tfrecords in S3 from a Sage Maker notebook instance following instructions here: https://www.tensorflow.org/versions/master/deploy/s3
import tensorflow as tf
import os
os.environ['AWS_ACCESS_KEY_ID'] = '<my-key>'
os.environ['AWS_SECRET_ACCESS_KEY'] = '<my-secret>'
from tensorflow.python.lib.io import file_io
print(file_io.stat('s3://<my-bucket>/data/DEMO-mnist/train.tfrecords'))
The above code fails with the error:
---------------------------------------------------------------------------
NotFoundError Traceback (most recent call last)
<ipython-input-7-770c0aef6d7b> in <module>()
1 from tensorflow.python.lib.io import file_io
----> 2 print(file_io.stat('s3://<my-bucket>/data/DEMO-mnist/train.tfrecords'))
~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/lib/io/file_io.py in stat(filename)
551 with errors.raise_exception_on_not_ok_status() as status:
552 pywrap_tensorflow.Stat(compat.as_bytes(filename), file_statistics, status)
--> 553 return file_statistics
~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py in __exit__(self, type_arg, value_arg, traceback_arg)
517 None, None,
518 compat.as_text(c_api.TF_Message(self.status.status)),
--> 519 c_api.TF_GetCode(self.status.status))
520 # Delete the underlying status object from memory otherwise it stays alive
521 # as there is a reference to status from this from the traceback due to
NotFoundError: Object s3://<my-bucket>/data/DEMO-mnist/train.tfrecords does not exist
However the same code works fine if I run from a regular EC2 instance without using SageMaker.
IAM role used for the notebook instance has full S3 access.
I reproduced the problem in us-west-2.
But after I manually export environment variable AWS_REGION='us-west-2', it worked.
Also I tried not exporting AWS_REGION and tested on a us-east-1 bucket. It worked too.
So for some reason, the region info in aws profile is not retrieved and used. If environment variable AWS_REGION is not used, it will be always us-east-1, the default.

Boto: how to get security group by id?

I`m trying to get the security group by group id.
Here is the code:
#!/usr/bin/env python2.7
import boto.ec2
import argparse
parser = argparse.ArgumentParser(description="")
parser.add_argument('sec_group_id', help='Security group id')
parser.add_argument('region_name', help='Region name')
args = parser.parse_args()
sec_group_id = args.sec_group_id
region_name = args.region_name
conn = boto.ec2.connect_to_region(region_name);
GivenSecGroup=conn.get_all_security_groups(sec_group_id)
When I execute this:
./sec_groups.py sg-45b9a12c eu-central-1
I get the output:
Traceback (most recent call last):
File "./sec_groups.py", line 22, in <module>
GivenSecGroup=conn.get_all_security_groups(sec_group_id)
File "/usr/lib/python2.7/dist-packages/boto/ec2/connection.py", line 2969, in get_all_security_groups
[('item', SecurityGroup)], verb='POST')
File "/usr/lib/python2.7/dist-packages/boto/connection.py", line 1182, in get_list
raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidGroup.NotFound</Code><Message>The security group 'sg-45b9a12c' does not exist in default VPC 'vpc-d289c0bb'</Message></Error></Errors><RequestID>edf2afd0-f552-4bdf-938e-1bccef798145</RequestID></Response>
So basically it says “The security group 'sg-45b9a12c' does not exist in default VPC 'vpc-d289c0bb'”
But this security group does exist in default VPC! Here is the prove:
AWS console screenshot
How can I make this work?
I would be grateful for your answer.
Short answer:
just change
GivenSecGroup=conn.get_all_security_groups(sec_group_id)
to
GivenSecGroup=conn.get_all_security_groups(group_ids=[sec_group_id])
Long Answer:
get_all_security_groups first argument is a list of security group names and the second is the list of ids:
def get_all_security_groups(self, groupnames=None, group_ids=None,
filters=None, dry_run=False):
"""
Get all security groups associated with your account in a region.
:type groupnames: list
:param groupnames: A list of the names of security groups to retrieve.
If not provided, all security groups will be
returned.
:type group_ids: list
:param group_ids: A list of IDs of security groups to retrieve for
security groups within a VPC.
I will show alternative boto3 answer beside #Vor.
IMHO, you should switch to boto3, the developer has make it clear that boto will not support new features. You don't need to specify region, you can tied the region inside credential file,etc.
import boto3
import argparse
ec2=boto3.client("ec2")
parser = argparse.ArgumentParser(description="")
parser.add_argument('sec_group_id', help='Security group id')
args = parser.parse_args()
sec_group_id = args.sec_group_id
my_sec_grp = ec2.describe_security_groups(GroupIds = [sec_group_id])
Boto3 are closely tied with AWS Cli. The current AWS cli has show features such "--query" that allow user to filter the results return. If AWS implement that features, that will be in boto3, not boto.