When creating a sagemaker training job using the sagemaker python package (using SKLearn or Tensorflow estimators), I pass in a number of hyperparameters. I create a single training job for each of our clients (we have 200), and the job gets named TrainModel-[client_id]-[int:1-50]-[uuid] (unsure if this is important, will explain why soon). The hyperparameters are all integers or strings, and none resemble an AWS access key or token. For 199 out of 200 clients, the training job is created and runs flawlessly.
For one, however, I get the following error: (ValidationException) when calling the CreateTrainingJob operation: Input Hyper parameter contains AWS access key id or access token. This validation is warned about in the documentation, and I have ensured that I do not include any AWS access key ids or tokens (otherwise how would the other 199 run without incident?).
I have examined the actual hyperparameters as they are recorded in the Estimator, and nothing is out of the ordinary. The only hyperparameters that change per client on a given day are the client name, client id, and the integer ranging 1-50 (which indicates a sub-division of the client).
I have read through the source code and cannot find the actual code that validates whether AWS access key ids and tokens exist in the inputs, and the Error and Traceback are utterly uninformative (I can't tell which parameter is the offender). Based on the qualities of the hyper parameters that change, the only culprit could possibly be the client name, which does not include anything to do with AWS keys or tokens. Is it possible that, for whatever reason, the validation function (I assume it's a regular expression) incorrectly identifies this particular client name, which again, has zero reference to anything in our AWS or in generic AWS terms, as an AWS key or token?
Has anyone encountered this before? If so what was your solution?
It turns out that, whatever the way AWS checks for access tokens and key IDs, it is not very accurate. It was detecting the client_name as an access token or key ID, which is was certainly not. Removing this parameter solved the issue. Watch out for random strings being miscategorized in this way.
Related
I am using the callback feature of Step Function where I get a Task Token generated internally by step function in my context object.
The link I referred to:- https://docs.aws.amazon.com/step-functions/latest/dg/connect-to-resource.html#connect-wait-example
Here in example they have given the example of token as :- "Token": "h7XRiCdLtd/83p1E0dMccoxlzFhglsdkzpK9mBVKZsp7d9yrT1W"
I was interested in knowing which all characters does this token contain? Like here I can see "slash" as well. There is no documentation regarding this as well.
TL;DR Most likely only the characters supported by base64
I don't believe there is any official AWS documentation explicitly stating the allowed character set in the TaskToken, only that the maximum length is 1024.
I poked around within my own account and noticed that the length of the $$.TaskToken is dependent on the name given to the step-function execution (at least for the eventbridge integration).
The documentation does however state the following:
You must pass task tokens from principals within the same AWS account. The tokens won't work if you send them from principals in a different AWS account.
https://docs.aws.amazon.com/step-functions/latest/dg/connect-to-resource.html#connect-wait-token
My assumption based on the previous two observations is that the token is a base64 encoded HMAC digest similar to the AWS Signature V4 signing process
Confirming via an AWS Support case asking if they can provide you with this info is advised if you're needing it for production workloads though.
This is a real challenge. I have been successful in everything up to this point in Fleet provisioning on an embedded device. I have subscribed and published to topics and received new certificates and keys. But, when I take the certificateOwnershipToken that has been given to me and I try to trigger a DeviceRegistration, I get:
{"statusCode":400,"errorCode":"InvalidCertificateOwnershipToken","errorMessage":"Certificate ownership token cannot be empty."}
My token is 466 characters long and I send it with 2 other items in this string:
{"certificateOwnershipToken":"eyF1ZXJzaW9uIjoiMjAxOTEwMjMiLCJjaXBoZXIiOiJBaURqMUdYMjBiVTUwQTFsTUV4eEJaM3ZXREU1dXZSSURoTy80cGpLS1o1VkVHdlZHQm81THNaS1VydG0zcTdoZGtVR0l1cmJZS0dLVkx2dTZNL2ViT2pkVFdIeDEwU3o3aFZPeExERkxWVlJ4OUIvL2RzcXRIeVp1WVo2RXZoU1k0L0txQ0doZ1lyRklwZGlLK05pUlNHMXlLQXJUSGJXSkNlVUxHcHRPWHJtdHJaNWJMUyt1MHFUcjNJVnlVLzNpcGZVVm1PanpmL3NCYzdSNkNyVGJPZ05Nc2xmOXdHSVRWM0tPUjd1aFFSbnZySnY0S1ZtU2hYc2REODI4K1crRE1xYnRYZGUxSXlJU29XZTVTSHh6NVh2aFF3OGl3V09FSzBwbG15Zi82dUgyeERoNjB1WS9lMD0ifQ==","parameters":{"SerialNumber":"82B910","CertificateId":"175b43a3d605f22d30821c4a920a6231978e5d846d3f2e7a15d2375d2fd5098c"}}
My templates looks right, my policy looks correct. The role which is attached to my template seem to cover my needs. I just don't know how AWS is failing without more information.
Does anyone have ideas on how to proceed?
I found my problem. In C/C++ aws iot sdk -- there is a data structure where you must specify the payload string and a few other things. One of those data elements is the length of the payload and I forgot to set that length before sending my payload to the $aws/provisioning-templates//provision/json topic. Once I set that length, then the submission worked and the template was acted upon and the thing was created
I am trying to write a Lua library for Amazon SES that will allow me to send API requests. I've poured over the documentation and various examples but I am continuing to get the following error:
The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
One of my functions somewhere along the line is formatting data incorrectly or doing something to cause the end result of my signing process to not match what Amazon is generating on their side, so my request is being rejected. However, Amazon doesn't provide any useful information in their error response such as showing me the canonical request that they generate so I can compare it to mine to see if there are any discrepencies. My best attempts to debug this is to use the examples they provide in their documentation (see below) as "known good" comparisons and to try and generate the same hashes with my functions... Except that they don't provide all of the necessary information to do so.
In Task 3 of their documentation process, they do share an example secrey key and I've been able to use that to verify that at least part of my code is indeed working as intended, but that key does not seem to generate the same hashes on the other tasks. Am I missing something here, or is there a better way to figure this problem out?
Below are the example keys I was able to pull out of various Task pages in their documentation:
api_key = "AKIDEXAMPLE"
api_secret = "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY"
In Amazon's Documentation for Task 1, they provide the final canonical request and a paired hash:
GET
/
Action=ListUsers&Version=2010-05-08
content-type:application/x-www-form-urlencoded; charset=utf-8
host:iam.amazonaws.com
x-amz-date:20150830T123600Z
content-type;host;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
f536975d06c0309214f805bb90ccff089219ecd68b2577efef23edd43b7e1a59
However when I use the above secret to hash the above canonical request, I get a different hash:
d2da54b4842d8ca1acf1cf197827f4d75a742918af868d472e883781624a8bb5
So they must being using a different secret in some examples without actually documenting them.. unless I missed something?
Documentation: https://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html
WIP Code: https://hastebin.com/mezugukefu.lua
I have a requirement to build a basic "3 failed login attempts and your account gets locked" functionality. The project uses AWS Cognito for Authentication, and the Cognito PreAuth and PostAuth triggers to run a Lambda function look like they will help here.
So the basic flow is to increment a counter in the PreAuth lambda, check it and block login there, or reset the counter in the PostAuth lambda (so successful logins dont end up locking the user out). Essentially it boils down to:
PreAuth Lambda
if failed-login-count > LIMIT:
block login
else:
increment failed-login-count
PostAuth Lambda
reset failed-login-count to zero
Now at the moment I am using a dedicated DynamoDB table to store the failed-login-count for a given user. This seems to work fine for now.
Then I figured it'd be neater to use a custom attribute in Cognito (using CognitoIdentityServiceProvider.adminUpdateUserAttributes) so I could throw away the DynamoDB table.
However reading https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-dg.pdf the section titled "Configuring User Pool Attributes" states:
Attributes are pieces of information that help you identify individual users, such as name, email, and phone number. Not all information about your users should be stored in attributes. For example, user data that changes frequently, such as usage statistics or game scores, should be kept in a separate data store, such as Amazon Cognito Sync or Amazon DynamoDB.
Given that the counter will change on every single login attempt, the docs would seem to indicate I shouldn't do this...
But can anyone tell me why? Or if there would be some negative consequence of doing so?
As far as I can see, Cognito billing is purely based on storage (i.e. number of users), and not operations, whereas Dynamo charges for read/write/storage.
Could it simply be AWS not wanting people to abuse Cognito as a storage mechanism? Or am I being daft?
We are dealing with similar problem and main reason why we have decided to store extra attributes in DB is that Cognito has quotas for all the actions and "AdminUpdateUserAttributes" is limited to 25 per second.
More information here:
https://docs.aws.amazon.com/cognito/latest/developerguide/limits.html
So if you have a pool with 100k or more it can create a bottle neck if wanted to update a Cognito user records with every login etc.
Cognito UserAttributes are meant to store information about the users. This information can then be read from the client using the AWS Cognito SDK, or just by decoding the idToken on the client-side. Every custom attribute you add will be visible on the client-side.
Another downside of custom attributes is that:
You only have 25 values to set
They cannot be removed or changed once added to the user pool.
I have personally used custom attributes and the interface to manipulate them is not excellent. But that is just a personal thought.
If you want to store this information, and not depend on DynamoDB, you can use Amazon Cognito Sync. Besides the service, it offers a client with great features that you can incorporate to your app.
AWS DynamoDb appears to be your best option, it is commonly used for such use cases. Some of the benefits of using it:
You can store separate record for each login attempt with as much info as you want such as ip address, location, user-agent etc. You can also add datetime that can be used by pre-auth Lambda to query by time range for example failed attempt within last 30 minutes
You don't need to manage table because you can set TTL for DynamoDb record so that record will be deleted automatically after specified time.
You can also archive items in S3
When I try to download all log files from a RDS instance, in some cases, I found this error in my python output:
An error occurred (InvalidParameterValue) when calling the
DownloadDBLogFilePortion operation: This file contains binary data and
should be downloaded instead of viewed.
I manage correctly the pagination and the throttling (using The Marker parameter and the sleep function).
This is my calling:
log_page=request_paginated(rds,DBInstanceIdentifier=id_rds,LogFileName=log,NumberOfLines=1000)
rds-> boto3 resource
And this is the definition of my function:
def request_paginated(rds,**kwargs):
return rds.download_db_log_file_portion(**kwargs)
Like I said, most of time this function works but sometime it returns:
"An error occurred (InvalidParameterValue) when calling the
DownloadDBLogFilePortion operation: This file contains binary data and
should be downloaded instead of viewed"
Can you help me please? :)
UPDATE: the problem is a known issue with downloading log files that contain non printable sign. As soon as possible I will try the proposed solution provide by the aws support
LATEST UPDATE: This is an extract of my discussion with aws support team:
There is a known issue with non binary characters when using the boto based AWS cli, however this issue is not present when using the older Java based cli.
There is currently no way to fix the issue that you are experiencing while using the boto based AWS cli, the workaround is to make the API call from the Java based cli
the aws team are aware of this issue and are working on a way to resolve this, however the do not have an ETA for when this will be released.
So the solutions is: use the java API
Giuseppe
LATEST UPDATE: This is an extract of my discussion with aws support team:
There is a known issue with non binary characters when using the boto based AWS cli, however this issue is not present when using the older Java based cli.
There is currently no way to fix the issue that you are experiencing while using the boto based AWS cli, the workaround is to make the API call from the Java based cli
the aws team are aware of this issue and are working on a way to resolve this, however the do not have an ETA for when this will be released. So the solutions is: use the java API
Giuseppe
http://docs.aws.amazon.com/AmazonRDS/latest/APIReference/CommonErrors.html
InvalidParameterValue : An invalid or out-of-range value was supplied
for the input parameter.
Invalid parameter in boto means the data pass does not complied. Probably an invalid name that you specified, possible something wrong with your variable id_rds, or maybe your LogFileName, etc. You must complied with the function arguments requirement.
response = client.download_db_log_file_portion(
DBInstanceIdentifier='string',
LogFileName='string',
Marker='string',
NumberOfLines=123
)
(UPDATE)
For example, LogFileName must be te exact file name exist inside RDS instance.
For the logfile , please make sure the log file EXISTS inside the instance. Use this AWS CLI to get a quick check
aws rds describe-db-log-files --db-instance-identifier <my-rds-name>
Do check Marker (string) and NumberOfLines (Integer) as well. Mismatch type or out of range. Skip them since they are not required, then test it later.