I am using the callback feature of Step Function where I get a Task Token generated internally by step function in my context object.
The link I referred to:- https://docs.aws.amazon.com/step-functions/latest/dg/connect-to-resource.html#connect-wait-example
Here in example they have given the example of token as :- "Token": "h7XRiCdLtd/83p1E0dMccoxlzFhglsdkzpK9mBVKZsp7d9yrT1W"
I was interested in knowing which all characters does this token contain? Like here I can see "slash" as well. There is no documentation regarding this as well.
TL;DR Most likely only the characters supported by base64
I don't believe there is any official AWS documentation explicitly stating the allowed character set in the TaskToken, only that the maximum length is 1024.
I poked around within my own account and noticed that the length of the $$.TaskToken is dependent on the name given to the step-function execution (at least for the eventbridge integration).
The documentation does however state the following:
You must pass task tokens from principals within the same AWS account. The tokens won't work if you send them from principals in a different AWS account.
https://docs.aws.amazon.com/step-functions/latest/dg/connect-to-resource.html#connect-wait-token
My assumption based on the previous two observations is that the token is a base64 encoded HMAC digest similar to the AWS Signature V4 signing process
Confirming via an AWS Support case asking if they can provide you with this info is advised if you're needing it for production workloads though.
Related
When creating a sagemaker training job using the sagemaker python package (using SKLearn or Tensorflow estimators), I pass in a number of hyperparameters. I create a single training job for each of our clients (we have 200), and the job gets named TrainModel-[client_id]-[int:1-50]-[uuid] (unsure if this is important, will explain why soon). The hyperparameters are all integers or strings, and none resemble an AWS access key or token. For 199 out of 200 clients, the training job is created and runs flawlessly.
For one, however, I get the following error: (ValidationException) when calling the CreateTrainingJob operation: Input Hyper parameter contains AWS access key id or access token. This validation is warned about in the documentation, and I have ensured that I do not include any AWS access key ids or tokens (otherwise how would the other 199 run without incident?).
I have examined the actual hyperparameters as they are recorded in the Estimator, and nothing is out of the ordinary. The only hyperparameters that change per client on a given day are the client name, client id, and the integer ranging 1-50 (which indicates a sub-division of the client).
I have read through the source code and cannot find the actual code that validates whether AWS access key ids and tokens exist in the inputs, and the Error and Traceback are utterly uninformative (I can't tell which parameter is the offender). Based on the qualities of the hyper parameters that change, the only culprit could possibly be the client name, which does not include anything to do with AWS keys or tokens. Is it possible that, for whatever reason, the validation function (I assume it's a regular expression) incorrectly identifies this particular client name, which again, has zero reference to anything in our AWS or in generic AWS terms, as an AWS key or token?
Has anyone encountered this before? If so what was your solution?
It turns out that, whatever the way AWS checks for access tokens and key IDs, it is not very accurate. It was detecting the client_name as an access token or key ID, which is was certainly not. Removing this parameter solved the issue. Watch out for random strings being miscategorized in this way.
So I got access to SimilarWeb ranking API from AWS(https://aws.amazon.com/marketplace/pp/prodview-clsj5k4afj4ma?sr=0-1&ref_=beagle&applicationId=AWSMPContessa).
I'm not able to figure out how to pass the authentication or how to give a request to retrieve the ranks for domains.
For ex. how will you pass the request for this URL in python?
URL: https://api-fulfill.dataexchange.us-east-1.amazonaws.com/v1/v1/similar-rank/amazon.com/rank
This particular product does not seem to be available any longer. Generally speaking, an AWS IAM principal with correct IAM permissions, can make API calls against AWS Data Exchange for APIs endpoints. The payload of the API call needs to adhere to the OpenAPI spec defined within the DataSet of the product used. The specific API call is 'SendApiAsset'. The easiest way to think about is to read the boto3 documentation for it, here: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dataexchange.html#DataExchange.Client.send_api_asset
Other AWS SDKs have the same call, idiomatic to the specific language.
The managed policy that describes the IAM permissions needed is named AWSDataExchangeSubscriberFullAccess, the dataexchange specific permission needed is 'dataexchange:SendApiAsset'.
The awscli way of making the call is described here: https://docs.aws.amazon.com/cli/latest/reference/dataexchange/send-api-asset.html
The required parameters are: asset-id, data-set-id, revision-id. You will likely also need to provide values for: method and body (and perhaps others also depending on the specific API you are calling.
The content of the 'body' parameter needs to adhere to the OpenAPI spec of the actual dataset provided as part of the product.
You can obtain the values for asset-id, data-set-id and revision-id from the AWS Data Exchange service web console describing the product/dataset.
This is a real challenge. I have been successful in everything up to this point in Fleet provisioning on an embedded device. I have subscribed and published to topics and received new certificates and keys. But, when I take the certificateOwnershipToken that has been given to me and I try to trigger a DeviceRegistration, I get:
{"statusCode":400,"errorCode":"InvalidCertificateOwnershipToken","errorMessage":"Certificate ownership token cannot be empty."}
My token is 466 characters long and I send it with 2 other items in this string:
{"certificateOwnershipToken":"eyF1ZXJzaW9uIjoiMjAxOTEwMjMiLCJjaXBoZXIiOiJBaURqMUdYMjBiVTUwQTFsTUV4eEJaM3ZXREU1dXZSSURoTy80cGpLS1o1VkVHdlZHQm81THNaS1VydG0zcTdoZGtVR0l1cmJZS0dLVkx2dTZNL2ViT2pkVFdIeDEwU3o3aFZPeExERkxWVlJ4OUIvL2RzcXRIeVp1WVo2RXZoU1k0L0txQ0doZ1lyRklwZGlLK05pUlNHMXlLQXJUSGJXSkNlVUxHcHRPWHJtdHJaNWJMUyt1MHFUcjNJVnlVLzNpcGZVVm1PanpmL3NCYzdSNkNyVGJPZ05Nc2xmOXdHSVRWM0tPUjd1aFFSbnZySnY0S1ZtU2hYc2REODI4K1crRE1xYnRYZGUxSXlJU29XZTVTSHh6NVh2aFF3OGl3V09FSzBwbG15Zi82dUgyeERoNjB1WS9lMD0ifQ==","parameters":{"SerialNumber":"82B910","CertificateId":"175b43a3d605f22d30821c4a920a6231978e5d846d3f2e7a15d2375d2fd5098c"}}
My templates looks right, my policy looks correct. The role which is attached to my template seem to cover my needs. I just don't know how AWS is failing without more information.
Does anyone have ideas on how to proceed?
I found my problem. In C/C++ aws iot sdk -- there is a data structure where you must specify the payload string and a few other things. One of those data elements is the length of the payload and I forgot to set that length before sending my payload to the $aws/provisioning-templates//provision/json topic. Once I set that length, then the submission worked and the template was acted upon and the thing was created
I am trying to write a Lua library for Amazon SES that will allow me to send API requests. I've poured over the documentation and various examples but I am continuing to get the following error:
The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
One of my functions somewhere along the line is formatting data incorrectly or doing something to cause the end result of my signing process to not match what Amazon is generating on their side, so my request is being rejected. However, Amazon doesn't provide any useful information in their error response such as showing me the canonical request that they generate so I can compare it to mine to see if there are any discrepencies. My best attempts to debug this is to use the examples they provide in their documentation (see below) as "known good" comparisons and to try and generate the same hashes with my functions... Except that they don't provide all of the necessary information to do so.
In Task 3 of their documentation process, they do share an example secrey key and I've been able to use that to verify that at least part of my code is indeed working as intended, but that key does not seem to generate the same hashes on the other tasks. Am I missing something here, or is there a better way to figure this problem out?
Below are the example keys I was able to pull out of various Task pages in their documentation:
api_key = "AKIDEXAMPLE"
api_secret = "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY"
In Amazon's Documentation for Task 1, they provide the final canonical request and a paired hash:
GET
/
Action=ListUsers&Version=2010-05-08
content-type:application/x-www-form-urlencoded; charset=utf-8
host:iam.amazonaws.com
x-amz-date:20150830T123600Z
content-type;host;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
f536975d06c0309214f805bb90ccff089219ecd68b2577efef23edd43b7e1a59
However when I use the above secret to hash the above canonical request, I get a different hash:
d2da54b4842d8ca1acf1cf197827f4d75a742918af868d472e883781624a8bb5
So they must being using a different secret in some examples without actually documenting them.. unless I missed something?
Documentation: https://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html
WIP Code: https://hastebin.com/mezugukefu.lua
Is there any limit on the number of pre signed URL's per object in AWS S3 presigned URL's. Say If I want to create 1000 presigned url's per object in a 2 minutes. Is that valid scenario ?
You can create as many signed URLs as you wish. Depending on your motivation and strategy, however, there is a practical limitation on the number of unique presigned URLs for the exact same object.
S3 (in S3 regions that were first deployed before 2014) supports two authentication algorithms, V2 and V4, and the signed urls look very different since the algorithms are very different.
In V2, the signed URL for a given expiration time will always look the same, if signed by the same AWS key.
If you sign the url for an object, set to expire one minute in the future... and immediately repeat the process, the two signed URLs will be identical.
Next, exactly one second later, sign a url for the same object to expire 59 seconds in the future, and that new signed URL will also be identical.
Why? Because in V2, the expiration time is an absolute wall clock time in UTC, and the particular time in history when you actually generated the signed URL doesn't change anything.
V4 is different. In the scenario above, the first two would still be identical, but the second one would not, because V4 auth includes the date and time when you created the signed url, or when you say you did. The expiration time is relative to the signing time, instead of absolute.
Note that both forms of signed URL are tamper-resistant -- the expiration time is embedded in the url, but attempting to tweak it after signing will invalidate the signing and make it useless.
If you need to generate a large number of signed urls for the same object, you'll need to increment the expiration time for each individual signing attempt in order to get unique values. (Edit: or not, if you're feeling clever... see below).
It also occurs to me that you may be under the impression that S3 has an active role in the signing process, but it doesn't. That's all done in your local code.
S3 isn't aware, in any sense, of the signed urls you generate unless or until they are used. When a signed request arrives, S3 does exactly the same thing your code will do -- it canonicalizes certain attributes of the request, and generates a signature. Then it compares what it generated with what your code should have generated, given precisely the same parameters. If their generated signature matches your provided signature (and the key you used has permission to perform the requested action) then the request succeeds.
Update: it turns out, there is an unofficial mechanism that allows you to embed additional "entropy" into the signing process, generating unique, per-user (for example) signed URLs for the same object and expiration time.
Under V2 authentication, which doesn't nornally want you to include non-S3-specific parameters in your signing logic, it looks suspiciously like a bug as well as a feature... add &x-amz-meta-{anything-here}={unique-value-here} query string parameters to your URL. These are used as headers in PUT request but are meaningless in a GET request, and yet, if present, S3 still requires them to be included in the signature calculation, even though the parameter keys and values will ultimately be discarded by S3... but the added values are tamper-resistant and can't be maliciously removed or altered without invalidating the signature.
The same mechanism works in V4, even though it's for a different reason.
Credit for this technique: http://www.bennadel.com/blog/2488-generating-pre-signed-query-string-authentication-amazon-s3-urls-with-user-specific-data.htm
The accepted answer is now outdated. For future viewers, there is no need to include anything as extra header as now AWS includes a Signature field in every signed url which is different everytime you make a request.
Yes. In fact, i believe AWS can't even limit that, as there is no such API call on S3. URL signing is done purely by the SDK.
But if creating so many URLs is a good idea or not is completely context dependent though...