I'm trying to use the createInvalidtion API that amazon offers, and I'm a bit confused about the authentication part, I saw this documentation about the AWS signature, but, do I really want to do all of this?! It seems too much for a simple request, also, I'm not sure of what each part should be, specially the canonical request part as the API I want to use is simple one with no much to build.
I'm using a programming language (TCL) with no built-in library for cloudFront, So I'm going to build everything from scratch, appreciate any help or if anyone went through a similar situation.
I saw this documentation about the AWS signature, but, do I really
want to do all of this?!
The AWS signature is built using a chain of HMAC computations, there is not all too much to doing this. Departing from the AWS documentation, this could look like:
set key "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY"
set dateStamp "20120215"
set regionName "us-east-1"
set serviceName "iam"
proc hmac-sha256 {str hexKey} {
lindex [exec openssl dgst -sha256 -mac hmac -macopt hexkey:$hexKey << [encoding convertto utf-8 $str]] 1
}
proc getSignatureKey {key dateStamp regionName serviceName} {
binary scan [encoding convertto utf-8 "AWS4$key"] H* hexKey
set kDate [hmac-sha256 $dateStamp $hexKey]
puts "kDate = $kDate"
set kRegion [hmac-sha256 $regionName $kDate];
puts "kRegion = $kRegion"
set kService [hmac-sha256 $serviceName $kRegion];
puts "kService = $kService"
set kSigning [hmac-sha256 "aws4_request" $kService]
puts "kSigning = $kSigning"
return $kSigning
}
getSignatureKey $key $dateStamp $regionName $serviceName
This prints out:
kDate = 969fbb94feb542b71ede6f87fe4d5fa29c789342b0f407474670f0c2489e0a0d
kRegion = 69daa0209cd9c5ff5c8ced464a696fd4252e981430b10e3d3fd8e2f197d7a70c
kService = f72cfd46f26bc4643f06a11eabb6c0ba18780c19a8da0c31ace671265e3c87fa
kSigning = f4780e2d9f65fa895f9c67b32ce1baf0b0d8a43505a000a1a9e090d414db404d
Some background:
Calling out to the openssl executable might not be adequate in some situations, you might need to check out for some Tcl built-ins (I am only aware of an Tcl command binding to OpenSSL/LibreSSL in NaviServer and tcllib's sha2 module)
The above implementation passes around the hexdumps of the binary strings (for educational, debugging purposes), you might want to use the binary strings directly.
As for the REST of the task, it boils down to assembling an HTTP request with custom request parameters and XML payload.
Related
I'm trying to write a Java app that behaves as a Thing, publishing data in AWS. The Documentation has this code sample:
String clientEndpoint = "<prefix>.iot.<region>.amazonaws.com"; // replace <prefix> and <region> with your own
String clientId = "<unique client id>"; // replace with your own client ID. Use unique client IDs for concurrent connections.
String certificateFile = "<certificate file>"; // X.509 based certificate file
String privateKeyFile = "<private key file>"; // PKCS#1 or PKCS#8 PEM encoded private key file
// SampleUtil.java and its dependency PrivateKeyReader.java can be copied from the sample source code.
// Alternatively, you could load key store directly from a file - see the example included in this README.
KeyStorePasswordPair pair = SampleUtil.getKeyStorePasswordPair(certificateFile, privateKeyFile);
AWSIotMqttClient client = new AWSIotMqttClient(clientEndpoint, clientId, pair.keyStore, pair.keyPassword);
// optional parameters can be set before connect()
client.connect();
I know what clientId is and how to find my ID, but I cannot understand the in clientEndpoint.
It's not the account's personal endpoint, but the Thing's endpoint.
Go to IoT Core -> Manage -> Things, select your thing -> Interact.
Its the URL under the HTTPS part. It should be in the form xxxxxxxxxxxxxxxxx.iot.region.amazonaws.com, where the x's should contain mainly lowercase letters, and maybe some numbers.
Call the DescribeEndpoint API.
In Java, this would be:
AWSIot awsIotClient = AWSIotClientBuilder.defaultClient();
DescribeEndpointRequest request = new DescribeEndpointRequest().withEndpointType("iot:Data");
DescribeEndpointResult result = awsIotClient.describeEndpoint(request);
String endpoint = result.getEndpointAddress();
I have seen pre-signed URL for S3 object. Is it possible to create pre-signed URL for API gateway. I have gone through documentation. I am using .NET. I would like to know if there is .NET library available to create pre-signed request for gateway API.
ISSUE
I have GET API something like this https://xxxxxx.execute-api.us-east-1.amazonaws.com/dev/pets?type=dog&page=1 and our client is going to invoke that API once in a while. The legacy tool that they are using only supports GET. So i wanted to create a pre-signed URL (with short expiry time) and give them when they ask for it. For each client i already have IAM user with their respective accesskey and secretkey
PreSigned URLs are typically signed with AWS SigV4 signing process.
You can generate SigV4 signed Urls for your API Gateway Hosted Endpoints. Typically, you will need to send SigV4 signature in Authorization Request Header. If you are clients are willing to send header, here is one sample Library you can try for .NET which creates a HTTP Request with signed header.
If your clients cannot send Authorization Header or cannot use above library then you can convert the signature to be a Query String Format and provide the pre-signed Urls to them.
This AWS Documentation has example in Python on how to generate Query String URL. Now, you can take python example and convert into .NET based code with following sample.
public string GetSig4QueryString(string host, string service, string region)
{
var t = DateTimeOffset.UtcNow;
var amzdate = t.ToString("yyyyMMddTHHmmssZ");
var datestamp = t.ToString("yyyyMMdd");
var canonical_uri = "/dev/myApigNodeJS";
var canonical_headers = "host:" + host+"\n";
var signed_headers = "host";
var credential_scope = $"{datestamp}/{region}/{service}/aws4_request";
var canonical_querystring = "X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=" + WebUtility.UrlEncode(_access_key + "/" + credential_scope)
+ "&X-Amz-Date=" + amzdate + "&X-Amz-SignedHeaders=" + signed_headers;
Console.WriteLine("canonical_querystring");
Console.WriteLine(canonical_querystring);
var payload_hash = Hash(new byte[0]);//No Payload for GET
var canonical_request = new StringBuilder();
canonical_request.Append("GET\n");
canonical_request.Append(canonical_uri + "\n");
canonical_request.Append(canonical_querystring + "\n");
canonical_request.Append(canonical_headers + "\n");
canonical_request.Append(signed_headers + "\n");
canonical_request.Append(payload_hash);
Console.WriteLine("canonical_request");
Console.WriteLine(canonical_request);
var string_to_sign = $"{algorithm}\n{amzdate}\n{credential_scope}\n" + Hash(Encoding.UTF8.GetBytes(canonical_request.ToString()));
Console.WriteLine("string_to_sign");
Console.WriteLine(string_to_sign);
var signing_key = GetSignatureKey(_secret_key, datestamp, region, service);
var signature = ToHexString(HmacSHA256(signing_key, string_to_sign));
var signed_querystring = canonical_querystring+"&X-Amz-Signature=" + signature;
return signed_querystring;
}
GetSig4QueryString("myApiId.execute-api.us-east-1.amazonaws.com","execute-api","us-east-1");
//Returned String --> X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential= AKIAIOSFODNN7EXAMPLE%2F20190104%2Fus-east-1%2Fexecute-api%2Faws4_request&X-Amz-Date=20190104T190309Z&X-Amz-SignedHeaders=host&X-Amz-Signature=7b830fce28f7800b3879a25850950f6c4247dfdc07775b6952295fa2fff03f7f
Full Endpoint Becomes -
https://myApiId.execute-api.us-east-1.amazonaws.com/dev/myApigNodeJS?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIOSFODNN7EXAMPLE%2F20190104%2Fus-east-1%2Fexecute-api%2Faws4_request&X-Amz-Date=20190104T190309Z&X-Amz-SignedHeaders=host&X-Amz-Signature=7b830fce28f7800b3879a25850950f6c4247dfdc07775b6952295fa2fff03f7f
Note -
This example code refers methods & variables from Github project I gave above.
Also, this example hard coded API Path /dev/myApigNodeJS and signs it and it will be different for you with full absolute path.
AWS recommends to sign all queryStrings, headers which you are planning to send in request. Go through .NET code of library I referred and understand how its doing that.
Let me know if you have questions.
When generating the presigned url a websocket service within the AWS API Gateway, I used the solution by Imran and added the "X-Amz-Security-Token" which is required.
I'm familiar with the Range HTTP header; however, the interface I'm using to query S3 (an img element's .src property) doesn't allow me to specify HTTP headers.
Is there a way for me to specify my desired range via a parameter in the query string?
It doesn't seem like there is, but I'm just holding out a shred of hope before I roll my own solution with ajax requests.
Amazon S3 supports Range GET requests, as do some HTTP servers, for example, Apache and IIS.
How CloudFront Processes Partial Requests for an Object (Range GETs)
I tried to get my S3 object via cURL:
curl -r 0-1024 https://s3.amazonaws.com/mybucket/myobject -o part1
curl -r 1025- https://s3.amazonaws.com/mybucket/myobject -o part2
cat part1 part2 > myobject
and AWS SDK for JavaScript:
var s3 = new AWS.S3();
var file = require('fs').createWriteStream('part1');
var params = {
Bucket: 'mybucket',
Key: 'myobject',
Range: 'bytes=0-1024'
};
s3.getObject(params).createReadStream().pipe(file);
These two methods work fine for me.
AWS SDK for JavaScript API Reference (getObject)
Below is the Java Code with AWS V2 SDK
format the range as below
var range = String.format("bytes=%d-%d", start, end);
and pass it in below api with GetObjectRequest builder
ResponseBytes<GetObjectResponse> currentS3Obj = client.getObjectAsBytes(GetObjectRequest.builder().bucket(bucket).key(key).range(range).build());
return currentS3Obj.asInputStream();
is there anyway to dynamically generate the response for the Amazon Product API with using just a URL string?
I see there are PHP and C# libraries but I am just trying to browse to a URL and see the response. I noticed one of the required fields of the URL is a timestamp which makes this tricky. The following page helped to generate the URLs but I can't seem to find a way to do it dynamically?
http://associates-amazon.s3.amazonaws.com/scratchpad/index.html
Thanks!
This is the dynamic search for amazon product
Download aws_signed_request.php from this url
include('aws_signed_request.php');
$public_key = 'xxxxxxxx';
$private_key = 'xxxxxxxxxx';
$associate_tag = 'xxxxxx';
$keywords= 'PHP';
$search_index = 'Books';
// generate signed URL
$request = aws_signed_request('com', array(
'Operation' => 'ItemSearch',
'Keywords' => "Php Books",
"SearchIndex" => "Books",
"Count" => '24',
'ResponseGroup' => 'Large,EditorialReview'), $public_key, $private_key, $associate_tag);
// do request (you could also use curl etc.)
$response = #file_get_contents($request);
Documentation URL HERE
I'm not sure I totally understand your question, but I think the answer is "the data in the Amazon product API is only available via 'signed' URLs". That way, Amazon can track abuse, etc back to the source (i.e. the signer).
If it were possible to get the data with a "static" URL, then you could post that URL all over the Internet and anyone could get the data without signing up with Amazon. It's their data, and they have rules on it's use, so that wouldn't fly with them.
That said, you can usually create URLs with a timestamp in the future (months or even years). But you would still be responsible for it's use/abuse.
I am current working with Google Apps script and am attempting to write & sign an HTTP request to AWS CloudWatch.
On the Amazon API documentation here regarding how to create a signing key, they use pseudo to explain that the HMAC algorithm is to return binary format.
HMAC(key, data) represents an HMAC-SHA256 function
that returns output in binary format.
Google apps script offers a method to do such a hash,
Utilities.computeHmacSignature(Utilities.MacAlgorithm.HMAC_SHA_256,
data,
key);
but the return type is always a byte array.
Byte[]
How do I convert the Byte[] to the binary data AWS wants? Or is there a vanilla javascript function I can use in Google Apps Script to compute the hash?
Thanks
I am quite sure it is a bug that Utilities.computeHmacSignature take key as an ASCII. But there was no way to parse byte[] to ASCII correctly in GAS
And the library writer is too stupid too just provide function which take key as byte[]
So I use this instead : http://caligatio.github.com/jsSHA/
Just copy SHA.js and SHA-256.js then it work fine
PS. it waste my time for whole 2 days so I'm very annoying
The conversion from byte array to the binary data required should be simple:
kDate = Utilities.computeHmacSignature(Utilities.MacAlgorithm.HMAC_SHA_256,
'20130618', 'AWS4' + kSecret);
kDate = Utilities.newBlob(kDate).getDataAsString();
kRegion = Utilities.computeHmacSignature(Utilities.MacAlgorithm.HMAC_SHA_256,
'eu-west-1', kDate);
BUT you have to look onto this open issue in the bugtracker - there could be some issues in conversion.
maybe you could try to make a String.fromCharCode() loop and avoid negative numers:
kDateB = Utilities.computeHmacSignature(Utilities.MacAlgorithm.HMAC_SHA_256,
'20130618', 'AWS4' + kSecret);
kDate = '';
for (var i=0; i<kDateB.length; i++)
kDate += String.fromCharCode(kDateB[i]<0?256+kDateB[i]:0+kDateB[i]);
kRegion = Utilities.computeHmacSignature(Utilities.MacAlgorithm.HMAC_SHA_256,
'eu-west-1', kDate);