Is it possible to use Postman to access Dynamodb via http and the dynamo api - amazon-web-services

I'm attempting to use Postman, the chrome extension api client Postman to access Dynamodb via http. This is actually an approach I read about in the Book Dynamodb Applied Design Patterns. It appears that I am doing something wrong because I am unable to sucessfully authenticate and gain access to the web service.
The url and headers I'm using in the Postman client with my most recent request parameters and error messages follow:
url: dynamodb.us-east-1.amazonaws.com
x-amz-date: 20150701T162011Z
x-amz-targe: DynamoDB_20120810.ListTables
authorizaton: AWS4-HMAC-SHA256 Credential=AMyAccessCode/20150701/us-east-1/dynamodb/aws4_request, SignedHeaders=host;x-amz-date;x-amz-target, Signature=8ngTnF8WH//njvBdY5bY5dSp5CAKi8qTXNFuv5Ws+30=
content-type: application/x-amz-json-1.0
The Body of my request is:
{
"ExclusiveStartTableName": "Owner",
"Limit": 3
}
I'm getting the following error message:
{"__type":"com.amazon.coral.service#InvalidSignatureException","message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been\n'POST\n/\n\nhost:dynamodb.us-east-1.amazonaws.com\nx-amz-date:20150701T162011Z\nx-amz-target:DynamoDB_20120810.ListTables\n\nhost;x-amz-date;x-amz-target\nb9e264461dcb0e94e69652f8b2d17c737a29506863d6f09c0f9fc98e9d560e5c'\n\nThe String-to-Sign should have been\n'AWS4-HMAC-SHA256\n20150701T162011Z\n20150701/us-east-1/dynamodb/aws4_request\n7a6da3d9e8ed6317e0cb9217e9ea1174d01e86871a159f339f5f6969283264d5'\n"}
Question/Oddity 1. When I issue a request I usually get response back indicating that my request has expired. The error message will include "valid" times that are four hours ahead of the actual time tha I used. When I use the time provided by the error message in the request, I no longer receive the exprired request error message.
Question 2. This message seems to indicate I have to calculate the hash of the canonical request. The documentation on amazon is a bit unclear on this. Do I need to calcuate the has of the canonical request and if so do I include that as a header" If so, what is the header name?
I have been able to calculate the signature and replicate the signature in the Amazon documentation.
Thanks for your input.

I was able to invoke using Postman 3.
CURL Exmaple:
curl -X POST \
https://dynamodb.eu-west-1.amazonaws.com/ \
-H 'authorization: AWS4-HMAC-SHA256 Credential=ABCDDEFSDFDDFDAG/20170725/eu-west-1/dynamodb/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-target, Signature=7402a0163c267385b7270f5c6ce49748518cac4f4d7e03addbd9ddedf9c9970' \
-H 'cache-control: no-cache' \
-H 'content-type: application/json' \
-H 'host: dynamodb.eu-west-1.amazonaws.com' \
-H 'postman-token: 0152b36c-2947-2a6f-91b9-87884b38fb50' \
-H 'x-amz-date: 20170725T181135Z' \
-H 'x-amz-target: DynamoDB_20120810.GetItem' \
-d '{
"TableName": "TableName",
"Key": {
"id": {"S": "KeyId"}
}
}'
Insert Body
Enter Credentials and Click on Update Request.You might need to click on Update Request more often to update x-amz-date
Update Content-Type Header to application/json

You're on the right path. As you imply, you're running into issues with the API authentication scheme that Amazon uses. See http://docs.aws.amazon.com/general/latest/gr/signature-version-4.html.
The way this works is that Amazon wants you to take a few pieces of information including the service you're calling and current timestamp and compute a hash that uses this data along with your AWS Secret Access Key (actually, they derive a key from your Secret Access Key, but don't worry about that for now).
One of the key design principles going on here is that Amazon wants to prevent Replay Attacks, where someone just sends a request you already sent, even if they can't read it. Amazon expects you to report the current timestamp in the request header, you'll include that exact same timestamp in the hash, and when Amazon receives your request, they will try to re-compute your hash using all the info they've got.
If the hash matches AND the reported timestamp is within 15 minutes of the current time, you're accepted. I suspect the 15-minutes issue is one of your problems.
As far as how to simulate these calls. I think you'll find it's probably easier to just write a very basic RESTful API of your own that does these calls using an SDK published by Amazon. Then you don't have to deal with any of the crazy signature computation and you can call your APIs however you want.

Copy-paste from https://docs.aws.amazon.com/apigateway/latest/developerguide/how-to-use-postman-to-call-api.html
Launch Postman.
Enter the endpoint URL of a request in the address bar and choose the appropriate HTTP method from the drop-down list to the left of the address bar.
3.If required, choose the Authorization tab. Choose AWS Signature for the authorization Type. Enter your AWS IAM user's access key ID in the AccessKey input field. Enter your IAM user secret key in SecretKey. Specify an appropriate AWS region that matches the region specified in the invocation URL. Enter execute-api in Service Name.
4.Choose the Headers tab. Optionally, delete any existing headers. This can clear any stale settings that may cause errors. Add any required custom headers. For example, if API keys are enabled, you can set the x-api-key:{api_key} name/value pair here.
5.Choose Send to submit the request and receive a response.

Related

Example code for AWS Cognito User Pool InitiateAuth with Username and Password via HTTPS call?

I am trying to use Cognito User Pool to authenticate with a PC application using an HTTPS call. I want to obtain the various tokens that I can then use to access the AWS resources without storing AWS secrets in the PC application.
The AWS documentation documents the InitiateAuth method and shows the AWS Endpoints, but it is not immediately apparent how to make the call over HTTPS. Most calls would require an AWS signature, but the InitiateAuth call should not, if I am just submitting Username and Password.
After some poking around, I was able to use the AWS CLI to successfully obtain tokens with this command:
aws cognito-idp initiate-auth --auth-flow USER_PASSWORD_AUTH --client-id the_cognito_client_id --auth-parameters USERNAME=the_users_email,PASSWORD=the_users_password
Where the_cognito_client_id is an approximately 26 character long string shown as App client id under General Settings / App clients.
Note that the USER_PASSWORD_AUTH flow is not enabled by default, so you will initially get an error with this. Go to the Cognito Console for the specific User Pool and look for General Settings/App Clients, click "Show Details" for your specific app client, and check the "Enable username password based authentication (ALLOW_USER_PASSWORD_AUTH)" and save.
Once you get back tokens and you know your call is working, you can use the aws history show command to show you the details of the actual https call. The first time you call
aws history show
You will get a message
Could not locate history. Make sure cli_history is set to enabled in the ~/.aws/config file
Go to that file and add
cli_history=enabled
Then, run your initiate-auth call again with the cli. Then, when you run
aws history show
You will get back the details of how the call was made. (At this point, you might consider removing the cli_history setting so you don't log all of your calls, with all of the credentials, in the future.) You will see
to URL: https://cognito-idp.us-east-1.amazonaws.com/
which tells you the URL to use, and you will see that it is a POST. You will note that "InitiateAuth" is not anywhere in that URL. However, you will see the headers include:
"X-Amz-Target": "AWSCognitoIdentityProviderService.InitiateAuth"
and
"Content-Type": "application/x-amz-json-1.1"
You need both of those headers, including the non-standard Content-Type, to make the HTTPS call work. You can use Postman to put the call together, although Postman does not like the non-standard Content-Type, so you have to turn off the standard Content-Type and manually add these two headers to the call. At that point, Postman is able to obtain the tokens as well.
Postman also provides an export to CURL function (click the link that says "Code"), which gives you:
curl --location --request POST 'https://cognito-idp.us-east-1.amazonaws.com/' \
--header 'X-Amz-Target: AWSCognitoIdentityProviderService.InitiateAuth' \
--header 'Content-Type: application/x-amz-json-1.1' \
--data-raw '{
"AuthFlow": "USER_PASSWORD_AUTH",
"AuthParameters": {
"PASSWORD": "the_users_password",
"USERNAME": "the_users_email"
},
"ClientId": "the_cognito_client_id"
}'
Submitting that on the command line also gives you the tokens you need.
To refresh using the refresh token, just use InitiateAuth, but the AuthFlow is REFRESH_TOKEN_AUTH and the only member of AuthParameters is REFRESH_TOKEN (which is, of course, the RefreshToken)
Now, I just need to figure out how to do USER_SRP_AUTH using HTTPS.
The other answer explains how to get the Tokens using the Username and Password. Next, we need to get the temporary credentials from the Cognito Identity Pool. This appears to require two steps. First, we need to call cognito-identity get-id and then cognito-identity get-credentials-for-identity
The get-id call requires the Identity Pool ID, which can be obtained from the Cognito Console for the Identity Pool. The identity-pool-id is available under "edit identity pool" as "identity pool ID." The Login key name comes from the User pool ID and is available under Authentication Providers / Cognito under the Edit Identity Pool. The Login key name is actually provided as "iss" in the encoded ID Token. You can use a service like https://jwt.io/ to decode the ID Token, and it will include an entry like
"iss": "https://cognito-idp.us-east-1.amazonaws.com/us-east-1_zAgxxxxxx"
You can use that, without the https://, as the key for Login. The value for Login is the JWT returned as "IdToken" from InitiateAuth.
Therefore, using the CLI, the first call is
aws cognito-identity get-id --identity-pool-id "us-east-1:aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee" --logins "{\"cognito-idp.us-east-1.amazonaws.com/us-east-1_zAgxxxxxx\": \"ThisIsTheVeryLong.IDTokenReturneFrom.TheCognitoUserPool\"}"
Which returns the ID:
{
"IdentityId": "us-east-1:11111111-2222-3333-4444-555555555555"
}
Then, we need to get the temporary credentials using get-credentials-for-identity. This call uses the IdentityId returned from get-id rather than the Identity Pool ID used by get-id. The Login JSON is the same as for get-id. Therefore, the call is:
aws cognito-identity get-credentials-for-identity --identity-id "us-east-1:11111111-2222-3333-4444-555555555555" --logins "{\"cognito-idp.us-east-1.amazonaws.com/us-east-1_zAgxxxxxx\": \"ThisIsTheVeryLong.IDTokenReturneFrom.TheCognitoUserPool\"}"
Which returns something like:
{
"IdentityId": "us-east-1:11111111-2222-3333-4444-555555555555",
"Credentials": {
"AccessKeyId": "ASIAXXXXXXXXXXXXXXXX",
"SecretKey": "Im0JN4PrvZZZZZZZZZZZZZZZZZZZZZZZZZZ/Y/XX",
"SessionToken": "ExtremelyLongSessionToken",
"Expiration": "2020-06-07T15:28:51-07:00"
}
}
You can then use that AccessKeyID and SecretKey to make any needed AWS calls; these temporary credentials are good for an hour.
Using the same methodology with aws history show and Postman described above, we can get the CURL equivalents:
curl --location --request POST 'https://cognito-identity.us-east-1.amazonaws.com/' \
--header 'X-Amz-Target: AWSCognitoIdentityService.GetId' \
--header 'Content-Type: application/x-amz-json-1.1' \
--data-raw '{
"IdentityPoolId": "us-east-1:aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee",
"Logins": {
"cognito-idp.us-east-1.amazonaws.com/us-east-1_zAgxxxxxx": "ThisIsTheVeryLong.IDTokenReturneFrom.TheCognitoUserPool"
}
}'
and then
curl --location --request POST 'https://cognito-identity.us-east-1.amazonaws.com/' \
--header 'X-Amz-Target: AWSCognitoIdentityService.GetCredentialsForIdentity' \
--header 'Content-Type: application/x-amz-json-1.1' \
--data-raw '{
"IdentityId": "us-east-1:11111111-2222-3333-4444-555555555555",
"Logins": {
"cognito-idp.us-east-1.amazonaws.com/us-east-1_zAgxxxxxx": "ThisIsTheVeryLong.IDTokenReturneFrom.TheCognitoUserPool"
}
}'
Note that the CLI converts the expiration timestamp to local time. When you call this with CURL, the "Expiration" value will be something like 1.591572729E9 . You can convert that to a human time with a site like https://www.epochconverter.com/

How to integrate a Google Natural Language API with Chatfuel?

My coding skills and familiarity with Google Cloud solutions are limited and I'm trying to consume a Machine Learning model from a chatbot build using the platform Chatfuel.
I've trained a Natural Language Machine learning model using Google NL and I wanted to use this code snippet provided in my model page:
curl -X POST \
-H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
-H "Content-Type: application/json" \
https://automl.googleapis.com/v1/projects/123456/locations/us-central1/models/ABC123:predict \
-d #request.json
I've then tried to pass this information in Chatbot like this:
The problem is it looks like I need to pass a token (API key seems not possible for this API) and I don't see how from the Chatfuel interface I could obtain a token for each request and pass it in my post request.
Edit: I've seen this post and created a service account but I don't see how to pass the credentials to Chatfuel.
How can I pass Google API service account credential to Chatfuel?
You should make a service account with Google Cloud.
It should give you a JSON with an API key, which is used to generate a new token every hour or so. Using the Google API for whatever server side language you are using (or just using the native http requests), you use the service key (which in general is loaded into your server as a .json file) to fetch a new token every hour; OR you can use Google Apps Script, if they have Chatfuel available (which I'm not sure if they do) and get the token there, and send it to your own app every 30 minutes or so via a trigger.
But the main thing is: using your service API key, you make an HTTP request to a certain api URL (available on with instructions from another page from that link above) every hour or so to generate a new token.
From the docs, that URL appears to be:
POST https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/SA-4#PROJECT-ID.iam.gserviceaccount.com:generateAccessToken
just replace, "`
SA-4#PROJECT-ID.iam.gserviceaccount.com
`"
with your own service account name
That's about it, its how it works for all Google Cloud APIs

Cannot authenticate into google datastore through REST Api

I tried to access Google's Datastore through their REST Api. It says that they allow authentication through the API-key. However it doesn't seems that I can get it to work any where. I copied the snippet generated from their Try this API page.
curl --request POST \
'https://datastore.googleapis.com/v1/projects/PROJECT_ID:runQuery?key=[YOUR_API_KEY]' \
--header 'Accept: application/json' \
--header 'Content-Type: application/json' \
--data '{"query":{"filter":{"compositeFilter":{"op":"AND","filters":[{"propertyFilter":{"property":{"name":"id"},"op":"EQUAL","value":{"stringValue":"ID"}}}]}},"kind":[{"name":"NAME"}]},"partitionId":{"namespaceId":"NAMESPACE_ID","projectId":"PROJECT_ID"}}' \
--compressed
But it keeps returning me an 401 error.
{
"error": {
"code": 401,
"message": "Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
"status": "UNAUTHENTICATED"
}
}
It looks like it require me to use OAuth instead, which is not what their documentation says. Anyone experienced something similar?
You are using an API key which is incorrect.
This link details which services support API Keys. Cloud Datastore is not one of them.
Using API Keys
You want to use an Access Token which is derived from Service Account credentials.
Review this document.
Using OAuth 2.0
The steps to generate an Access Token:
Load the service account credentials json file.
Extract the client_email, private_key and private_key_id.
Create a json payload.
Call the authorization URL: https://www.googleapis.com/oauth2/v4/token
This returns a json object. Extract the access_token.
Use the access_token instead of an API Key.
There are examples on the Internet in various languages. The link will get you started. The process appears complicated, and it is, but once you understand it, generating Access Tokens is easy and they can be reused until they expire (typically 60 minutes which you control).
This document on Google Cloud Storage authentication is the same for Cloud Datastore. The key is understanding "Authorization: Bearer" which is a header you need to include with your curl request.
Authentication

Using an API key in Amazon API Gateway

I have created an API Key and added it to my functions. I have then deployed the api and tested it but still get:
"message": "Forbidden"
How do I pass the api key with my JSON request as I have been using "x-api-key": "theKey"?
The x-api-key parameter is passed as a HTTP header parameter (i.e. it is not added to the JSON body). How you pass HTTP headers depend on the HTTP client you use.
For example, if you use curl and assuming that you POST the JSON payload, a request would look something like (where you replace [api-id] with the actual id and [region] with the AWS region of your API):
$ curl -X POST -H "x-api-key: theKey" -H "Content-Type: application/json" -d '{"key":"val"}' https://[api-id].execute-api.[region].amazonaws.com
I had to add an API Usage plan, and then link the plan to the API stage.
Seems like this is the only way to link the key to the API, not sure if this is a recent change on AWS.
If you set 'API Key Required' option to true, please check below.
you have to pass 'x-api-key' HTTP Header Parameter to API Gateway.
The API Key had to be created.
In addition, you need to check a Usage Plan for the API Key on API Gateway Console.
If you set 'API' key required to true, you need to pass the api key as header.
API Key is passed as header field 'x-api-key'. Even after adding this field in header, this issue may occur. In that case, please validate below points
Do you have a Usage Plan? if not need to create one.
Link you API with Usage Plan. For that add a stage, it will link your API
Do you have API Key? if not you need to create an API Key and enable it.
Add the Usage Plan which is linked with your API to this API Key. For that, add Usage Plan.
I hope you are not missing to link the API key with the API
I was able to get a successful response from Lambda using below configuration in Postman native app -
Under authorization tab (For some reason this didn't work when i passed the same parameters under header)
Key : x-api-key
Value : your-api-key-value
Add to : Header
I don't have enough reputation to set this as a comment, But I was finally able to find the document specifying that 'x-api-key' belongs in the header for API Gateway calls that come from outside clients (like postman, swagger, etc.) in the AWS Documentation.
The relevant part:
To use header-sourced API keys:
Create an API with desired API methods. And deploy the API to a
stage.
Create a new usage plan or choose an existing one. Add the deployed
API stage to the usage plan. Attach an API key to the usage plan or
choose an existing API key in the plan. Note the chosen API key
value.
Set up API methods to require an API key.
Redeploy the API to the same stage. If you deploy the API to a new
stage, make sure to update the usage plan to attach the new API
stage.
The client can now call the API methods while supplying the x-api-key
header with the chosen API key as the header value.
Choose an API key source
For Private API Gateways accessed through public DNS, we need to pass additional header of 'x-apigw-api-id' with the api id along with 'x-api-key' if configured.
curl -v https://{vpce-id}.execute-api.{region}.vpce.amazonaws.com/test -H 'x-apigw-api-id:{api-id}'
Its documented below,
https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-private-api-test-invoke-url.html#w20aac13c16c28c11
Here a good resource explaining different reasons why we could be getting a Forbidden. The two most important are the request URL and the x-api-key header:
https://{api_id}.execute-api.{region}.amazonaws.com/{stage_name}/{resource_name}
Missing stage name will give you 403 for ex. Maybe for security reasons the response is not revealing an issue with the stage name, and thus you get a generic Forbidden.
I faced the same problem today. I had already mapped the API key to the usage plan (which was linked to the api gateway stage). I was also passing the api key in header correctly.
When none of these solutions work, do remember to check if your API is linked to WAF policy with only a certain ip-addresses permitted. Apparently, my IP address had changed today. So, WAF was blocking me. That can be an additional reason to get {"message": "Forbidden"} error.

Most secure way to secure Jersey REST Services

I'm looking really for just advice. I have a system up and running on my amazon cloud instance that is basically a bunch of REST services running on JBoss. My next step is to secure these services as there will be credit card information flowing through them. I also need to authentication as well so my question is, what is the most secure methods that one can use for REST services?
SSL CA Certs of course so encrypt the data using a CA is probably where i'll start of course.
Is go daddy reputable for this? or do i have to shell out alot of money for verisign?
For authentication, would it be sufficient to simply do basic auth or maybe just having caller sign the request somehow? Any other methods?
OH i forgot to mention, the client application is an iPad application. Thanks for the advice.
You want to be able to protect yourself from man in the middle attacks and prevent replaying of requests.
Any time there is payment related information being relayed I would opt for signing the request using a nonce and timestamp.
This involves signing the request using a shared secret between the client and server. The secret can be passed one time on login.
Use a timestamp and client generated unique nonce value as part of the signed bytes.
These values are also passed back as headers in the request so the server can reassemble the request.
A typical request executed from curl might look like this:
curl -v -H "Content-Type: application/json" -H "Authorization: ff7b93ad-27d0-49f6-90bd-9937951e5fcc:ncYoA5n5s2nFSm7qyvf5hDgL4pmmPOUP3zo/UYfaQKg=" -H "x-date:2013-03-28T19:34:00+00:00" -H "nonce:2d1321d32a" -X GET 'http://localhost/orders/123'
The Authorization header contains an id to identify the requester and then the hash of the signed request.
The date header should be within a certain offset of the server time (15 mins is a reasonable limit).
I have a full code example here
Using certs is a great start to security. I found Thawte a good balance between value and supported clients. When I looked (a while back) GoDaddy wasn't supported by enough of the clients I might have expected (Java, Objective-C/iPad, browsers) but that may have changed by now. You definitely want to make sure the certificate you get is supported by the clients you care about (Objective-C in your case for the iPad).
Basic auth is ok over https, just make sure you don't expose anything in URLs such as ids or tokens since the URL itself is visible. If you POST all your data over https you will be off to a good start.