Is it possible to call AWS CloundFront API with CURL? I would like to trigger an invalidation without installing much dependency. But I'm unable to make a simple GET request to the API:
curl -v -X GET \
-H "Date: $(date -R)" \
-H "Authorization: AWS ${CLOUDFRONT_ACCESS_KEY}:$(echo -en ${CLOUDFRONT_ACCESS_KEY} | openssl sha1 -hmac ${CLOUDFRONT_SECRET_ACCESS_KEY} -binary | base64)" \
https://cloudfront.amazonaws.com/2020-05-31/distribution/EMC3WW4JXXXXX/invalidation/IXMUICGG7L77A
Results
<?xml version="1.0" encoding="UTF-8"?>
<ErrorResponse xmlns="http://cloudfront.amazonaws.com/doc/2020-05-31/">
<Error>
<Type>Sender</Type>
<Code>IncompleteSignature</Code>
<Message>Authorization header requires 'Credential' parameter. Authorization header requires 'Signature' parameter. Authorization header requires 'SignedHeaders' parameter. Authorization=AWS AKIAJG77PBXLMN5YQI7A:K62YyDlWiVf/yr44YSs7BbsQYDQ=</Message>
</Error>
<RequestId>f9e5b7de-bce6-4bfd-951e-2986ae5bc1a3</RequestId>
</ErrorResponse>
You can take a look at Signing AWS requests with Signature Version 4 for signing the request.
Plus the invalidation API call needs more params for making the call as per the documentation
I usually keep this handy s3-rest-api-with-curl
Related
when I execute
curl --request GET "https://${ES_DOMAIN_ENDPOINT}/my_index_pattern-*/my_type/_mapping" \
--user $AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY \
--aws-sigv4 "aws:amz:ap-southeast-2:es"
where $ES_DOMAIN_ENDPOINT is my AWS Elasticsearch endpoint, I'm getting the following response:
{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."}
I'm confident that my $AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY are correct.
However, when I send the same postman request with the AWS Authentication and the parameters above, the response is coming through. I compared the verbose output of both requests and they have very minor differences, such as timestamps and signature.
I'm wondering, what is wrong with the --aws-sigv4 config?
This issue happens due to the* character in the path. There is a bug report in curl repository to fix this issue https://github.com/curl/curl/issues/7559.
Meanwhile, to mitigate the error you should either remove a * from the path or build curl from the branch https://github.com/outscale-mgo/curl-appimage/tree/http_aws_sigv4_encoding.
I'm trying to post a request using curl to my es cluster in AWS using my accessKey and secretKey. I have successfully done this through postman (details here) where you can specify AWS credentials but I would like to make this work with curl. Postman can auto-generate your curl request for you but all I get are errors.
This is the generated curl request along with the response
curl -X GET \
https://search-00000000000001.eu-west-1.es.amazonaws.com/_cat/indices \
-H 'Authorization: AWS4-HMAC-SHA256 Credential=11111111111111111111/20181119/eu-west-1/es/aws4_request, SignedHeaders=cache-control;content-type;host;postman-token;x-amz-date, Signature=11111111116401882398f46011f14fdb9d55e012a4fb912706d67c1111111111' \
-H 'Content-Type: application/x-www-form-urlencoded' \
-H 'Host: search-00000000000001.eu-west-1.es.amazonaws.com' \
-H 'Postman-Token: 00000000-0000-4001-8006-9291e208a000' \
-H 'X-Amz-Date: 20181119T220000Z' \
-H 'cache-control: no-cache'
{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."}%
IDs have been changed to protect the innocent.
I have checked all my keys and region, and like i said this works through postman. Is it possible to access this AWS service using my keys through curl?
This is quite a long rabbit hole. Thanks to Adam for the comment that sent me in the correct direction. The link https://docs.aws.amazon.com/apigateway/api-reference/signing-requests/ really helps you understand what you need to do.
I've since found a script that follows the signing requests method outlined above. It runs in bash and whilst it is not written for use with elasticsearch requests it can be used for them.
https://github.com/riboseinc/aws-authenticating-secgroup-scripts many thanks to https://www.ribose.com for putting this on github.
If your host contains ':443' remove it and try again.
This worked for me.
"My initial problem: If I access it with Postman using the same url, I get the same error, but removing the ‘:443/’, it works fine, so it’s nothing wrong with the key and secret I’m using."
I'm looking for a secure way to directly upload large files to S3 (for performance issue).
After few hours of research, I've come to the (maybe incorrect) conclusion that I should use "Browser-Based Uploads Using POST".
As referenced in this thread:
Amazon S3 direct file upload from client browser - private key disclosure
Before trying this directly, I thought about making a cURL POC with direct upload and signature computation.
I've failed to make it work, and haven't found a successful POC on the web using Authentification Version 4 manual settings.
My signature is OK.
The only issue is that Amazon is double SHA256ing my file content and is thus not validation my x-amz-content-sha-256 header.
lower(SHA256(e8379a31b13fb9423928fe28dd41a5e3204a52072634503c31e8b3ea42605b46))
= 4fa84cd7d18e0d33dbd62d0492eca4a159e122391ae0a3e636bd3cf527680c87
I'm not sure of understanding what should I put in my cURL and canonical request payload (and the linked content-length value) and the x-amz-content-sha-256 header.
Should it all be the same values ?
If yes, then the Amazon doc specifies it should all be encrypted to SHA256, so I've no idea why Amazon reSHA256 my already SHA256 payload...
Error:
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>XAmzContentSHA256Mismatch</Code>
<Message>The provided 'x-amz-content-sha256' header does not match what was computed.</Message><ClientComputedContentSHA256>e8379a31b13fb9423928fe28dd41a5e3204a52072634503c31e8b3ea42605b46</ClientComputedContentSHA256><S3ComputedContentSHA256>4fa84cd7d18e0d33dbd62d0492eca4a159e122391ae0a3e636bd3cf527680c87</S3ComputedContentSHA256>
<RequestId>419A185269B0F891</RequestId><HostId>QHWxK0Mzz6AfG44ypXBti3W0tYx1xkG9lZGqc2kUKyMF9STwP18M3racio0k06aH5+1ok/Irdn8=</HostId>
</Error>
cURL command:
curl
-v https://??.s3.amazonaws.com/recordtest/test.jpg
-H "Authorization: AWS4-HMAC-SHA256 Credential=??/20170228/eu-west-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=43750caa762314eb70aace1f7f8ae34633b93352aa25646433ef21e48dd79429" -H "Content-Length: 64"
-H "Content-Type: application/octet-stream"
-H "x-amz-content-sha256: e8379a31b13fb9423928fe28dd41a5e3204a52072634503c31e8b3ea42605b46"
-H "x-amz-date: 20170228T111828Z"
-d "e8379a31b13fb9423928fe28dd41a5e3204a52072634503c31e8b3ea42605b46"
-X PUT
Generated canonical request:
PUT
/recordtest/test.jpg
content-length:64
content-type:application/octet-stream
host:??.s3.amazonaws.com
x-amz-content-sha256:e8379a31b13fb9423928fe28dd41a5e3204a52072634503c31e8b3ea42605b46
x-amz-date:20170228T111200Z
content-length;content-type;host;x-amz-content-sha256;x-amz-date
e8379a31b13fb9423928fe28dd41a5e3204a52072634503c31e8b3ea42605b46
Wso2 API Manager 1.10.0. PUT method does not work in API store.
GET, POST, DELETE methods work, but PUT dose not work.
Error,
Curl
curl -X PUT --header "Content-Type: application/json" --header "Accept: application/json" --header "Authorization: Bearer c6c90338514e561c8a02a7b6a1c96553" -d "{\"payment\":\"PAID\"}" "https://10.138.31.22:8243/Starbucks_API/1/order_1"
Request URL
https://10.138.31.22:8243/Starbucks_API/1/order_1
Response Body
<ams:fault xmlns:ams="http://wso2.org/apimanager/security">
<ams:code>900906</ams:code>
<ams:message>No matching resource found in the API for the given request</ams:message>
<ams:description>Access failure for API: /Starbucks_API/1, version: 1. Check the API documentation and add a proper REST resource path to the invocation URL</ams:description>
</ams:fault>
Response Code
403
Response Headers
{
"content-type": "application/xml; charset=UTF-8"
}
Resource given in the request is actually matching, which I mentioned in API.
Any help is appreciated.
Resource Definition
You are having two PUT with same URL. You can't have two different HTTP resource with same URL.
I'm using WSO2 API Manager 1.10.0. If I publish API and subscribe and then call this API with version number, all is OK. Default version of API is checked. When I send request to this API without version in URL I get exception:
TID: [-1234] [] [2016-05-06 07:23:01,782] ERROR {org.wso2.carbon.apimgt.usage.publisher.APIMgtFaultHandler} - Cannot publish event. null {org.wso2.carbon.apimgt.usage.publisher.APIMgtFaultHandler}
java.lang.NumberFormatException: null
at java.lang.Long.parseLong(Long.java:552)
at java.lang.Long.parseLong(Long.java:631)
at org.wso2.carbon.apimgt.usage.publisher.APIMgtFaultHandler.mediate(APIMgtFaultHandler.java:76)
at org.apache.synapse.mediators.ext.ClassMediator.mediate(ClassMediator.java:84)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:81)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:48)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:155)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:220)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:81)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:48)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:155)
at org.apache.synapse.rest.API.process(API.java:389)
at org.apache.synapse.rest.RESTRequestHandler.dispatchToAPI(RESTRequestHandler.java:97)
at org.apache.synapse.rest.RESTRequestHandler.process(RESTRequestHandler.java:65)
at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:295)
at org.apache.synapse.core.axis2.SynapseMessageReceiver.receive(SynapseMessageReceiver.java:83)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.synapse.transport.passthru.ServerWorker.processNonEntityEnclosingRESTHandler(ServerWorker.java:317)
at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:149)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Our API manager is configured to work with WSO2 IS 5.1.0 and WSO2 DAS.
This issue will be fixed in the next release and you can use following workaround in order to solve the issue.
For existing APIs
Edit the synapse config file of the default api which is in /repository/deployment/server/synapse-configs/default/api folder.
Ex:
File name of Version API - admin--CountryTest_v1.0.0.xml,
File name of default API - admin--CountryTest.xml
Replace the uri-template value inside the resource tag and save the file.
current value - uri-template="$util.escapeXml($resource.getUriTemplate())"
new value - uri-template="/*"
For new APIs
You can change the default_api_template.xml file in order to apply the fix for new APIs.
Edit the default_api_template.xml which is in /repository/resources/api_templates folder replace the resource tag from the new value.
Current Value -
<resource xmlns="http://ws.apache.org/ns/synapse"
#if($resource.getUriTemplate().contains("{") ||
($resource.getUriTemplate().contains("*") && !$resource.getUriTemplate().endsWith("/*")))
uri-template="$util.escapeXml($resource.getUriTemplate())"
#else
url-mapping="$resource.getUriTemplate()"
#end
methods="$resource.getMethodsAsString()" faultSequence="fault">
New Value -
<resource methods="POST PATCH GET DELETE HEAD PUT OPTIONS"
uri-template="/*"
faultSequence="fault">
I have tested your use case but for me, it's working fine. Can you please double check whether that API was properly saved and published after you checked the Make this the Default Version check box.
You can check it by listing APIs(Metadata-> list-> API) in the carbon console and then click on the API name.
I simply tested it using the CalculatorAPI sample which takes some query parameters. I just published the samples and I subscribed to the CalculatorAPI. Below is the findings.
Just after the publish(without checked the Make this the Default Version check box)
1st API call with version
curl -k -X GET --header "Accept: application/json" --header "Authorization: Bearer afbbb795dadf272cec9c44efc78dc81b" "https://192.168.1.4:8243/calc/1.0/add?x=2&y=2"
Response
{"answer": "4.0"}
2nd API call without version
curl -k -X GET --header "Accept: application/json" --header "Authorization: Bearer afbbb795dadf272cec9c44efc78dc81b" "https://192.168.1.4:8243/calc/add?x=2&y=2"
response
am:fault xmlns:am="http://wso2.org/apimanager"><am:code>404</am:code><am:type>Status report</am:type><am:message>Not Found</am:message><am:description>The requested resource (/calc/add?x=2&y=2) is not available.</am:description></am:fault>
Then I edited the API and checked the Make this the Default Version check box and then published the API. Now I called the API and below is the output.
1st API call with version
curl -k -X GET --header "Accept: application/json" --header "Authorization: Bearer afbbb795dadf272cec9c44efc78dc81b" "https://192.168.1.4:8243/calc/1.0/add?x=2&y=2"
Response
{"answer": "4.0"}
2nd API call without version
curl -k -X GET --header "Accept: application/json" --header "Authorization: Bearer afbbb795dadf272cec9c44efc78dc81b" "https://192.168.1.4:8243/calc/add?x=2&y=2"
response
{"answer": "4.0"}
I have found a answer or some reason why endpoint without version doesn't work. If I put some parameter to URL Pattern in API definition (for instance /a/{param}, or /a?param={param}, the resource in API description (xml file) is generated with attribute url-template=/a/{param} and in this case such endpoint doesn't work without version. If I remove {param} from URL pattern and insert parameter by button "Add Parameter" - endpoint without version (default) works OK. In this case in API description resource is generated with attribute url-mapping not with url-template (like in ESB, there are two types of URL in resource, I think).
So, I have to refuse to use path and query parameters explicitly in URL pattern when I wont to use endpoints without version (default).
Maybe there are some workaround?
There is some additional small question. If I wonted to add "path" parameter by pushing button ADD Parameter in GUI I couldn't do it, because of there aren't option "path" in select list, there are query, header, formData only.