I am attempting to use a GET request to use the Amazon Mechanical Turk GetFileUploadURL function. However, I get this error code when attempting it.
AWS.NotAuthorized The identity contained in the request is not authorized to use this AWSAccessKeyId
This is the code I'm using to create the request.
now = DateTime.now
#For creating the signature hash
data = "AWSMechanicalTurkRequesterGetFileUploadURL" + now.to_s
sha256 = OpenSSL::Digest::SHA256.new
sig = OpenSSL::HMAC.digest(sha256, Rails.configuration.secret_key, data)
signature = Base64.encode64(sig)
puts "https://mechanicalturk.amazonaws.com", "/?Service=AWSMechanicalTurkRequester&AWSAccessKeyId=#{Rails.configuration.aws_key}&Version=2014-08-15&Operation=GetFileUploadURL&Signature=#{sig}&Timestamp=#{now}&AssignmentId=#{mturk_results[0][:AssignmentId]}&QuestionIdentifier=file1"
puts "\n\n"
uri = URI('https://mechanicalturk.amazonaws.com')
params = {:Service=>"AWSMechanicalTurkRequester", :AWSAccessKeyId=>Rails.configuration.aws_key, :Version=>"2014-08-15", :Operation=>"GetFileUPloadURL", :Signature=>sig, :Timestamp=>now, :AssignmentId=>mturk_results[0][:AssignmentId], :QuestionIdentifier=>"file1"}
uri.query = URI.encode_www_form(params)
res = 0
Net::HTTP.start(uri.host, uri.port,
:use_ssl => uri.scheme == 'https') do |http|
request = Net::HTTP::Get.new uri
res = http.request request # Net::HTTPResponse object
end
Any ideas as to what I'm doing wrong?
You'll need to format your TimeStamp correctly. The time stamp must be in UTC and in the following ISO 8601 format:
YYYYMMDD'T'HHMMSS'Z'. For example, 20150830T123600Z is a valid time stamp. Do not include milliseconds in the time stamp.
It needs to be of the form:
Timestamp=2016-04-23T08:00:05Z
While Ruby's DateTime.now method returns them of the form:
2016-08-05T10:43:27-07:00
You can read more about Timestamp and AWS signatures here:
http://docs.aws.amazon.com/general/latest/gr/sigv4-date-handling.html
Related
String distributionDomain = "d21geuebylb7j1.cloudfront.net";
String privateKeyFilePath = "/Users/Desktop/rsa-private-key.der";
String s3ObjectKey = "small.mp4";
String policyResourcePath = "http://" + distributionDomain + "/" + s3ObjectKey;
System.out.println(privateKeyFilePath);
byte[] derPrivateKey = null;
I am trying to make signed URL for my cloudfront distribution but I am getting invalid key error. I am getting issue with my rsa-private-key.der file. I have made this file from pem file as mentioned in Cloudfront documentation.
Below is my error logs:
Exception in thread "main" org.jets3t.service.CloudFrontServiceException: java.security.spec.InvalidKeySpecException: java.security.InvalidKeyException: invalid key format
at org.jets3t.service.CloudFrontService.signUrlCanned(CloudFrontService.java:2148)
at test.SignedURL.main(SignedURL.java:74)
Caused by: java.security.spec.InvalidKeySpecException: java.security.InvalidKeyException: invalid key format
at java.base/sun.security.rsa.RSAKeyFactory.engineGeneratePrivate(RSAKeyFactory.java:216)
at java.base/java.security.KeyFactory.generatePrivate(KeyFactory.java:390)
at org.jets3t.service.security.EncryptionUtil.signWithRsaSha1(EncryptionUtil.java:526)
at org.jets3t.service.CloudFrontService.signUrlCanned(CloudFrontService.java:2133)
... 1 more
Caused by: java.security.InvalidKeyException: invalid key format
at java.base/sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:330)
at java.base/sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:356)
at java.base/sun.security.rsa.RSAPrivateCrtKeyImpl.<init>(RSAPrivateCrtKeyImpl.java:91)
at java.base/sun.security.rsa.RSAPrivateCrtKeyImpl.newKey(RSAPrivateCrtKeyImpl.java:75)
at java.base/sun.security.rsa.RSAKeyFactory.generatePrivate(RSAKeyFactory.java:315)
at java.base/sun.security.rsa.RSAKeyFactory.engineGeneratePrivate(RSAKeyFactory.java:212)
... 4 more
I had same issue this solved my issue.
You can try this:
public enum CloudFrontUrlSigner
extends Enum<CloudFrontUrlSigner>
Utility class for generating pre-signed URLs for serving private CloudFront content. All dates must be in UTC. Use Calendar to set the timezone specifically before converting to a Date object, or else use DateUtils to turn a UTC date String into a Date object.
Protocol protocol = Protocol.http;
String distributionDomain = "d1b2c3a4g5h6.cloudfront.net";
File privateKeyFile = new File("/path/to/cfcurlCloud/rsa-private-key.pem");
String s3ObjectKey = "a/b/images.jpeg";
String keyPairId = "APKAJCEOKRHC3XIVU5NA";
Date dateLessThan = DateUtils.parseISO8601Date("2012-11-14T22:20:00.000Z");
Date dateGreaterThan = DateUtils.parseISO8601Date("2011-11-14T22:20:00.000Z");
String ipRange = "192.168.0.1/24";
String url1 = CloudFrontUrlSigner.getSignedURLWithCannedPolicy(
protocol, distributionDomain, privateKeyFile,
s3ObjectKey, keyPairId, dateLessThan);
String url2 = CloudFrontUrlSigner.getSignedURLWithCustomPolicy(
protocol, distributionDomain, privateKeyFile,
s3ObjectKey, keyPairId, dateLessThan,
dateGreaterThan, ipRange);
here is the link of AWS Documentation: https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/cloudfront/CloudFrontUrlSigner.html
I create a pre-signed URL and get back something like
https://s3.amazonaws.com/MyBucket/MyItem/
?X-Amz-Security-Token=TOKEN
&X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Date=20171206T014837Z
&X-Amz-SignedHeaders=host
&X-Amz-Expires=3600
&X-Amz-Credential=CREDENTIAL
&X-Amz-Signature=SIGNATURE
I can now curl this no problem. However, if I now add another query parameter, I will get back a 403, i.e.
https://s3.amazonaws.com/MyBucket/MyItem/
?X-Amz-Security-Token=TOKEN
&X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Date=20171206T014837Z
&X-Amz-SignedHeaders=host
&X-Amz-Expires=3600
&X-Amz-Credential=CREDENTIAL
&X-Amz-Signature=SIGNATURE
&Foo=123
How come? Is it possible to generate a pre-signed url that supports custom queries?
It seems to be technically feasible to insert custom query parameters into a v4 pre-signed URL, before it is signed, but not all of the AWS SDKs expose a way to do this.
Here's an example of a roundabout way to do this with the AWS JavaScript SDK:
const AWS = require('aws-sdk');
var s3 = new AWS.S3({region: 'us-east-1', signatureVersion: 'v4'});
var req = s3.getObject({Bucket: 'mybucket', Key: 'mykey'});
req.on('build', () => { req.httpRequest.path += '?session=ABC123'; });
console.log(req.presign());
I've tried this with custom query parameters that begin with X- and without it. Both appeared to work fine. I've tried with multiple query parameters (?a=1&b=2) and that worked too.
The customized pre-signed URLs work correctly (I can use them to get S3 objects) and the query parameters make it into CloudWatch Logs so can be used for correlation purposes.
Note that if you want to supply a custom expiration time, then do it as follows:
const Expires = 120;
const url = req.presign(Expires);
I'm not aware of other (non-JavaScript) SDKs that allow you to insert query parameters into the URL construction process like this so it may be a challenge to do this in other languages. I'd recommend using a small JavaScript Lambda function (or API Gateway plus Lambda function) that would simply create and return the customized pre-signed URL.
The custom query parameters are also tamper-proof. They are included in the signing of the URL so, if you tamper with them, the URL becomes invalid, yielding 403 Forbidden.
I used this code to generate your pre-signed URL. The result was:
https://s3.amazonaws.com/MyBucket/MyItem
?Foo=123
&X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Credential=AKIA...27%2Fus-east-1%2Fs3%2Faws4_request
&X-Amz-Date=20180427T0012345Z
&X-Amz-Expires=3600
&X-Amz-Signature=e3...7b
&X-Amz-SignedHeaders=host
None of this is a guarantee that this technique will continue to work, of course, if AWS changes things under the covers but for right now it seems to work and is certainly useful.
Attribution: the source of this discovery was aws-sdk-js/issues/502.
If you change one of the headers or add / subtract, then you have to resign the URL.
This is part of the AWS signing design and this process is designed for higher levels of security. One of the AWS reasons for changing to signing version 4 from signing version 2.
The signing design does not know which headers are important and which are not. That would create a nightmare trying to track all of the AWS services.
I created this solution for Ruby SDK. It is sort of a hack, but it works as expected:
require 'aws-sdk-s3'
require 'active_support/core_ext/object/to_query.rb'
# Modified S3 pre signer class that can inject query params to the URL
#
# Usage example:
#
# bucket_name = "bucket_name"
# key = "path/to/file.json"
# filename = "download_file_name.json"
# duration = 3600
#
# params = {
# bucket: bucket_name,
# key: key,
# response_content_disposition: "attachment; filename=#{filename}",
# expires_in: duration
# }
#
# signer = S3PreSignerWithQueryParams.new({'x-your-custom-field': "banana", 'x-some-other-field': 1234})
# url = signer.presigned_url(:get_object, params)
#
# puts "url = #{url}"
#
class S3PreSignerWithQueryParams < Aws::S3::Presigner
def initialize(query_params = {}, options = {})
#query_params = query_params
super(options)
end
def build_signer(cfg)
signer = super(cfg)
my_params = #query_params.to_h.to_query()
signer.define_singleton_method(:presign_url,
lambda do |options|
options[:url].query += "&" + my_params
super(options)
end)
signer
end
end
While not documented, you can add parameters as arguments to the call to presigned_url.
obj.presigned_url(:get,
expires_in: expires_in_sec,
response_content_disposition: "attachment"
)
https://bucket.s3.us-east-2.amazonaws.com/file.txt?response-content-disposition=attachment&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=PUBLICKEY%2F20220309%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20220309T031958Z&X-Amz-Expires=43200&X-Amz-SignedHeaders=host&X-Amz-Signature=SIGNATUREVALUE
If you are looking on for JavaScript SDK V3:
import { HttpRequest } from "#aws-sdk/protocol-http";
import { S3RequestPresigner } from "#aws-sdk/s3-request-presigner";
import { parseUrl } from "#aws-sdk/url-parser";
import { Sha256 } from "#aws-crypto/sha256-browser";
import { Hash } from "#aws-sdk/hash-node";
import { formatUrl } from "#aws-sdk/util-format-url";
// Make custom query in Record<string, string | Array<string> | null> format
const customQuery = {
hello: "world",
};
const s3ObjectUrl = parseUrl(
`https://${bucketName}.s3.${region}.amazonaws.com/${key}`
);
s3ObjectUrl.query = customQuery; //Insert custom query here
const presigner = new S3RequestPresigner({
credentials,
region,
sha256: Hash.bind(null, "sha256"), // In Node.js
//sha256: Sha256 // In browsers
});
// Create a GET request from S3 url.
const url = await presigner.presign(new HttpRequest(s3ObjectUrl));
console.log("PRESIGNED URL: ", formatUrl(url));
Code template taken from: https://aws.amazon.com/blogs/developer/generate-presigned-url-modular-aws-sdk-javascript/
I'm using Python Requests to call an API using GET
I generate a signature using hmac and hashlib which results in a signature that looks like:
4epwTDKhWcIJL6bMM5f2hmBrOoXXIGD9UwX8ErfYzqU%3D
When I use Requests and specify the params as a string, the API call is successful because the signature is used as-is and doesn't change.
However when I use Requests and specify the params as a dictionary, the signature above is somehow URL encoded again, resulting in the % being encoded to %25, causing the signature to display as follows (note the %25):
4epwTDKhWcIJL6bMM5f2hmBrOoXXIGD9UwX8ErfYzqU%253D
For more context, here is the string params:
url = 'https://example.com/API'
payload = '¶meterA=valueA¶meterB=valueB¶meterC=valueC& apikey='+apikey+'&salt='+salt+'&signature='+sig
#payload = {'parameterA': 'valueA', 'parameterB': 'valueB', 'apikey': apikey, 'salt': salt, 'signature': sig}
r = requests.get(url, params=payload)
print r.url
results in a URL of:
https://example.com/API&
¶meterA=valueA
¶meterB=valueB
&apikey=0e1026af-40ce-e354-f1f2-72d280ca122
&salt=12345
&signature=4epwTDKhWcIJL6bMM5f2hmBrOoXXIGD9UwX8ErfYzqU%3D
Now, the dictionary params:
url = 'https://example.com/API'
payload = {'parameterA': 'valueA', 'parameterB': 'valueB', 'apikey': apikey, 'salt': salt, 'signature': sig}
r = requests.get(url, params=payload)
print r.url
results in a URL of:
https://example.com/API
&signature=4epwTDKhWcIJL6bMM5f2hmBrOoXXIGD9UwX8ErfYzqU%253D
&salt=12345
&apikey=0e1026af-40ce-e354-f1f2-72d280ca122
¶meterA=valueA
¶meterB=valueB
Note again that the % in the signature has changed to %25
Thanks!
When passing a dictionary to params it gets url encoded. Your string is url encoded already so it gets double encoded, resulting in a malformed 'signature'.
You can use urllib.unquote to decode sig (or you can just replace '%3D' with '=')
payload = {
'parameterA': 'valueA', 'parameterB': 'valueB',
'apikey': apikey, 'salt': salt, 'signature': urllib.unquote(sig)
}
Note that in python3 unquote is located in urllib.parse.
Hi I'm new with moodle and I'm getting an error when calling the webservice.
Currently I'm trying to retrieve a user from moodle with the following function core_user_get_users_by_field and I'm using rest service to do so. I already managed to create a user thus I am authenticated to use the service.
the error that I'm receiving is
Missing required key in single structure: field
The bellow is the code was used to create a User. the issue that I got from the error is that the parameter that I need to send for the post is not formatted well. Does anyone know how to search correctly with this method or any other method.
String token = "token";
String postData = "username=username";
string createRequest = string.Format("http://domain/webservice/rest/server.php?wstoken={0}&wsfunction={1}&moodlewsrestformat=json", token, "core_user_get_users_by_field");
// Call Moodle REST Service
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(createRequest);
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";
// Encode the parameters as form data:
byte[] formData =
UTF8Encoding.UTF8.GetBytes(postData);
req.ContentLength = formData.Length;
// Write out the form Data to the request:
using (Stream post = req.GetRequestStream())
{
post.Write(formData, 0, formData.Length);
}
// Get the Response
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
Stream resStream = resp.GetResponseStream();
StreamReader reader = new StreamReader(resStream);
string contents = reader.ReadToEnd();
// Deserialize
JavaScriptSerializer serializer = new JavaScriptSerializer();
if (contents.Contains("exception"))
{
// Error
MoodleException moodleError = serializer.Deserialize<MoodleException>(contents);
}
else
{
// Good
}
The webservice core_user_get_users_by_field needs an associative array given as parameter with the following key:values
'field': 'id'
'values': array of integers (must be an array, possibly with just one value)
In PHP it would be, for example:
$parameters = array('field' => 'id', 'values' => array(13));
It means: the user whose 'id' has the value of 13. Of course, you can use other parameters as well: ('field'=>'lastname', 'values'=> array('Smith'))
The parameters you can choose are the fields of the Moodle 'user' table.
Try to build these parameters in your postData variable.
Here's URL that work with my put this url in postman and set http method to post method
hostname/webservice/rest/server.php?wstoken=any_token&wsfunction=core_user_get_users_by_field&field=email&values[0]=h#fci.com
&moodlewsrestformat=json
I have to fetch the retweets for the tweets and create the JSON file with retweets,user id etc using the python script. Kindly help me to sort it our this issues.
Thanks in advance!!
This task require some fields of knowledge, and since you ask in a general way, I reckon you need a script to run immediately, but setting up this process requires sometime
This part to get connect to twitter API
from twython import Twython, TwythonError
APP_KEY = 'YOUR_APP_KEY'
APP_SECRET = 'YOUR_APP_SECRET'
twitter = Twython(APP_KEY, APP_SECRET)
Use Twitter API call from Twython,
you can find a list here https://twython.readthedocs.io/en/latest/api.html, the param is the same as twitter API
response = twitter.get_retweets(id, 100)
Pagnation
each call to API have limit of returns, in example for engine.get_friends_ids was limited to 5000 (https://dev.twitter.com/rest/reference/get/friends/ids), if you want to get more than 5000, you have to use the cursor in the returned result (if cur = 0 in json returned means no more results), following is example of how to handling cursor
#Set a temp to loop
cur = -1
#Stop when no more result
while cur !=0:
response = twitter.get_friends_ids(user_id=user_id, cursor=cur)
#Some code to handle the response
cur = response["next_cursor"]
API key
Key expires after some calls (https://dev.twitter.com/rest/public/rate-limits), so you need to set some code to auto change your key, or wait for some period (key reached limit return error code 429)
Response
The response from API was in JSON format, which was easy to use, you can access data by selecting base on response[key], in example
reponse["ids"] or response["next_cursor"]