What is wrong with my timestamp in my MWS request? - amazon-web-services

If I submit a request to MWS via the scratchpad(AmazonServices/Scratchpad),
it is successful, and I am able to view the details of the successful request. In particular, the timestamp on the request looks like this:
&Timestamp=2018-08-14T18%3A30%3A02Z
If I literally take this timestamp, as is, and try to use it in my code to make the same exact request, I get an error:
<Message>Timestamp 2018-08-14T18%3A30%3A02Z must be in ISO8601
format</Message>\n
Here is the function I am trying to place it in: (some chars changed in sensitive params)
exports.sendRequest = () => {
return agent
.post('https://mws.amazonservices.com/Products/2011-10-01')
.query({
AWSAccessKeyId: encodeURIComponent('BINAJO5TPTZ5TTRLNGQA'),
Action: encodeURIComponent('GetMatchingProductForId'),
SellerId: encodeURIComponent('H1N1R958BK8TTH'),
SignatureVersion: encodeURIComponent('2'),
Timestamp: '2018-08-14T18%3A30%3A02Z',
Version: encodeURIComponent('2011-10-01'),
Signature: encodeURIComponent(exports.generateSignature()),
SignatureMethod: encodeURIComponent('HmacSHA256'),
MarketplaceId: encodeURIComponent('ATVPDKIKX0DER'),
IdType: encodeURIComponent('UPC'),
'IdList.Id.1': encodeURIComponent('043171884536')
})
.then(res => {
console.log('here is the response');
console.log(res)
})
.catch(error => {
console.log('here is the error');
console.log(error);
})
}
What is even more strange, is that this is the path the request is being sent to:
path: '/Products/2011-10-01?
AWSAccessKeyId=BINAJO5ZPTZ5YTTPNGQA&Action=GetMatchingProductForId&SellerId=H1N1R958ET8THH&SignatureVersion=2&Timestamp=2018-08-14T18%253A30%253A02Z&Version=2011-10-01&Signature=LwZn5of9NwCAgOOB0jHAbYMeQT31M6y93QhuX0d%252BCK8%253D&SignatureMethod=HmacSHA256&MarketplaceId=ATVPDKIKX0DER&IdType=UPC&IdList.Id.1=043171884536' },
The timestamp is not the same as the one I placed in the query. Why is this happening?

Your HTTP library is already doing the url-encoding for you, so you're double-encoding things. Remove all references to encodeURIComponent() and format your timestamp normally, with : and not %3A. Observe what happens to the generated URL.
Why? URL-encoding isn't safe to do repeatedly.
: becomes %3A with one pass, but it becomes %253A with a second pass, which is wrong.

Related

PayloadTooLargeError for Expo in React Native

What is causing the PayloadTooLargeError error? I get it sometimes and also when the payload is a few KB (as far as I can figure out).
PayloadTooLargeError: request entity too large
at readStream (/usr/local/lib/node_modules/expo-cli/node_modules/#expo/dev-server/node_modules/raw-body/index.js:155:17)
at getRawBody (/usr/local/lib/node_modules/expo-cli/node_modules/#expo/dev-server/node_modules/raw-body/index.js:108:12)
at read (/usr/local/lib/node_modules/expo-cli/node_modules/#expo/dev-server/node_modules/body-parser/lib/read.js:77:3)
at jsonParser (/usr/local/lib/node_modules/expo-cli/node_modules/#expo/dev-server/node_modules/body-parser/lib/types/json.js:135:5)
at call (/usr/local/lib/node_modules/expo-cli/node_modules/connect/index.js:239:7)
at next (/usr/local/lib/node_modules/expo-cli/node_modules/connect/index.js:183:5)
at serveStatic (/usr/local/lib/node_modules/expo-cli/node_modules/serve-static/index.js:75:16)
at call (/usr/local/lib/node_modules/expo-cli/node_modules/connect/index.js:239:7)
at next (/usr/local/lib/node_modules/expo-cli/node_modules/connect/index.js:183:5)
I found some solutions that you can set the limit to a higher value, but that's not specifically for Expo.io
There is no console.log used in the app
The error you are seeing could be caused by one of the packages you are using which uses body parser.
In body parser there is an option to limit to request body size:
limit
Controls the maximum request body size. If this is a number, then the value specifies the number of bytes; if it is a string, the value is passed to the bytes library for parsing. Defaults to '100kb'.
Taken from here.
You can see a related SO questions here and here.
I also saw this GitHub issue for Expo-Cli
I had the same problem, after a lot of trials, I figure it out. The issue is related to the way you fetch the data, you are continuously fetching the data from database, which cause this error.
The solution is to fetch the data only once,
useEffect(() => {
}, []);
Here is code example to
useEffect(() => {
const fetchUser = async () => {
try {
let user = await AsyncStorage.getItem('user_id');
let parsed = JSON.parse(user);
setUserId(parsed);
//Service to get the data from the server to render
fetch('http://myIpAddress/insaf/mobileConnection/ClientDashboard/PrivateReplyConsulationTab.php?ID=' + parsed)
//Sending the currect offset with get request
.then((response) => response.json())
.then((responseJson) => {
//Successful response from the API Call
setOffset(offset + 1);
const pri = Object.values(responseJson[0].reply_list);
setPrivateReplies(pri);
setLoading(false);
})
.catch((error) => {
console.error(error);
});
}
catch (error) {
alert(error + "Unkbown user name")
}
}
fetchUser();
}, []);
I think the storage is merging the old requests, so can you reset the async storage
AsyncStorage.clear()

POSTMAN not sending anything in the body in POST requests

I am trying to test the /login API via POSTMAN (via FE it works fine) but it doesn't
show anything in the body part even though I am sending body data.
but when printing the request from the BE, the body is empty...
....
body: {},
....
unlike when using FE:
....
body: {data: { username: 'admin', password: 'admin' }},
....
Any idea what's going on? If anything else is needed to be provided - pls let me know
I know it's going through because the server responds with 500 and the message
TypeError: Cannot read property 'username' of undefined
The weird part is, that the data I am sending, are nowhere to be found in the request object at all :(
EDIT:
This is how I call it from the FE:
return axios.post('login', { data: user })
and the user is:
user: {
username: 'admin',
password: 'admin'
}
So the format should be right
data: {
username: 'admin',
password: 'admin'
}
Because that's how I access it on the BE side
req.body.data.username
EDIT2:
The ultra-super-rare-weird part is, that JEST is working fine :)
const creds = {
data: {
username: 'admin',
password: 'admin'
}
}
return request(app)
.post("/api/v1/login")
.send(creds)
.expect(200)
.then(res => {
expect(res.body).toMatchSnapshot()
})
and this test passes .... f**k me guys.. what's going on?
if you are working with an express server try to parse your body in the server as soon as you initialize express app before routing
const app = express();
app.use(express.json());
The syntax of your body looks like JSON, yet you've specified the type of the body as "raw text". This will set the Content-type header of your request to "text/plain", which is probably causing your backend to be unable to actually read the body (because it expects a JSON object).
Simply switch from "Text" to "JSON", wrap your current body in curly braces (so that you're actually sending a single JSON object with a data property set) and try sending the request again. The content-type header will be correctly set to "application/json" this time and your backend will read the data successfully.
Add the following parameters in the request headers configuration in Postman:
Content-Type: application/json
Content-Length
The value of Content-Length is automatically calculated by Postman when a request is sent. Both values are used to identify the media type of the request body and to parse it accurately. When missing, the body may be ignored completely (depending on the server).

Is it theoretically possible to use `putRecord` in Kinesis via singular http POST request?

I've been experiencing some issues with AWS Kinesis inasmuch as I have a stream set up and I want to use a standard http POST request to invoke a Kinesis PutRecord call on my stream. I'm doing this because bundle-size of my resultant javascript application matters and I'd rather not import the aws-sdk to accomplish something that should (on paper) be possible.
Just so you know, I've looked at this other stack overflow question about the same thing and It was... sort of informational.
Now, I already have a method to sigv4 sign a request using an access key, secret token, and session token. but when I finally get the result of signing the request and send it using the in-browser fetch api, the service tanks with (or with a json object citing the same thing, depending on my Content-Type header, I guess) as the result.
Here's the code I'm working with
// There is a global function "sign" that does sigv4 signing
// ...
var payload = {
Data: { task: "Get something working in kinesis" },
PartitionKey: "1",
StreamName: "MyKinesisStream"
}
var credentials = {
"accessKeyId": "<access.key>",
"secretAccessKey": "<secret.key>",
"sessionToken": "<session.token>",
"expiration": 1528922673000
}
function signer({ url, method, data }) {
// Wrapping with URL for piecemeal picking of parsed pieces
const parsed = new URL(url);
const [ service, region ] = parsed.host.split(".");
const signed = sign({
method,
service,
region,
url,
// Hardcoded
headers : {
Host : parsed.host,
"Content-Type" : "application/json; charset=UTF-8",
"X-Amz-Target" : "Kinesis_20131202.PutRecord"
},
body : JSON.stringify(data),
}, credentials);
return signed;
}
// Specify method, url, data body
var signed = signer({
method: "POST",
url: "https://kinesis.us-west-2.amazonaws.com",
data : JSON.stringify(payload)
});
var request = fetch(signed.url, signed);
When I look at the result of request, I get this:
{
Output: {
__type: "com.amazon.coral.service#InternalFailure"},
Version: "1.0"
}
Now I'm unsure as to whether Kinesis is actually failing here, or if my input is malformed?
here's what the signed request looks like
{
"method": "POST",
"service": "kinesis",
"region": "us-west-2",
"url": "https://kinesis.us-west-2.amazonaws.com",
"headers": {
"Host": "kinesis.us-west-2.amazonaws.com",
"Content-Type": "application/json; charset=UTF-8",
"X-Amz-Target": "Kinesis_20131202.PutRecord",
"X-Amz-Date": "20180613T203123Z",
"X-Amz-Security-Token": "<session.token>",
"Authorization": "AWS4-HMAC-SHA256 Credential=<access.key>/20180613/us-west-2/kinesis/aws4_request, SignedHeaders=content-type;host;x-amz-target, Signature=ba20abb21763e5c8e913527c95a0c7efba590cf5ff1df3b770d4d9b945a10481"
},
"body": "\"{\\\"Data\\\":{\\\"task\\\":\\\"Get something working in kinesis\\\"},\\\"PartitionKey\\\":\\\"1\\\",\\\"StreamName\\\":\\\"MyKinesisStream\\\"}\"",
"test": {
"canonical": "POST\n/\n\ncontent-type:application/json; charset=UTF-8\nhost:kinesis.us-west-2.amazonaws.com\nx-amz-target:Kinesis_20131202.PutRecord\n\ncontent-type;host;x-amz-target\n508d2454044bffc25250f554c7b4c8f2e0c87c2d194676c8787867662633652a",
"sts": "AWS4-HMAC-SHA256\n20180613T203123Z\n20180613/us-west-2/kinesis/aws4_request\n46a252f4eef52991c4a0903ab63bca86ec1aba09d4275dd8f5eb6fcc8d761211",
"auth": "AWS4-HMAC-SHA256 Credential=<access.key>/20180613/us-west-2/kinesis/aws4_request, SignedHeaders=content-type;host;x-amz-target, Signature=ba20abb21763e5c8e913527c95a0c7efba590cf5ff1df3b770d4d9b945a10481"
}
(the test key is used by the library that generates the signature, so ignore that)
(Also there are probably extra slashes in the body because I pretty printed the response object using JSON.stringify).
My question: Is there something I'm missing? Does Kinesis require headers a, b, and c and I'm only generating two of them? Or is this internal error an actual failure. I'm lost because the response suggests nothing I can do on my end.
I appreciate any help!
Edit: As a secondary question, am I using the X-Amz-Target header correctly? This is how you reference calling a service function so long as you're hitting that service endpoint, no?
Update: Followinh Michael's comments, I've gotten somewhere, but I still haven't solved the problem. Here's what I did:
I made sure that in my payload I'm only running JSON.stringify on the Data property.
I also modified the Content-Type header to be "Content-Type" : "application/x-amz-json-1.1" and as such, I'm getting slightly more useful error messages back.
Now, my payload is still mostly the same:
var payload = {
Data: JSON.stringify({ task: "Get something working in kinesis" }),
PartitionKey: "1",
StreamName: "MyKinesisStream"
}
and my signer function body looks like this:
function signer({ url, method, data }) {
// Wrapping with URL for piecemeal picking of parsed pieces
const parsed = new URL(url);
const [ service, region ] = parsed.host.split(".");
const signed = sign({
method,
service,
region,
url,
// Hardcoded
headers : {
Host : parsed.host,
"Content-Type" : "application/json; charset=UTF-8",
"X-Amz-Target" : "Kinesis_20131202.PutRecord"
},
body : data,
}, credentials);
return signed;
}
So I'm passing in an object that is partially serialized (at least Data is) and when I send this to the service, I get a response of:
{"__type":"SerializationException"}
which is at least marginally helpful because it tells me that my input is technically incorrect. However, I've done a few things in an attempt to correct this:
I've run JSON.stringify on the entire payload
I've changed my Data key to just be a string value to see if it would go through
I've tried running JSON.stringify on Data and then running btoa because I read on another post that that worked for someone.
But I'm still getting the same error. I feel like I'm so close. Can you spot anything I might be missing or something I haven't tried? I've gotten sporadic unknownoperationexceptions but I think right now this Serialization has me stumped.
Edit 2:
As it turns out, Kinesis will only accept a base64 encoded string. This is probably a nicety that the aws-sdk provides, but essentially all it took was Data: btoa(JSON.stringify({ task: "data"})) in the payload to get it working
While I'm not certain this is the only issue, it seems like you are sending a request body that contains an incorrectly serialized (double-encoded) payload.
var obj = { foo: 'bar'};
JSON.stringify(obj) returns a string...
'{"foo": "bar"}' // the ' are not part of the string, I'm using them to illustrate that this is a thing of type string.
...and when parsed with a JSON parser, this returns an object.
{ foo: 'bar' }
However, JSON.stringify(JSON.stringify(obj)) returns a different string...
'"{\"foo\": \"bar\"}"'
...but when parsed, this returns a string.
'{"foo": "bar"}'
The service endpoint expects to parse the body and get an object, not a string... so, parsing the request body (from the service's perspective) doesn't return the correct type. The error seems to be a failure of the service to parse your request at a very low level.
In your code, body: JSON.stringify(data) should just be body: data because earlier, you already created a JSON object with data: JSON.stringify(payload).
As written, you are effectively setting body to JSON.stringify(JSON.stringify(payload)).
Not sure if you ever figured this out, but this question pops up on Google when searching for how to do this. The one piece I think you are missing is that the Record Data field must be base64 encoded. Here's a chunk of NodeJS code that will do this (using PutRecords).
And for anyone asking, why not just use the SDK? I currently must stream data from a cluster that cannot be updated to a NodeJS version that the SDK requires due to other dependencies. Yay.
const https = require('https')
const aws4 = require('aws4')
const request = function(o) { https.request(o, function(res) { res.pipe(process.stdout) }).end(o.body || '') }
const _publish_kinesis = function(logs) {
const kin_logs = logs.map(function (l) {
let blob = JSON.stringify(l) + '\n'
let buff = Buffer.from(blob, 'binary');
let base64data = buff.toString('base64');
return {
Data: base64data,
PartitionKey: '0000'
}
})
while(kin_logs.length > 0) {
let data = JSON.stringify({
Records: kin_logs.splice(0,250),
StreamName: 'your-streamname'
})
let _request = aws4.sign({
hostname: 'kinesis.us-west-2.amazonaws.com',
method: 'POST',
body: data,
path: '/?Action=PutRecords',
headers: {
'Content-Type': 'application/x-amz-json-1.1',
'X-Amz-Target': 'Kinesis_20131202.PutRecords'
},
}, {
secretAccessKey: "****",
accessKeyId: "****"
// sessionToken: "<your-session-token>"
})
request(_request)
}
}
var logs = [{
'timeStamp': new Date().toISOString(),
'value': 'test02',
},{
'timeStamp': new Date().toISOString(),
'value': 'test01',
}]
_publish_kinesis(logs)

Why does React fetch retrieve blank object?

I have a working Django REST API which returns this:
{
"id": 1,
"brand": "peugeot",
"model": "3008",
"variant": "allure"
}
I am using the following code to fetch the above data:
render() {
const { brand, model, variant } = this.props;
let url = `http://127.0.0.1:8000/api/car/${brand}/${model}/${variant}/`;
console.log(url) <== url is correct when checked in console
fetch(url)
.then(response => response.json())
.then(data => data.length === 0 ? this.setState({
data : data
}) : null ) <== I have used a condition for setState to stop fetching infintely
const { data } = this.state;
console.log(data) <== This is a blank object with no data in console
console.log(data.id) <== This is undefined in console
return (
<div>
{data.id} <== No data is shown on webpage
Car Details
</div>
);
}
No error is shown when I try to fetch the data on my webpage. What am I doing wrong?
P.S. Data can be fetched from the same API server when I have an array of objects, and I use map to loop over the data. Over here I am trying to fetch a single item so there is no array, just an object. Am I doing something wrong with the syntax?
You should never fetch or setState inside the render function.
render is called many times due to all kinds of side effects, i.e scrolling, clicking, props changing etc. This kind of code could cause all kinds of trouble.
If you need to perform the request once, call the fetch function inside componentDidMount. Also, I believe your callbacks should look something like this:
fetch(url)
.then(response => response.json())
.then(data => this.setState({ data : data }))
Taken from the docs:
componentDidMount() is invoked immediately after a component is mounted. Initialization that requires DOM nodes should go here. If you need to load data from a remote endpoint, this is a good place to instantiate the network request.
I changed the condition before 'setState' to JSON.stringify(data) !== JSON.stringify(this.state.data) and now it works.
should it be:
.then(data => data.length > 0 ? this.setState({ data }) : null )

Mobile Application Using Sencha Touch - JSON Request Generates Syntax Error

I started playing a bit with Sencha Touch.
So I've built a really simple application based on one of the examples just to see how it goes.
Basically it creates a JSON Request which executes a Last.FM web service to get music events near the user's location.
Here's the JSON code:
var makeJSONPRequest = function() {
Ext.util.JSONP.request({
url: 'http://ws.audioscrobbler.com/2.0/',
params: {
method: 'geo.getEvents',
location: 'São+Paulo+-+SP',
format: 'json',
callback: 'callback',
api_key: 'b25b959554ed76058ac220b7b2e0a026'
},
callback: function(result) {
var events = result.data.events;
if (events) {
var html = tpl.applyTemplate(events);
Ext.getCmp('content').update(html);
}
else {
alert('There was an error retrieving the events.');
}
Ext.getCmp('status').setTitle('Events in Sao Paulo, SP');
}
})
};
But every time I try to run it, I get the following exception:
Uncaught SyntaxError: Unexpected token :
Anyone has a clue?
A couple of things. First of all the "Uncaught SyntaxError: Unexpected token :" means the browser javascript engine is complaining about a colon ":" that has been put in the wrong place.
The problem will most likely be in the returned JSON. Since whatever the server returns will be run though the eval("{JSON HTTP RESULT}") function in javascript, the most likely thing is that your problem is in there somewhere.
I've put your code on a little sencha test harness and found a couple of problems with it.
First: My browser was not too happy with the "squiggly ã" in location: 'São+Paulo+-+SP', so I had to change this to location: 'Sao+Paulo,+Brazil', which worked and returned the correct results from the audioscribbler API.
Second: I notice you added a callback: 'callback', line to your request parameters, which changes the nature of the HTTP result and returns the JSON as follows:
callback({ // a function call "callback(" gets added here
"events":{
"event":[
{
"id":"1713341",
"title":"Skank",
"artists":{
"artist":"Skank",
"headliner":"Skank"
},
// blah blah more stuff
"#attr":{
"location":"Sao Paulo, Brazil",
"page":"1",
"totalpages":"1",
"total":"2"
}
}
}) // the object gets wrapped with extra parenthesis here
Instead of doing that I think you should be using the callbackKey: 'callback' that comes with the example in http://dev.sencha.com/deploy/touch/examples/ajax/index.js.
Something like this for example:
Ext.util.JSONP.request({
url: 'http://ws.audioscrobbler.com/2.0/',
params: {
method: 'geo.getEvents',
location: 'Sao+Paulo,+Brazil',
format: 'json',
api_key: 'b25b959554ed76058ac220b7b2e0a026'
},
callbackKey: 'callback',
callback: function(result) {
// Output result to console (Firebug/Chrome/Safari)
console.log(result);
// Handle error logic
if (result.error) {
alert(result.error)
return;
}
// Continue your code
var events = result.data.events;
// ...
}
});
That worked for me so hopefully it'll work for you too. Cherio.