This is a good way to grab the request before the response: useragent = getHttpRequestData().headers["User-Agent"];
What I noticed is that it will not grab the request unless it is on the actual list of header request. An example is I that it seems to only pull the basic request data. For instance if I set the cache control in the web.config file it does set cache, max age and etag, but when setting etags = getHttpRequestData().headers["ETag"]; and trying to output the data for the ETag generated by the web.config file/server it will not grab the ETag data to output. A few others that I tested are:
useragent = getHttpRequestData().headers["User-Agent"];
acceptencoding = getHttpRequestData().headers["Accept-Encoding"];
acceptlanugage = getHttpRequestData().headers["Accept-Language"];
cachecontrol = getHttpRequestData().headers["Cache-Control"];
connection = getHttpRequestData().headers["Connection"];
accept = getHttpRequestData().headers['Accept'];
contentlength = getHttpRequestData().headers['Content-Length'];
Request data is sent from the browser. You can see that with ColdFusion. But IIS sets response headers (such as etag) after ColdFusion is done processing. It's a response not a request. You cannot see that with ColdFusion, but you can in your browser. EX:
Related
As I see from the answer for this question: Karate will automatically send any cookies returned by the server in the next request.
But when I send the request I see two sets of cookies in Set-Cookie of response: one is auto-created and another is real, that returned from the server.
When I printed responseCookies, I saw there only automatic cookies
and for the next request new cookies are generated and sent.
For my test I need to use cookies returned after the first request because it is a call to login service.
Feature: Using cookies in next request
Background:
Given url baseUrl
And path LOGOUT_SERVICE_ENDPOINT
And configure headers = read('classpath:headers.js')
And def filename = 'classpath:resources/users/' + brand.toLowerCase() + '/user.json'
And json user = read(filename)
Scenario: Login
When def login = callonce read('classpath:features/login_service/login.feature') user
* print login.responseCookies
And request { arg1: '#(brand)'}
And method post
Then status 200
What is wrong in my feature or it is Karate issue?
two sets of cookies in Set-Cookie of response:
Maybe that is a bug in the server ?
Also try using "shared scope", because cookies also will be part of the "global" variables etc.
* callonce read('classpath:features/login_service/login.feature') user
* request { arg1: '#(brand)'}
If you are still stuck, please follow this process: https://github.com/intuit/karate/wiki/How-to-Submit-an-Issue
this code snippet is taken from Postman. cURL taken from the postman works fine and java code generated from postman gives a 200 response for the particular call. but the response body is not there.
what should be the user agent header?
Do I need to use this postman token in my java code as well?
Do I need to add additional headers?
My Goal is to fetch some data from this GET call.
OkHttpClient client = new OkHttpClient();
Request request = new Request.Builder()
.url("http://blahblah=60041441&attributes=blah,blah,blah")
.get()
.addHeader("User-Agent", "PostmanRuntime/7.13.0")
.addHeader("Accept", "*/*")
.addHeader("Cache-Control", "no-cache")
.addHeader("Postman-Token", "7af03a15-blah,364c160f-92d7-459f-b261-4993801944a7")
.addHeader("Host", "blahblah.na.blah.net:9081")
.addHeader("cookie", "someURL=1800; com.ibm.isim.lastActivity=blahblahToekn; JSESSIONID=blahblahblah:1ajblahi8; LtpaToken2=blahblahbalah")
.addHeader("accept-encoding", "gzip, deflate")
.addHeader("Connection", "keep-alive")
.addHeader("cache-control", "no-cache")
.addHeader("User-Agent", "postman")
.build();
okhttp3.Response response= client.newCall(request).execute();
System.out.println(response.body().toString());
Suppose for simple get request following will do just fine, all other details can be omitted:
Request request = new Request.Builder()
.url("http://blahblah=60041441&attributes=blah,blah,blah")
.get()
.build();
Most of the headers (like user-agent, accept-encoding etc) will be automatically added by OkHttp client, so you can safely remove those from request:
.addHeader("User-Agent", "PostmanRuntime/7.13.0")
.addHeader("Host", "blahblah.na.blah.net:9081")
.addHeader("accept-encoding", "gzip, deflate")
.addHeader("Cache-Control", "no-cache")
.addHeader("Connection", "keep-alive")
.addHeader("cache-control", "no-cache")
.addHeader("User-Agent", "postman")
Since / is a wildcard, suppose you can skip it as well.
.addHeader("Accept", "*/*")
If you endpoint requires authentication, suppose before sending this particular Get request you need to send authentication request first. To automatically handle authentication cookies you can try to add CookieJar to your client, so those can be omitted as well (assume headers names were altered somehow, btw?):
.addHeader("Postman-Token", "7af03a15-blah,364c160f-92d7-459f-b261-4993801944a7")
.addHeader("cookie", "someURL=1800; com.ibm.isim.lastActivity=blahblahToekn; JSESSIONID=blahblahblah:1ajblahi8; LtpaToken2=blahblahbalah")
You can also check answers for that question about the ways to add CookieJar.
I am generating a presigned URL server-side to allow my client application to upload a file directly to the S3 bucket. Everything works fine unless the client application is running on a computer in a timezone that is technically a day ahead of my server clock.
I can recreate the issue locally by setting my system clock ahead to a timezone on the next day.
Here is how I am generating the presigned URL using the .NET SDK (I originally had DateTime.Now instead of UTCNow):
var request = new GetPreSignedUrlRequest
{
BucketName = bucketName,
Key = objectName,
Verb = HttpVerb.PUT,
Expires = DateTime.UtcNow.AddDays(5),
ContentType = "application/octet-stream"
};
request.Headers["x-amz-acl"] = "bucket-owner-full-control";
request.Metadata.Add("call", JsonConvert.SerializeObject(call).ToString());
return client.GetPreSignedURL(request);
and then I am using that presigned URL in the client application like this:
using (var fileStream = new FileStream(recordingPath, FileMode.Open))
using (var client = new WebClient())
{
HttpContent fileStreamContent = new StreamContent(fileStream);
var bytes = await fileStreamContent.ReadAsByteArrayAsync();
client.Headers.Add("Content-Type", "application/octet-stream");
//include metadata in PUT request
client.Headers.Add("x-amz-meta-call", JsonConvert.SerializeObject(Call));
await client.UploadDataTaskAsync(new Uri(presignedUrl), "PUT", bytes);
}
Here is the error I am receiving from AWS:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>{access}</AWSAccessKeyId><StringToSign>....
The requests appear mostly identical to me in Fiddler.
Works:
PUT https://{bucketname}.s3.amazonaws.com/1c849c76-dd2a-4ff7-aad7-23ec7e9ddd45_encoded.opus?X-Amz-Expires=18000&x-amz-security-token={security_token}&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential={cred}&X-Amz-Date=20190312T021419Z&X-Amz-SignedHeaders=content-type;host;x-amz-acl;x-amz-meta-call;x-amz-security-token&X-Amz-Signature={sig} HTTP/1.1
x-amz-meta-call: {json_string}
x-amz-acl: bucket-owner-full-control
Content-Type: application/octet-stream
Host: {bucketname}.s3.amazonaws.com
Content-Length: 28289
Expect: 100-continue
{file}
Does not work:
PUT https://{bucketname}.s3.amazonaws.com/4cca3ec3-9f3f-4ba4-9d81-6336090610c0_encoded.opus?X-Amz-Expires=18000&x-amz-security-token={security_token}&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential={credentials}&X-Amz-Date=20190312T021541Z&X-Amz-SignedHeaders=content-type;host;x-amz-acl;x-amz-meta-call;x-amz-security-token&X-Amz-Signature={sig} HTTP/1.1
x-amz-meta-call: {json_string}
x-amz-acl: bucket-owner-full-control
Content-Type: application/octet-stream
Host: {bucketname}.s3.amazonaws.com
Content-Length: 18714
Expect: 100-continue
{file}
In both scenarios, the presigned URL has the same x-amz-date parameter generated. I have even tried parsing out the x-amz-date parameter from the URL and explicitly setting it as a header in my PUT but that did not work either.
What am I missing?
It turned out to me that I was using a different version of the signature.
v4 worked perfectly for me.
In JS, require S3 as
const s3 = new AWS.S3({
signatureVersion: 'v4'
});
So the issue ended up being within the metadata. In our setup, we had the client application posting a JSON string up to our API along with the file to generate the presigned URL. We were using Json.net to deserialize into the C# class:
var call = JsonConvert.DeserializeObject<Call>(request.Params["metadata"]);
Apparently, this call converts any timestamps in the Json to local time. This means that we would sign the URL with metadata timestamps local to the API server, but actually upload the file with metadata timestamps local to the client. This difference is why the calculated signatures are different.
I have created an API using AWS api gateway like https://api.mydomain.com/v1/download?id=1234". The download resource has GET method. And the GET method is invoking lambda function using Lambda Proxy Integration.
The Lambda function needs to act as Proxy. It needs to resolve correct backend endpoint based on header x-clientId and then forward the request to that backend endpoint and return response as it is. So it needs to be generic to handle GET request of different content-type.
My lambda function looks like ( .NET Core)
public async Task<APIGatewayProxyResponse> Route(APIGatewayProxyRequest input, ILambdaContext context)
{
var clientId = headers["x-clientId"];
var mappings = new Mappings();
var url = await mappings.GetBackendUrl(clientId, input.Resource);
var httpClient = new HttpClient();
var response = await httpClient.GetAsync(url);
response.EnsureSuccessStatusCode();
var proxyResponse = new APIGatewayProxyResponse()
{
Headers = new Dictionary<string, string>(),
StatusCode = (int)System.Net.HttpStatusCode.OK,
IsBase64Encoded = false,
Body = await response.Content.ReadAsString())
};
}
The handler above works as long as request and response's content-type is application/json or application/xml. However i am not sure how to handle response when backend returns stream.
For download API, the backend returns Content-Disposition: attachment; filename="somefilename and ContentType may be one of the following:
application/pdf
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
application/vnd.openxmlformats-officedocument.wordprocessingml.document
application/x-zip-compressed
application/octet-stream
For these streams, How do i set APIGatewayProxyResponse.Body?
For Excel file I have tried setting body like below
var proxyResponse = new APIGatewayProxyResponse()
{
Headers = new Dictionary<string, string>(),
StatusCode = (int)System.Net.HttpStatusCode.OK,
IsBase64Encoded = true,
Body = Convert.ToBase64String(await response.Content.ReadAsByteArrayAsync())
};
proxyResponse.Headers.Add("Content-Type", "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet");
proxyResponse.Headers.Add("Content-Disposition", "attachment; filename=\"Report.xlsx\"");
When i access the Url from the browser and try to open the file. I get error
Excel cannot open the fileReport.xlsxbecuase the file format or file extension is not valid. Verify that the file has not been corrupted and that the extention matches the format of the file
I think the issue is how i am setting the response body
Update 1
So based on AWS doc Binary Data Now Supported by API Gateway. Now as per the documentation
you can specify if you would like API Gateway to either pass the
Integration Request and Response bodies through, convert them to text
(Base64 encoding), or convert them to binary (Base64 decoding). These
options are available for HTTP, AWS Service, and HTTP Proxy
integrations. In the case of Lambda Function and Lambda Function Proxy
Integrations, which currently only support JSON, the request body is
always converted to JSON.
I am using Lambda Function Proxy, which currently support JSON. However the example here shows how to do it with Lambda Proxy.
I think what i am missing here is Binary Media Types setting and Method Response settings. Below is my setting. Not sure if these settings are correct
Binary Media
Method Response
here how solved it
1>add Binary Media Types. API->Settings->Binary Media Types -> add
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
2>In Method Response Add Content-Disposition and Content-Type headers for thestatus 200
3>In Integration Response map these headers to headers that are coming from the backend. And also set content handling convert to binary. (our backend api is returning file blob in body)
I am trying to load a web page and monitor it for XHR (XMLHttpRequests). To do this I am using Ghost.py with Python2.7. I can see the XHR being made and can read the URL and response, however I would like to read the request body so that I can recreate these requests at a later date.
from ghost import Ghost, Session
ghost = Ghost()
with ghost.start():
session = Session(ghost, download_images=False, display=False)
page, rs = session.open("https://www.example.com/", timeout=60)
assert page.http_status == 200
result, resources = session.wait_while_selector('[class=loading]:not([style="display: none;"])', timeout=60)
for resource in resources:
print resource.url
print resource.content
I have searched within the documentation, but cannot find any references to XHR request body and I have searched within the returned resources object for references to the request, but can only find request.headers (which does not include the POST body) and the _reply data.
Can you save the POST body of a XHR request as well as the response?