I am trying to use the Amazon-S3 REST API to upload large file in chunks.
As per the API documentation here I formed my request with postman as follows.
After Initiating CreateMultipartUpload Post Request, I'm successfully getting UploadId for chunk put requests.
This is Working Fine.
I understand Multipart order, but when executing the step uploading partNumber & UploadId chunks using POSTMAN , I'm getting SHA256Mismatch error because POSTMAN calculating Content-MD5 for whole file not for chunks.
Header
Params
Body
I have found multiple solutions on various forums but those solutions didn't work.
Am I missing something here?
Successfully uploaded to Amazon S3 using PostMan using Amazon's multipart upload and the key (for me) was to add a Content-MD5 header manually, and paste in the Base64 encoded MD5 hash for that part (details below). This may not be the exact problem the OP was having, but still, I wanted to share how to use PostMan, provided you have good working IAM key id and secret to your Amazon S3 bucket.
First, I split a 9 MB "mytest.pdf" file into two parts to test this (I used Linux/WSL command: split -b 5242880 mytest.pdf) making sure the first part is larger than 5MB (the last part can be smaller than 5 MB).
Next, set up PostMan with following four requests:
CreateMultipartUpload (e.g., POST
https://{{mybucket}}.s3.{{myregion}}.amazonaws.com/mytest.pdf?uploads
UploadPart1 (e.g., PUT
https://{{mybucket}}.s3.{{myregion}}.amazonaws.com/mytest.pdf?partNumber=1&uploadId=297a2XMl9kNDqw1BaKl7jk6uK_Mop0mCV68TmWU2n8xjsrM6sgt0hu.93J92Qgw8yaEHlrlj0MSoc9ljmU3sD3dlQsGJixMq9hugPDRTkikM0KV6rmLdpmHjFcWzDEDO)
UploadPart2 (e.g., PUT
https://{{mybucket}}.s3.{{myregion}}.amazonaws.com/mytest.pdf?partNumber=2&uploadId=297a2XMl9kNDqw1BaKl7jk6uK_Mop0mCV68TmWU2n8xjsrM6sgt0hu.93J92Qgw8yaEHlrlj0MSoc9ljmU3sD3dlQsGJixMq9hugPDRTkikM0KV6rmLdpmHjFcWzDEDO)
CompleteMultipartUpload (e.g., POST
https://{{mybucket}}.s3.{{myregion}}.amazonaws.com/mytest.pdf?uploadId=297a2XMl9kNDqw1BaKl7jk6uK_Mop0mCV68TmWU2n8xjsrM6sgt0hu.93J92Qgw8yaEHlrlj0MSoc9ljmU3sD3dlQsGJixMq9hugPDRTkikM0KV6rmLdpmHjFcWzDEDO)
I pasted in my IAM key id and access key in the authorization section of PostMan (separate articles for that)
Ran the CreateMultipartUpload POST to get the UploadId from Amazon.
Next, calculated MD5 hash of each of my two file parts (I used 7-zip but use tool of your choice). Now, converted that result into Base64. However, I had to make sure that I ended up with 22 characters followed optionally by two equals signs == . When I converted MD5 as text to Base64, I ended up with a longer string, which ended with a single equal sign, not double (this is a sign that it is not encoded in a way that Amazon expects it and will produce an InvalidDigest error). For example, if you use 7-zip to calculate MD5 hash on file part and get the value 58942651efd0f5886810d04ed9df502f, and then use a tool such as the Bases64 encoder below and choose input at Text you get NTg5NDI2NTFlZmQwZjU4ODY4MTBkMDRlZDlkZjUwMmY= but if you choose "HEX" as the input, you get a smaller string WJQmUe/Q9YhoENBO2d9QLw== that is 24 characters (22 characters + 2 equals signs). That's the one you want.
(no implied endorsement of this tool - not affiliated https://emn178.github.io/online-tools/base64_encode.html)
If you get it wrong, Amazon will reply with the InvalidDigest error below
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>InvalidDigest</Code>
<Message>The Content-MD5 you specified was invalid.</Message>
<Content-MD5>thisisbad</Content-MD5>
<RequestId>8274DC9566D4AAA8</RequestId>
<HostId>H6kSy4cl+54nMon1Hq6AGjmTX/MfTVMQQr8vEVNXUnPlfMtIt8HPdObfusckhBpwpG/CJ6ORWv16c=</HostId>
</Error>
Ran both UploadPart1 and 2
Finally ran CompleteMultipartUpload with copied and pasted Etag values from the Headers of the previous 2 requests
<CompleteMultipartUpload>
<Part>
<PartNumber>1</PartNumber>
<ETag>"c716d98e83db1edb27fc25fd03e0ae32"</ETag>
</Part>
<Part>
<PartNumber>2</PartNumber>
<ETag>"58942651efd0f5886810d04ed9df502f"</ETag>
</Part>
</CompleteMultipartUpload>
I am load testing a web application scenario in which:
1.) First I need to upload an excel file it will be parsed and sheets (tabs in excel) loads into a drop-down.
2.)Then selects an option in drop-down and clicks on import button.
3.) After clicking on import a new pop up open with some data loaded in the grid with "update to database" button. Now clicked on the update to database button.
This is my scenario of load test here I need to load test "import" and "update to database" requests
In this scenario 4 requests works.
In point 1: servicename url is: /xxx/xxx/uploadLoader.aspx
in params are: 2 viewstates, button, appuser, company, filename (with multi-part data)
response: it returns HTML of file upload control
Point 2 using 2 requests:
a) service url is: /xxxx/xxxx.asmx/updateImportSubTableFormat
in params:
{"as_connstring":["GET"],"alist_import":[{"__type":"ApplecartDS.DataLib.ImportTableMain","rowstate":"OLD","tableno":"019","oldtableno":"019","formatid":"I0000012","oldformatid":"I0000012","columnname":"pay_billrefno","oldcolumnname":"","columndesc":"Bill No","aormdesc":"Actual","excelcolumnname":"D","excelcolumnno":"3","excelcolumnnameadd1":"","excelcolumnnoadd1":-1,"excelcolumnnameadd2":"","excelcolumnnoadd2":-1,"serialno":"1","oldserialno":2,"formatname":"","startingrowno":0,"endingrowno":0,"keycolumn":"","keycolumndesc":"","payiddesc":null,"attributetype":"","paytype":"A",".............**some more output data here**.................... 1,"excelcolumnnameadd2":"","excelcolumnnoadd2":-1,"serialno":"3","oldserialno":3,"formatname":"","startingrowno":0,"endingrowno":0,"keycolumn":"","keycolumndesc":"","payiddesc":null,"attributetype":"","paytype":"A",""provision_actual":"P","dsexecstatus":"","dsmsg":"","slno":0,"as_fromatid":"I0000012","ai_xlcolumnname":"A","ai_xlcolumnno":"1","ai_startrow":"2","ai_endrow":0,"ai_header":"1","as_payid":"MEDRE","as_leaveid":"MEDRE","as_wfdcolname":"","ai_wfdcolno":0,"as_arrcolname":"","ai_arrcolno":0,"as_transafertype":"A"}
response:
{"d":[{"__type":"ApplecartDS.DataLib.ImportTableMain","rowstate":"OLD","tableno":"","oldtableno":"","formatid":"","oldformatid":"","columnname":"","oldcolumnname":"","columndesc":"","aormdesc":"","excelcolumnname":"","excelcolumnno":0,"excelcolumnnameadd1":"","excelcolumnnoadd1":-1,"excelcolumnnameadd2":"","excelcolumnnoadd2":-1,"serialno":0,"oldserialno":0,"formatname":"","startingrowno":0,"endingrowno":0,.............**some more output data here**...................."","valuefrom":"E","default_value":"","aorm":"A","benefitsid":"","licpayid":"","createmaster":"","allowzero":"N","groupid":"","groupdesc":"","reimbpayid":"","provision_actual":"P","dsexecstatus":"","dsmsg":"","slno":0}]}
Note: If I hit same request wit same in parmas then it returns correct same response.
b) service url is: /AAA/AAA.asmx/importXLData
in params:
{"as_connstring":["GET"],"as_filename":"C:\\Websites\\xxxx\\xxxx\\admin\\001\\admin115.248.100.6825520164659134Import_Template.xlsx","ai_header":2,"as_tablename":"rm_claimsmain","as_sheetname":"'Reim claim$'","as_tableno":"019","as_fromatid":"I0000012","as_creditperiodid":"","isUpdate":false,"isupdatemonattribute":false,"alistUpdatedata":[],"as_excelfilepath":"C:\\Websites\\xxxx\\xxxx\\admin\\001\\admin115.248.100.6825520164659134Import_Template.xlsx"}
response
{"d":[{"__type":"ApplecartDS.DataLib.ExcelDataImport","CValues1":"","CValues2":"","CValues3":"","CValues4":"","CValues5":"","CValues6":"","CValues7":"","CValues8":"",..........**very long out put data here**.......
,"NValues50":0,"NValues51":0,3":0,"IValues4":0,"rowmodify":"N","rowstate":"OLD","dsexecstatus":null,"dsmsg":""}]}
Note: But if I hit the same request again then it displays response as below that are different from above response (first-time response). Because the file name changes every time in case of the manual but I am not able to get the new file name to pass it in JMeter request.
second-time response:
{"d":[{"__type":"ApplecartDS.DataLib.ExcelDataImport","CValues1":"","CValues2":"","CValues3":"","CValues4":"","CValues5":"","CVal","CValues17":"","CValues18":"","CValues19":................................. dsexecstatus":"FAIL","dsmsg":"Exception : System.Data.OleDb.OleDbException (0x80004005): The Microsoft Office Access database engine could not find the object \u0027C:\\xxxx\\xxxx\\xxx\\admin\\001\\admin115.248.100.6825520165121629Import_Template_test_case_9May2016.xlsx\u0027. Make sure the object exists and that you spell its name and the path name correctly.
point 3 is using 1 request
service url: /xxxx/xxxx.asmx/importXLData
in params
{"as_connstring":["GET"],"as_filename":"C:\\xxxx\\xxxx\\temp\\admin\\001\\admin115.248.100.682552016571324rm_claimsmain.xml;C:\\Websites\\cPayv8html\\temp\\admin\\001\\admin115.248.100.682552016571324rm_cliams.xml","ai_header":"1","as_tablename":"rm_claimsmain","as_sheetname":"'Reim claim$'","as_tableno":"019","as_fromatid":"I0000012","as_creditperiodid":"","isUpdate":true,"isupdatemonattribute":false,"alistUpdatedata":[{"__type":"ApplecartDS.DataLib.ExcelDataImport","CValues1":"Employee No;C;30;em_empno","CValues2":"Bill No;C;30;","CValues3":"Bill Date;D;30;","CVal/Date(-62135596800000)\/"..........**some more in data**..........,"DValues10":"\/Date(-62135596800000)\/","ls_msg1":"5","ls_msg2":"C:\\xxxx\\xxxx\\xx\\admin\\001\\admin115.248.100.682552016571324rm_claimsmain.xml;C:\\xxxx\\xxx\\xx\\admin\\001\\admin115.248.100.682552016571324rm_cliams.xml","ls_msg3":"","ls_msg4":"","NValues1":0,"NV4":0,"rowmodify":"N","rowstate":"OLD","dsexecstatus":null,"dsmsg":""}],"as_excelfilepath":"C:\\xxxx\\xxxx\\xxx\\admin\\001\\admin115.248.100.6825520164659134Import_Template_test_case_9May2016.xlsx"}
response
{"d":[{"__type":"ApplecartDS.DataLib.ExcelDataImport","CValues1":"","CValues2":"","CValues3":"","CValues4":"","CValues5":"","CValues6":"","CValues7":"","CValues8":"","CValues9":"","CValues10":"","CValues11":"","CValues12":"","CValues13":"","CValues14":"","CValues15":"","CValues16":"","CValues17":"","CValues18":"","................................ "rowmodify":"N","rowstate":"OLD","dsexecstatus":"SUCCESS","dsmsg":"Update to Database success ! New Rows added :0 \u003c\u003e Rows Updated : 49"}]}
If I hit same request again then second time response is:
......................"dsexecstatus":"FAIL","dsmsg":"Exception : Could not find file \u0027C:\\Websites\\cPayv8html\\temp\\admin\\001\\admin115.248.100.682552016525264rm_claimsmain.xml\u0027."}]}
My issue is in second-time response it is not showing correct response as the file name is changing randomly and I am not able to trace it to pass changed file path to the request.
Is there any way to handle it? Please help me out, how can I load test it with an authentic response.
${__RandomString(6,a12zeczclk, MYVAR)}
will return a random string of 6 characters picked from a12zeczclk set and store the result in MYVAR, MYVAR will contain string like 2z22ak or z11kce, …
see:
http://jmeter.apache.org/usermanual/functions.html#__RandomString
Looking into
Could not find file \u0027C:\Websites\cPayv8html\temp\admin\001\admin115.248.100.682552016525264rm_claimsmain.xml\u0027.
It seems that when you upload the file, it's being stored somewhere in a temporary location and when the file is imported into the database - it's deleted. So it's classic "correlation" example where you need to get a dynamic parameter from previous response and add it to the next request.
I would suggest carefully inspect server responses and to get the temporary filename from there.
As your server responds with JSON I would suggest to use
JSON Path PostProcessor - if you're using JMeter 3.0
JSON Path Extractor - if you're sitting on a previous JMeter version
JSON Path Syntax is very easy, check the following references:
JSONPath - XPath for JSON
JSONPath Online Evaluator
Advanced Usage of the JSON Path Extractor in JMeter