Using Microsoft.WindowsAzure.Storage 2.1.0.3
Attemping to write a conosle app to move documents from one Azure Storage account to another account. The app lists all the containers using sourceClient.ListContainers(), loops through all containers in a foreach block getting a Shared Access Token for each, and then fires a StartCopyFromBlob request for each blob. The destination blob has the same naming structure, but is in a different account (e.g. sourceAzureUrl/testContainer/filename.ext -> destAzureUrl/testContainer/filename.ext).
Most of the files (98%) copy just fine, but when certain requests are sent, it returns with this exception: "The remote server returned an error: (403) Forbidden." When the CloudBlockBlob reference to the destination blob is created, the URL does not seem to be properly URL escaped which results in the exception. However, when this code is run inside of an MVC controller, the request is somehow properly URL escaped and the request is completed without error. The copied blob contains the unescaped name as well.
It appears that the reason is the blob/filename contains a "[" and/or "]" character (e.g. Roger_Smith[1].doc). If the the filename is URL encoded beforehand, the request is completed without error, but the filename in Azure Storage is its URL escaped incarnation and not the original filename (Roger_Smith%255b1%255d.doc instead of Roger_Smith[1].doc).
Is there a way to properly URL escape the copy request and still have the result blob have the unescaped name?
Can you confirm you are running with .net 4.5? There is a uri escaping issue for the characters you mentioned (brackets) that was introduced in .net 4.5 that is incompatible with .net 4.0 (which is what the service is using to validate the response). As such it appears as if the message signature was incorrectly formed. We are working with the .net team on long term resolution for this. In the meantime you may consider running under .net 4.0 (with .net 4.5 installed to take advantage of GC improvements) or avoiding the bracket characters in the file name.
Related
So here is my case, I am trying to implement concurrency test using jMeter with over 100 users. I got the foundation set up. However, the challenge I am encountering is that I have two APIs on postman one which I need to get accident case as UIID that is the first API and the second API is an API in which I register the accident. Each accident api requires different accident case. In other words, all 100 users will have different accident case. I usually do that manually when manual test but how do I do that automated in jMeter
enter image description here
Thank you and best regards
You can use __UUID
"accidentCase": "${__UUID()}",
The UUID function returns a pseudo random type 4 Universally Unique IDentifier (UUID).
If you're getting an accident case in 1st request and need to use it in 2nd request you can extract it from the 1st request response using a suitable JMeter Post-Processor and save it into a JMeter Variable
Then in 2nd request you need to replace the 21d2a592-f883-45f7-a1c4-9f55413f01b9 with the JMeter Variable holding the UUID from the first request.
The process is known as correlation and there is a lot of information on it in the Internet.
I have a system where after a file is uploaded to S3, a Lambda job raises a queue message and I use it to maintain a list of keys in a MySQL table.
I am trying to generate a pre-signed URL based on the records in my table.
i have two records currently
/41jQnjTkg/thumbnail.jpg
/41jQnjTkg/Artist+-+Song.mp3
Generating pre-signed URL using :
var params = {
Bucket: bucket,
Expires: Settings.UrlGetTimeout,
Key: record
};
S3.getSignedUrl('getObject', params);
The URL with thumbnail.jpg works perfectly fine, but the one with +-+ fails. The original file name on local disk was "Artist - Song.mp3". S3 replaced spaces with '+'. Now when I am generating a URL using the exact same filename that S3 uses, it doesn't work; I get a "Specified Key doesn't exist" error from S3.
What must I do to generate URLs consistently for all filenames?
I solved this after a little experimentation.
Instead of directly storing key that S3 provides in their S3 event message, I am first replacing '+' character with space (as they are originally on the disk) and then URL decoding it.
return decodeURIComponent(str.replace(/\+/img, " "));
Now generating a S3 Pre-Signed URL works as expected.
Before MySQL has the following records:
/41jQnjTkg/thumbnail.jpg
/41jQnjTkg/Artist+-+Song.mp3
Now:
/41jQnjTkg/thumbnail.jpg
/41jQnjTkg/Artist - Song.mp3
I personally feel there is an inconsistency with S3's api/event messages.
Had i generated a Signed URL directly using the Key that S3 itself provided in SQS event message, It wouldn't have worked. One must do this string replacement step & URL decoding on the key in order to use it to get a proper working url.
Not sure if this is by design or a bug.
The second file's name is coming to you form-urlencoded. The + is actually a space, and if you had other characters (like parenthesis) they would be percent-escaped. You need to run your data through a URL decoder before working with it further.
Side-note: if the only thing your Lambda function does is create an SQS message, you can do that directly from S3 without writing your own function.
I'm upgrading a website to a .NET website using MVC5 using ImageResizer with the images stored on AWS S3. The images stored on S3 are fine, have public read access and load without a problem when calling the S3 URL.
When I use the ImageResizer plugin S3Reader2 I get the following error on most of my images: "The string was not recognized as a valid DateTime. There is an unknown word starting at index 26."
You can find the ImageResizer Diagnostics here: Diagnostics
You can find the stack trace here: Stacktrace
Any help would be highly appreciated!
The failing blobs have an invalid Expires header set. Mon, 28 Apr 2025 21:50:04 G4T does not conform to the RFC for datetime values in HTTP headers, and AWSSDK correctly throws an exception when it encounters the malformed date.
The bad metadata should be replaced or removed from the failing blobs.
In the web app I am testing, a unique authorization code is generated at runtime and submitted with each http request during that [web browser] session. Jmeter script recorder stores the hardcoded value, which again is only valid during that session.
I can use the Regular Expression Extractor to capture the authorization code, which is generated on the 10th http request in the test plan. I need to tell JMETER to use this new authorization code in all subsequent http requests in the test. I can manually replace all subsequent occurrences of the code with the new value extracted, but this would mean hundreds of manual changes.
Is there any way to tell JMETER that from a given point going forward, ignore the hardcoded value and use the value extracted during the 10th call. This is important because each test plan has hundreds of calls and there will be multiple test plans for the web site.
THX
Don't use hard-coded value as in runtime it will be quite hard to replace it. Use a JMeter Variable instead. Something like ${AUTH_CODE} wherever it is required.
Make sure that you use the same Reference Name AUTH_CODE in your Regular Expression Extractor at 10th request of your test plan.
If your Test Plan contains multiple thread groups use JMeter Property instead. See How to Use Variables in Different Thread Groups for step-by-step implementation guide
How does one go about automating the creation of Webservices on the IBMi?
Using the GUI at http://my.ibmi:2001/HTTPAdmin is a painfully slow process, and prone to fat finger errors. Thus my quest for automation.
During my research, I was unable to find the CL commands to export my RPGLE *SRVPGM procedures as Web Services.
Automation for IBMi V6R1 or V7R1.
Reference Sites:
http://www.ibm.com/systems/power/software/i/iws/
http://www-03.ibm.com/systems/resources/integrated_web_application_server_information.pdf
http://www-03.ibm.com/systems/resources/systems_i_software_iws_pdf_WebServicesClient_new.pdf
CL based method for adding HTTP Servers on IBMi system
http://www.ibm.com/support/knowledgecenter/ssw_ibm_i_61
Answer found at
http://www.volubis.fr/news/liens/courshtm/AS400web/V6R1_LWI.htm
All scripts relating to Webservices are in
/QIBM/ProdData/OS/WebServices/V1/server/bin
Step:
STRQSH
cd /QIBM/ProdData/OS/WebServices/V1/server/bin
installWebService.sh -h
Command usage:
installWebService.sh
-server 'server-name' -programObject 'program-object'
[-service 'service-name'] [-pcml 'pcml-file'] [-userid 'userid']
[-libraryList 'library-list'] [-libraryListPosition '*FIRST|*LAST']
[-disableNillableWSDLElements] [-disableOptionalWSDLElements]
[-addUnderscoreToWSDLElementNames] [-libraryListPosition '*FIRST|*LAST']
[-printErrorDetails] [-help]
Where:
server is the name of the Web services server in which the service
will be installed.
programObject is the integrated file system path to the ILE program or
service program. For example, '/QSYS.LIB/MYLIB.LIB/MYPGM.[SRVPGM or PGM]'
service is the name of Web service to be installed. This parameter is
optional. If not specified, the program object name will be used.
pcml is a path to the PCML file. This parameter is optional. If not
specified, the program object must contain the PCML data.
userid is the user profile the Web service will be modified to run under.
This parameter is optional. If not specified, the Web service will run
under the Web services server user profile. Note: The Web service server's
user profile needs to have *USE authority to the user profile specified on
this parameter. A value of *SERVER will ensure the Web service runs under the
same user profile as the Web services server.
libraryList is a list of libraries, that will be added to the library
list prior to invoking the Web service. Each library in the list must
be delimited by a colon. This parameter is optional.
libraryListPosition is the position in the user portion of the job
library list where the list of libraries specified in -libraryList will be
placed. A value of *FIRST inserts the libraries at the beginning of
the user portion of the library list. A value of *LAST inserts the libraries
at the end of the user portion of the library list. This parameter is
optional. If not specified the default value of *LAST is used.
disableNillableWSDLElements will define elements in generated WSDL as
not nillable. If not specified, elements are defined as nillable.
This parameter is ignored for version 1.3 of Web services engine.
disableOptionalWSDLElements will define elements in generated WSDL as
not optional. If not specified, elements are defined as optional.
This parameter is ignored for version 1.3 of Web services engine.
addUnderscoreToWSDLElementNames will generate WSDL element names
that start with the underscore (_) character, as was always done in
version 1.3 of the Web services engine. If not specified, element names
in WSDL will not start with the underscore character.
This parameter is ignored for version 1.3 of Web services engine.
printErrorDetails will show additional error information, such as stack
traces and error codes, if the command fails. This parameter is optional.
help displays the command syntax. This parameter is optional.