How to process an incoming Payload(SOAP) which is COMPRESSED, in my webservices which I have created using Axis ???
Assuming you'r content is GZip Encoded
1.Check the content encoding. If it is gzip then
2.wrapper the input stream as a GZIPInputStream
read as usual from input stream.
Related
When I call a Lambda function from Lex by calling my number set up in AWS Connect, intent_request['outputDialogMode'] always seems to return "text" when using a simple if/else statement.
Is this expected?
That is not what I would expect, but if that is what you are getting then AWS must be doing it differently than what we would think.
Based on the Input/Response Format (bold added):
"outputDialogMode": "Text or Voice, based on ContentType request header in runtime API request",
And the ContentType is set depending on what type of content is being sent.
Based on PostContent API:
contentType
You pass this value as the Content-Type HTTP header.
Indicates the audio format or text. The header value must start with one of the following prefixes:
PCM format, audio data must be in little-endian byte order.
audio/l16; rate=16000; channels=1
audio/x-l16; sample-rate=16000; channel-count=1
audio/lpcm; sample-rate=8000; sample-size-bits=16; channel-count=1; is-big-endian=false
Opus format
audio/x-cbr-opus-with-preamble; preamble-size=0; bit-rate=256000; frame-size-
milliseconds=4
Text format
text/plain; charset=utf-8
Required: Yes
Basically, the outputDialogMode is set to Voice only when the contentType is set to either PCM or Opus audio format.
So this must mean that AWS Connect is not actually sending the audio input to your Lex bot. It therefore must be processing the audio through speech-to-text elsewhere and then passing the text output to your Lex bot instead.
Unfortunately, this means that outputDialogMode is not a reliable indicator of a voice channel unless you are using PostContent API to send an audio input file to your Lex bot yourself.
There is a limit of 256KB as the max size of message which can be published to AWS-SNS. Can we compress a message using GZIP and send publish the compressed message to overcome the size limit ?
You can gzip the message body -- however -- SNS message bodies only support UTF-8 character data. Gzipped data is binary, so that is not directly compatible with SNS because not every possible sequence of bytes is also a valid sequence of UTF-8 characters.
So, after gzipping your payload, you need to encode that binary data using a scheme such as base-64. Base-64 encodes arbitrary binary data (8 bits per byte) using only 64 (which is 2^6, giving effectively 6 bits per byte) symbols and so the byte count inflates by 8/6 (133%) as a result of this encoding. This means 192KB of binary data encodes to 256KB of base-64-encoded data, so the maximum allowable size of your message after gzip becomes 192K (since the SNS limit is 256KB). But all the base-64 symbols are valid single-byte UTF-8 characters, which is a significant reason why this encoding is so commonly used, despite its size increase. That, and the fact that gzip typically has a compression ratio far superior to 1.33:1 (which is the break-even point for gzip + base-64).
But if your messages will gzip to 192K or lower, this definitely does work with SNS (as well as SQS, which has the same character set and size limits).
You already take a look at this? https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html
If you think that the file can increase on the time I suggest another approach.
Put the file on S3 bucket and attach the S3 Event Notification to SNSTopic so all consumer will be notified when a new file is ready to be processed.
In other word the message of the SNS will be the location of the file and not the file it self.
Think about it.
You can also use the SNS/SQS extended client library for large message payloads.
https://aws.amazon.com/about-aws/whats-new/2020/08/amazon-sns-launches-client-library-supporting-message-payloads-of-up-to-2-gb
I have an issue trying to decompress an imap message compressed using deflate method. The things I've tryed so far were isolating one of the directions of an IMAP conversation (using wireshark's follow tcp function) and saving the message data in an raw format that I hope it contains only the deflated message part. I then found some programs like tinf (1st and 3rd example) and miniz (tgunzip example) and tryed to inflate back that file, but with no succes.
I am missing something? Thank you in advance.
tinf - http://www.ibsensoftware.com/download.html
Miniz - https://code.google.com/archive/p/miniz/source/default/source
Try piping that raw data to:
perl -MCompress::Zlib -pe 'BEGIN{$i = inflateInit(-WindowBits => -15)}
$_=$i->inflate($_)'
The important part is the -WindowBits => -15 that changes the expected format into a raw one without adler checksum.
(that's derived from the dovecot source, works for me on Thunderbird to gmail network capture).
From RFC4978 that specifies IMAP compression (emphasis mine):
When using the zlib library (see RFC1951), the functions
deflateInit2(), deflate(), inflateInit2(), and inflate() suffice to
implement this extension. The windowBits value must be in the range
-8 to -15, or else deflateInit2() uses the wrong format.
deflateParams() can be used to improve compression rate and resource
use. The Z_FULL_FLUSH argument to deflate() can be used to clear the
dictionary (the receiving peer does not need to do anything).
i write a c++ http server ( Microsoft http server api )
that send html page file in gzip format
and gzip file is static
for example file page1.htm and page1.htm.gz are in same directory
according to
http://en.wikipedia.org/wiki/Gzip
i know that gzip is deflate with extra header
and deflate is part of gzip
how i can sending gzip instade deflate by skip header
fileHandle=CreateFile( "page1.htm.gz" ,dwAccess,dwShare,NULL,dwCreationFlags,dwAttributes,NULL);
....
ADD_KNOWN_HEADER(response, HttpHeaderContentEncoding, "deflate" );
HTTP_DATA_CHUNK dataChunk;
{
HTTP_DATA_CHUNK dataChunk;
response.EntityChunkCount = 1;
dataChunk.DataChunkType = HttpDataChunkFromFileHandle;
dataChunk.FromFileHandle.FileHandle =fileHandle;
dataChunk.FromFileHandle.ByteRange.StartingOffset.QuadPart =9;// 9 is gzip header len
dataChunk.FromFileHandle.ByteRange.Length.QuadPart = HTTP_BYTE_RANGE_TO_EOF;
response.pEntityChunks=&dataChunk;
}
.....
The deflate and gzip encoding are not quite the same thing, although the differences are minor.
When you are sending gzip, change your code to:
ADD_KNOWN_HEADER(response, HttpHeaderContentEncoding, "gzip" );
You should do that of course if gzip is listed in Accept-Encoding.
Here's an excerpt from the gzip FAQ:
“ What's the difference between the "gzip" and "deflate" HTTP 1.1
encodings?
"gzip" is the gzip format, and "deflate" is the zlib format. They
should probably have called the second one "zlib" instead to avoid
confusion with the raw deflate compressed data format. While the HTTP
1.1 RFC 2616 correctly points to the zlib specification in RFC 1950 for the "deflate" transfer encoding, there have been reports of
servers and browsers that incorrectly produce or expect raw deflate
data per the deflate specficiation in RFC 1951, most notably
Microsoft. So even though the "deflate" transfer encoding using the
zlib format would be the more efficient approach (and in fact exactly
what the zlib format was designed for), using the "gzip" transfer
encoding is probably more reliable due to an unfortunate choice of
name on the part of the HTTP 1.1 authors.”
http://www.gzip.org/zlib/zlib_faq.html#faq38
I have a X bytes file. And I want to compress it in block of size 32Kb, for example.
Is there any lib that Can I do this?
I used Zlib for Delphi but I just can compress a full file in new compressed file.
Tranks a lot,
Pedro
Why don't you use a simple header to determine block boundaries? Consider this:
Read fixed amount of data from input into a buffer (say 32 KiB)
Compress that buffer with a "freshly created" deflate stream (underlying compression algorithm of ZLIB).
Write compressed size to output stream
Write compressed data to output stream
Go to step 1 until you reach end-of-file.
Pros:
You can decompress any block even in multi-threaded fashion.
Data corruption only limited to corrupted block. Rest of data can be restored.
Cons:
You loss most of contextual information (similarities between data). So, you will have lower compression ratio.
You need slightly more work.