I want to inflate HTML webpages. I am using zlib functions
inflateInit2(&zstream,15+32);
and then
inflate(&zstream,Z_SYNC_FLUSH);
It works for lots of webpages correctly but for "www.tabnak.ir" it does not work correctly.
invalid distance too far back is the ERROR I got for this website.
This webpage is also gzip and utf8.
How should I deal with that?
This is For Bing.com which works Fine
1f 8b 08 00 ef 8c 77 56 00 ff ec 5a eb 73 9c 46
12 ff 9e aa fc 0f 04 d5 9d ad 78 1f c0 3e b4 0b
96 52 b2 24 2b ba 73 1c 9d 2d 27 b9 8a af b6 06
This is For tabnak.ir which results in invalid distance too far back Error
1f 8b 08 00 00 00 00 00 00 03 ed fd db 73 5b d7
99 2f 8a 3e ab ab d6 ff 30 ac ae ac d8 3b 82 80
39 71 a7 6d 55 39 89 7b 75 f7 4a d2 7d 92 74 af
The zlib/gzip format performs compression saying things like "The next 22 bytes are the same as the 22 bytes we saw 1013 bytes ago.
In this case the record describing the repetition, is from before the size of the 'window'.
Given you have specified a maximum size of window, the likelihood, is that the data format has changed a bit, or the data you received is not the same as was sent.
Some things to check.
You are using the latest zlib library.
Standard utilities (e.g. gunzip, winzip) can decompress the data.
The data you are getting is not being mangled by a text filter ('rb' vs 'rt')
If that hasn't helped, try walking through the data and understanding what the failure in gzip is.
It would seem that the file you are trying to "inflate" (decompress using zlib) is not a valid zip file. Since bing.com is most likely not a zlib file, it might be pure coincidence that you found something quite early that prevented decompression.
Related
I'm experimenting. I created a .txt file with the word hi and run gzip test.txt to compress it.
This gives me a file test.txt.gz with the following bytes:
1F 8B 08 08 E6 E8 3F 60 00 03 62 2E 74 78 74 00
CB C8 04 00 AC 2A 93 D8 02 00 00 00
With the software 101 editor, I found out that the first line is the header.
CB C8 04 00 are the compressed data
AC 2A 93 D8 is "CRC of the data section"
02 00 00 00 is the "size of the uncompressed input"
What I'm trying to do (I don't know if it is even possible): I want to have my own characters as "compressed" data but want the .gz file to be still valid.
I tried replacing CB C8 04 00 with 62 62 62 62 (letter 'b' 4 times) but the file is invalid then. Then I tried to replace AC 2A 93 D8, too with the CRC32 value of "bbbb", but the file is still invalid. I can't decompress it. Running gzip -d test.txt.gz returns "unexpected end of file".
Is it possible what I'm trying to do? If yes: what am I doing wrong?
CB C8 04 00 is a valid deflate stream. 62 62 62 62 is not. A gzip member is a gzip header, a valid deflate stream, and a gzip trailer.
Deflate streams are defined in RFC 1951.
I'm having trouble with encrypted base64 encoded values I'm using in Google Deployment Manager via runtimeconfig.v1beta1.config resource declarations.
After I perform the deployment, the value that I stored using Deployment Manager appear to be quite different to what I retrieve using gcloud beta runtime-configs. As a result, I can't decrypt the value.
First I encrypted and base64 encoded some secret text:
$ echo "secret"|gcloud kms encrypt --key my-crypto-key \
--keyring my-keyring --location australia-southeast1 \
--plaintext-file - --ciphertext-file - | base64 -w0
CiQAsOSNmVXBs2ayUjRePnE5+Oi5dUPuVvjn6UKKUXgxMTA56koSMABDkVUGnXlocFgdUEsQ5qLCF3PVIz5zit+ZCSXjSvNzEAO5XRv6WBRkxBJMjVcheg==
Which I then store in a deployment manager YAML file:
resources:
- name: my-config
type: runtimeconfig.v1beta1.config
properties:
config: my-config
description: "A demo configuration"
- name: dummy-secret
type: runtimeconfig.v1beta1.variable
properties:
parent: $(ref.my-config.name)
variable: 'dummy/secret'
value: "CiQAsOSNmVXBs2ayUjRePnE5+Oi5dUPuVvjn6UKKUXgxMTA56koSMABDkVUGnXlocFgdUEsQ5qLCF3PVIz5zit+ZCSXjSvNzEAO5XRv6WBRkxBJMjVcheg=="
Then I create the deployment (which completes without errors or warnings):
$ gcloud deployment-manager deployments create my-config \
--config my-config.yaml
But when I try extracting the variable value, it is completely different from what I stored:
$ gcloud beta runtime-config configs variables \
get-value 'dummy/secret' --config-name my-config|base64 -w0
CiQAPz8/P1U/P2Y/UjRePnE5Pz8/dUM/Vj8/P0I/UXgxMTA5P0oSMABDP1UGP3locFgdUEsQPz8/F3M/Iz5zPz8/CSU/Sj9zEAM/XRs/WBRkPxJMP1cheg==
This is repeatable / reproducible and I haven't a clue what I'm doing wrong. I don't have this problem using gcloud beta runtime-config variables set followed by get-value.
Looking at the decoded base64 binary of your content, we notice that all the bytes with values >= 0x80 have been changed to 0x3F, ASCII '?'. We suspect you're passing the binary data through the shell or some other pipe which isn't binary-clean.
Corrupted value:
dierks#dierks:~$ base64 -d | hexdump -C
CiQAPz8/P1U/P2Y/UjRePnE5Pz8/dUM/Vj8/P0I/UXgxMTA5P0oSMABDP1UGP3locFgdUEsQPz8/F3M/Iz5zPz8/CSU/Sj9zEAM/XRs/WBRkPxJMP1cheg==
00000000 0a 24 00 3f 3f 3f 3f 55 3f 3f 66 3f 52 34 5e 3e |.$.????U??f?R4^>|
00000010 71 39 3f 3f 3f 75 43 3f 56 3f 3f 3f 42 3f 51 78 |q9???uC?V???B?Qx|
00000020 31 31 30 39 3f 4a 12 30 00 43 3f 55 06 3f 79 68 |1109?J.0.C?U.?yh|
00000030 70 58 1d 50 4b 10 3f 3f 3f 17 73 3f 23 3e 73 3f |pX.PK.???.s?#>s?|
00000040 3f 3f 09 25 3f 4a 3f 73 10 03 3f 5d 1b 3f 58 14 |??.%?J?s..?].?X.|
00000050 64 3f 12 4c 3f 57 21 7a |d?.L?W!z|
00000058
Original value:
dierks#dierks:~$ base64 -d | hexdump -C
CiQAsOSNmVXBs2ayUjRePnE5+Oi5dUPuVvjn6UKKUXgxMTA56koSMABDkVUGnXlocFgdUEsQ5qLCF3PVIz5zit+ZCSXjSvNzEAO5XRv6WBRkxBJMjVcheg==
00000000 0a 24 00 b0 e4 8d 99 55 c1 b3 66 b2 52 34 5e 3e |.$.....U..f.R4^>|
00000010 71 39 f8 e8 b9 75 43 ee 56 f8 e7 e9 42 8a 51 78 |q9...uC.V...B.Qx|
00000020 31 31 30 39 ea 4a 12 30 00 43 91 55 06 9d 79 68 |1109.J.0.C.U..yh|
00000030 70 58 1d 50 4b 10 e6 a2 c2 17 73 d5 23 3e 73 8a |pX.PK.....s.#>s.|
00000040 df 99 09 25 e3 4a f3 73 10 03 b9 5d 1b fa 58 14 |...%.J.s...]..X.|
00000050 64 c4 12 4c 8d 57 21 7a |d..L.W!z|
I am struggling for a few weeks with zlib inflate alghorithm.
I would like to decompress packets from popular game called Tibia. They have compressed it with zlib inflate alghorithms. But it seems something is changed. Can you check it, maybe you will spot something I am missing?
Packet compressed: - cannot be inflated directly (its raw huffman fixed coding - probably)
DA 22 A6 CB 10 99 5F AA 50 9C 9A AA 90 A8 90 05 B4 2F B5 44 41 C3 B1 28 D7 CA 50 53 8F CB B3 44 A1 3C 35 33 3D A3 58 C1 C8 44 CF C0 40 21 BF 4A 0F
Packet decompressed using Reverse Engineering:
B4 16 2D 00 59 6F 75 20 73 65 65 20 61 20 6A 61 63 6B 65 74 20 28 41 72 6D 3A 31 29 2E 0A 49 74 20 77 65 69 67 68 73 20 32 34 2E 30 30 20 6F 7A 2E
Decompressed packet compressed with zlib deflate using CyberChef deflate / php deflate functions:
db 22 a6 cb 10 99 5f aa 50 9c 9a aa 90 a8 90 95 98 9c 9d 5a a2 a0 e1 58 94 6b 65 a8 a9 c7 e5 59 a2 50 9e 9a 99 9e 51 ac 60 64 a2 67 60 a0 90 5f a5 07 00
Data matches in a few places, but generally, its different. Do you know what could be the case?
I attach a picture from RE an inflating function: Screenshot from IdaPro
Here are packets in correct order:
http://wklej.org/hash/6aee9e223f0/txt/ - inflated correctly
http://wklej.org/hash/bd371e7f510/txt/ - inflated correctly
http://wklej.org/hash/8f15935dc15/txt/ - inflated correctly
And here is the packet that cannot be inflated...
CA059BC6043619009FC9FFFFE831
Your packet that cannot be inflated is likely part of a longer stream of compressed data, with other packets preceding it and following it. You need to decompress all of them as a single stream for the decompression to succeed.
Your first example is a portion of a deflate stream that references data that preceded it. So it is part of a larger deflate stream. You need all of the compressed data that preceded that piece in order to decompress that piece. Your last example (CA05...) also references preceding data, so it too is part of a larger stream with compressed data that preceded it.
Background:
I am using the "base block" to power and access the UART2 console on the edison.
I have an arduino block with a sketch to print data to serial.
https://www.sparkfun.com/products/13036
The serial data is transmitted from the arduino and I intend to use a node script to read that data and perform network related functions.
Intel Edison and NodeJS serial port communication: I am receiving strange characters
I attempted to follow the edison serial instructions:
https://communities.intel.com/message/265411#265411
However at the GPIO configuration at line 2 it already fails.
root#edison:~# echo -n "214" > /sys/class/gpio/export
sh: write error: No such device
Interestingly if I run the "uart.js" from the "Intel Edison and NodeJS..." question I do get data, but it's not what's expected:
<Buffer 84 e9 f3 f4 e1 ee c3 e5 7a 20 72 75 75 0d 0a 8d c5 f3 f3 c1 c7 e5 20 f3 e5 ee f4 7a 60 70 0d 0a c2 c5 e1 ed 40 e2 f2 cf eb e5 ce 7a 60 30 0d 0a>
<Buffer 84 c9 f3 f4 e1>
<Buffer ce e3 e5 7a 60 72 75 75 0d 0a 8d e5 f3 f3 e1 e7 c5 60 f3 c5 ce f4 3a 40 70 0d 0a 82 e5 c1 ed 60 e2 f2 ef eb c5 ee 7a 40 70 0d 0a>
Lastly using mraa.getPinCount(); I always get 56 back. (Found that while searching around the net)
Thanks for the help!
It appears 115200 is an unusable baud for the Edison serial to Arduino block!
I set the baud to 9600 and have successfully gotten the expected data.
Additionally, the gpio pins for the Arduino block is 130 and 131. (130==rx, 131==tx)
echo -n 130 > /sys/class/gpio/export //may not need, may already be exported
echo -n 131 > /sys/class/gpio/export //may not need, may already be exported
echo mode1 > /sys/kernel/debug/gpio_debug/gpio130/current_pinmux // potentially unneeded as well.
echo mode1 > /sys/kernel/debug/gpio_debug/gpio131/current_pinmux // potentially unneeded as well.
echo in > /sys/class/gpio/gpio130/direction
echo out > /sys/class/gpio/gpio131/direction
..
cat /dev/ttyMFD1
I am currentlly on a project that requires me to load a JavaCard application Applet ( a .cap ) file to a JavaCard. Our framework is based on Visual C++ and PCSC, so we need to load the same applet to a series of JavaCards. Does anyone know how this can be processed? I mean, where should I start. Thanks!
You are correct that this is not a trivial job.
There are differences between different javacards, but generally you need to do 4 things:
initialize secure communications with the card (because many javacards are "global platform" cards they require a secure channel)
send a command saying "i wanna install an applet"
send the binary data for the applet to be installed
send a command to "instantiate" the applet after the binary data is sent
I'd recommend using the eclipse plugin to install the applet initially, because you can see the APDUs generated by the plugin to do the steps above. Once you know the APDU commands you must send to install your applet, you can directly send these commands using the PCSC interface from your C++ code to automate installation on a large number of cards.
My company makes a web browser plugin called Card Boss for doing this kind of thing (card communications via pcsc) from a browser - there's a web page you can use where you can type your own APDUs and send them to the card at the follwing URL:
https://cardboss.cometway.com/content.agent?page_name=Card+Boss+Lab
If you use our tool, your applet installation script should look something like this (note that this is a script for a JCOP card using the default jcop keys)
MESSAGE BOX Installing applets...
INIT CHANNEL 40 41 42 43 44 45 46 47 48 49 4a 4b 4c 4d 4e 4f, 40 41 42 43 44 45 46 47 48 49 4a 4b 4c 4d 4e 4f
// INSTALL CAP:
SEND 80 E6 02 00 1D 10 A0 00 00 00 09 00 03 FF FF FF FF 89 10 71 00 01 08 A0 00 00 00 03 00 00 00 00 00 00
// LOADING CAP:
SEND 80 E8 00 00 FA C4 82 01 03 01 00 25 DE CA FF (snip, I removed a bunch
of binary data representing the cap file to shorten this post, and you might
need multiple SEND commands because of limits on the size of APDUS)
// INSTANTIATING Applet
SEND 80 E6 0C 00 1E 05 63 6F 6D 65 74 07 63 6F 6D 65 74 00 01 05 00 00 00 00 00 01 00 06 C9 04 68 2C 00 03 00 00