base64 decode string - emacs different than jvm? - clojure

Question

With the base64-encoded string JVBERi0xLjENCiXi48/TDQoxIDAgb2JqDQo8PCAN I am getting difference results from emacs than from the clojure code below.
Can anyone explain to me why?
The elisp below gives the correct output, giving me ultimately a valid pdf document (when i past the entire string). I am sure my emacs buffer is set to utf-8:
(base64-decode-string "JVBERi0xLjENCiXi48/TDQoxIDAgb2JqDQo8PCAN")
"%PDF-1.1
%âãÏÓ
1 0 obj
<<
Here is the same output with the chars in decimal (i think):
"%PDF-1.1
%\342\343\317\323
1
The clojure below gives incorrect output, rendering the pdf document invalid when i give the entire string:
(import 'java.util.Base64 )
(defn decode [to-decode]
(let [
byts (.getBytes to-decode "UTF-8")
decoded (.decode (java.util.Base64/getDecoder) byts)
]
(String. decoded "UTF-8")))
(decode "JVBERi0xLjENCiXi48/TDQoxIDAgb2JqDQo8PCAN")
"%PDF-1.1
%����
1 0 obj
<<
Same output, chars in decimal (i think). I couldn't even copy/paste this, i had to type it in. This is what it looks like when i opened the PDF in text-mode for the first three columns:
"%PDF-1.1
%\357\277\275\357\277\275\357\277\275\357\277\275
1"
Edit Taking emacs out of the equation:
If i write the encoded string to a file called encoded.txt and pipe it through the linux program base64 --decode i get valid output and a good pdf also:
This is clojure:
(defn decode [to-decode]
(let [byts (.getBytes to-decode "ASCII")
decoded (.decode (java.util.Base64/getDecoder) byts)
flip-negatives #(if (neg? %) (char (+ 255 %)) (char %))
]
(String. (char-array (map flip-negatives decoded)) )))
(spit "./output/decoded.pdf" (decode "JVBERi0xLjENCiXi48/TDQoxIDAgb2JqDQo8PCAN"))
(spit "./output/encoded.txt" "JVBERi0xLjENCiXi48/TDQoxIDAgb2JqDQo8PCAN")
Then this at the shell:
➜ output git:(master) ✗ cat encoded.txt| base64 --decode > decoded2.pdf
➜ output git:(master) ✗ diff decoded.pdf decoded2.pdf
2c2
< %áâÎÒ
---
> %����
➜ output git:(master) ✗
update - this seems to work
Alan Thompson's answer below put me on the correct track, but geez what a pain to get there.
Here's the idea of what works:
(def iso-latin-1-charset (java.nio.charset.Charset/forName "ISO-8859-1" ))
(as-> some-giant-string-i-hate-at-this-point $
(.getBytes $)
(String. $ iso-latin-1-charset)
(base64/decode $ "ISO-8859-1")
(spit "./output/a-pdf-that-actually-works.pdf" $ :encoding "ISO-8859-1" ))

Answer 1

Returning the results as a string, I get:
(b64/decode-str "JVBERi0xLjENCiXi48/TDQoxIDAgb2JqDQo8PCAN")
=> "%PDF-1.1\r\n%����\r\n1 0 obj\r\n<< \r"
and as a vector of ints:
(mapv int (b64/decode-str "JVBERi0xLjENCiXi48/TDQoxIDAgb2JqDQo8PCAN"))
=> [37 80 68 70 45 49 46 49 13 10 37 65533 65533 65533 65533 13 10 49 32 48
32 111 98 106 13 10 60 60 32 13]
Since both the beginning and end of the string look OK, I suspect the B64 string might be malformed?
Update
I went to http://www.base64decode.org and got the result
"Malformed input... :("
Update #2
The root of the problem is that the source characters are not UTF-8 encoded. Rather, they are ISO-8859-1 (aka ISO-LATIN-1) encoded. See this code:
(defn decode-bytes
"Decodes a byte array from base64, returning a new byte array."
[code-bytes]
(.decode (java.util.Base64/getDecoder) code-bytes))
(def iso-latin-1-charset (java.nio.charset.Charset/forName "ISO-8859-1" )) ; aka ISO-LATIN-1
(let [b64-str "JVBERi0xLjENCiXi48/TDQoxIDAgb2JqDQo8PCAN"
bytes-default (vec (.getBytes b64-str))
bytes-8859 (vec (.getBytes b64-str iso-latin-1-charset))
src-byte-array (decode-bytes (byte-array bytes-default))
src-bytes (vec src-byte-array)
src-str-8859 (String. src-byte-array iso-latin-1-charset)
]... ))
with result:
iso-latin-1-charset => <#sun.nio.cs.ISO_8859_1 #object[sun.nio.cs.ISO_8859_1 0x3edbd6e8 "ISO-8859-1"]>
bytes-default => [74 86 66 69 82 105 48 120 76 106 69 78 67 105 88 105 52 56 47 84 68 81 111 120 73 68 65 103 98 50 74 113 68 81 111 56 80 67 65 78]
bytes-8859 => [74 86 66 69 82 105 48 120 76 106 69 78 67 105 88 105 52 56 47 84 68 81 111 120 73 68 65 103 98 50 74 113 68 81 111 56 80 67 65 78]
(= bytes-default bytes-8859) => true
src-bytes => [37 80 68 70 45 49 46 49 13 10 37 -30 -29 -49 -45 13 10 49 32 48 32 111 98 106 13 10 60 60 32 13]
src-str-8859 => "%PDF-1.1\r\n%âãÏÓ\r\n1 0 obj\r\n<< \r"
So the java.lang.String constructor will work correctly with a byte[] input, even when the high bit is set (making them look like "negative" values), as long as you tell the constructor the correct java.nio.charset.Charset to use for interpreting the values.
Interesting that the object type is sun.nio.cs.ISO_8859_1.
Update #3
See the SO question below for a list of libraries that can (usually) autodetect the encoding of a byte stream (e.g. UTF-8, ISO-8859-1, ...)
What is the most accurate encoding detector?

Answer 2

I think you need to verify the actual bytes that are produced in both scenarios. I would save both decoded results in a file and then compare them using for example xxd command line tool to get the hex display of the bytes in the files.
I suspect your emacs and clojure application uses different font causing the same non-ASCII bytes to be rendered differently, e.g. the same byte value is rendered as â in emacs and � in clojure output.
I would also check if elisp indeed creates the resulting string using UTF-8. base64-decode-string mentions unibytes and I am not sure it's really UTF-8. Unibyte sounds like encoding characters using always one byte per character whereas UTF-8 uses one to four bytes per character.

Answer 3

Update
#glts made a correct point in his comment to the question. If we go to http://www.utilities-online.info/base64/ (for example), and we try to decode the original string, we get a third, different result:
%PDF-1.1
%⣏Ӎ
1 0 obj
<<
However, if we try to encode the data the OP posted, we get a different Base64 string: JVBERi0xLjEKICXDosOjw4/DkwogMSAwIG9iagogPDwg, which if we run using the original decode implementation as written by the OP we get the same output:
(decode "JVBERi0xLjEKICXDosOjw4/DkwogMSAwIG9iagogPDwg")
"%PDF-1.1\n %âãÏÓ\n 1 0 obj\n << "
No need to make any conversions. I guess you should check out the encoder.
Original answer
This problem is due to java's Byte being signed.. So much fun!
When you convert it to string, it truncates all negative values to 65533, which is plain wrong:
(map long (decode "JVBERi0xLjENCiXi48/TDQoxIDAgb2JqDQo8PCAN"))
;; (37 80 68 70 45 49 46 49 13 10 37 65533 65533 65533 65533 13 10 49 32 48 32 111 98 106 13 10 60 60 32 13)
lets see what happens:
(defn decode [to-decode]
(let [byts (.getBytes to-decode "UTF-8")
decoded (.decode (java.util.Base64/getDecoder) byts)]
decoded))
(into [] (decode "JVBERi0xLjENCiXi48/TDQoxIDAgb2JqDQo8PCAN"))
;; [37 80 68 70 45 49 46 49 13 10 37 -30 -29 -49 -45 13 10 49 32 48 32 111 98 106 13 10 60 60 32 13]
See the negatives? lets try to fix that:
(into [] (char-array (map #(if (neg? %) (char (+ 255 %)) (char %))(decode "JVBERi0xLjENCiXi48/TDQoxIDAgb2JqDQo8PCAN"))))
;; [\% \P \D \F \- \1 \. \1 \return \newline \% \á \â \Î \Ò \return \newline \1 \space \0 \space \o \b \j \return \newline \< \< \space \return]
And if we turn this into a string, we get what emacs gave us:
(String. (char-array (map #(if (neg? %) (char (+ 255 %)) (char %)) (decode "JVBERi0xLjENCiXi48/TDQoxIDAgb2JqDQo8PCAN"))))
;; "%PDF-1.1\r\n%áâÎÒ\r\n1 0 obj\r\n<< \r"