I am lost with decoding the following base64 string
nVJPb4IwFL/7KUjvAgUM8CIuZiabicsSNR68deXhWKBteGVx336FbJnz4MG+U997/f1L5yTaxsBGn3Rvt0hGK0LPO7eNIhhnBes7BVpQTaBEiwRWwm75soHID8F02mqpGzZZrwpGScZjkUgpMolpFCfRLH/DPKlmaZXGMkqrMq/CMi6Zd8COaq0K5lCYtybqca3ICmVdK+TZlIfTONxzDtEMeHZk3grJ1krY8dW7tQaCgEepH7rikLoTEHaf2AWNPtXqodUlFonDVr++9rpgH1jq82BsusT8eWPa1yd9RLHdf7HFZD4MYBTTXWRwOwJBjnZQxRaDKnKy6tL4RFrWnWzQl7qdBxfIPzwGdlbYnu4I+wrh0Tm9A8U7iKbH28s0EsCulxKJBuLgmvm693f//6sW3w==
It should be valid base64 data representing deflate data of original XML. When I try online decoder here: https://www.samltool.com/decode.php it gives me the proper XML.
I am doing these two steps:
string text = MyClass::decode_base64(input);
text = MyClass::stringDeflateDecode(text);
First I decode the base64 string:
string MyClass::decode_base64(string str)
{
using namespace boost::archive::iterators;
typedef transform_width<binary_from_base64<remove_whitespace<string::const_iterator> >, 8, 6> ItBinaryT;
try {
boost::erase_all(str, "\r");
boost::erase_all(str, "\n");
// If the input isn't a multiple of 4, pad with =
size_t num_pad_chars((4 - str.size() % 4) % 4);
str.append(num_pad_chars, '=');
size_t pad_chars(std::count(str.begin(), str.end(), '='));
std::replace(str.begin(), str.end(), '=', 'A'); // replace '=' by base64 encoding of '\0'
string output(ItBinaryT(str.begin()), ItBinaryT(str.end()));
output.erase(output.end() - pad_chars, output.end());
return output;
} catch (...) {
return string("");
}
}
The code is basically from here Decode Base64 String Using Boost and it was working fine for text-only base64 decoding (no binary deflate data).
Then I would like to decode the deflate:
string MyClass::stringDeflateDecode(const std::string& data)
{
stringstream compressed(data);
stringstream decompressed;
boost::iostreams::filtering_streambuf<boost::iostreams::input> in;
in.push(boost::iostreams::zlib_decompressor());
in.push(compressed);
boost::iostreams::copy(in, decompressed);
return decompressed.str();
}
but ::copy operation throws an exception: zlib error: iostream error
Thanks for any hints!
That is Base-64 encoded raw deflate data. That means compressed data in the deflate format, but no zlib nor gzip wrapper around that deflate data. It looks like zlib_decompressor has a noheader option that you should set to true.
Wikipedia specifies:
SAML requests or responses transmitted via HTTP Redirect have a SAMLRequest or SAMLResponse query string parameter, respectively. Before it's sent, the message is deflated (without header and checksum), base64-encoded, and URL-encoded, in that order. Upon receipt, the process is reversed to recover the original message.
The problem here is the absense of the header and checksum. I don't think boost has the library functions you need.
Related
Please explain if this is a Zlib bug or I misunderstand the use of Zlib.
I am trying to do the following:
-I have two strings - data from which I need to compress: string_data_1 and string_data_2 and which I compress with Zlib as raw data.
-Next, I create a third string and copy the already compressed data into this single row.
-Now I'm decompressing this combined compressed data and there is a problem.
Zlib decompressed only the "first" part of the compressed data, did not decompress the second part. Is that how it should be?
For an example in the facebook/zstd:Zstandard library - exactly the same action - leads to unpacking - all compressed data and the first and second parts.
Here is a simple code:
#include <iostream>
#include <string>
#include <zlib.h>
int my_Zlib__compress__RAW(std::string& string_data_to_be_compressed, std::string& string_compressed_result, int level_compressed)
{
//-------------------------------------------------------------------------
uLong zlib_uLong = compressBound(string_data_to_be_compressed.size());
string_compressed_result.resize(zlib_uLong);
//-------------------------------------------------------------------------
//this is the standard Zlib compress2 function - with one exception: the deflateInit2 function is used instead of the deflateInit function and the windowBits parameter is set to "-15" so that Zlib compresses the data as raw data:
int status = my_compress2((Bytef*)&string_compressed_result[0], &zlib_uLong, (const Bytef*)&string_data_to_be_compressed[0], string_data_to_be_compressed.size(), level_compressed);
if (status == Z_OK)
{
string_compressed_result.resize(zlib_uLong);
return 0;
}
else
{
return 1;
}
}
int my_Zlib__uncompress__RAW(std::string& string_data_to_be_uncompressed, std::string& string_compressed_data, size_t size_uncompressed_data)
{
//-------------------------------------------------------------------------
string_data_to_be_uncompressed.resize(size_uncompressed_data);
//-------------------------------------------------------------------------
//this is the standard Zlib uncompress function - with one exception: the inflateInit2 function is used instead of the inflateInit function and the windowBits parameter is set to "-15" so that Zlib uncompresses the data as raw data:
int status = my_uncompress((Bytef*)&string_data_to_be_uncompressed[0], (uLongf*)&size_uncompressed_data, (const Bytef*)&string_compressed_data[0], string_compressed_data.size());
if (status == Z_OK)
{
return 0;
}
}
int main()
{
int level_compressed = 9;
//------------------------------------------Compress_1-------------------------------------------
std::string string_data_1 = "Hello12_Hello12_Hello125"; //The data to be compressed.
std::string string_compressed_result_RAW_1; //Compressed data will be written here
int status = my_Zlib__compress__RAW(string_data_1 , string_compressed_result_RAW_1, level_compressed);
//----------------------------------------------------------------------------------------------
//--------------------------------------Compress_2----------------------------------------------
std::string string_data_2= "BUY22_BUY22_BUY223"; //The data to be compressed.
std::string string_compressed_result_RAW_2; //Compressed data will be written here
status = my_Zlib__compress__RAW(string_data_2 , string_compressed_result_RAW_2, level_compressed);
//----------------------------------------------------------------------------------------------
std::string Total_compressed_data = string_compressed_result_RAW_1 + string_compressed_result_RAW_2; //Combine two compressed data into one string
//Now I want to uncompress the data in a string - "Total_compressed_data"
//--------------------------------------Uncompress--------------------------------
std::string string_uncompressed_result_RAW; //Uncompressed data will be written here
int size_that_should_be_when_unpacking = string_data_1.size() + string_data_2.size();
status = my_Zlib__uncompress__RAW(string_uncompressed_result_RAW, Total_compressed_data, size_that_should_be_when_unpacking , level_compressed);
//--------------------------------------------------------------------------------
std::cout<<string_uncompressed_result_RAW<<std::endl; //Hello12_Hello12_Hello125
}
Zlib decompressed only the "first" part of the compressed data, did not decompress the "second" part.
Is that how it should be?
As noted in the comments, a concatenation of zlib streams is not a zlib stream. You need to uncompress again for the second zlib stream. Or compress the whole thing as one zlib stream in the first place.
You would need to use a variant of uncompress2(), not uncompress(), since the former will return the size of the first decompressed zlib stream in the last parameter, so that you know where to start decompressing the second one.
Better yet, you should use the inflate() functions instead for your application. The retention of the uncompressed size for use in decompression means that you'd need that on the other end. How do you get that? Are you transmitting it separately? You do not need that. You should use inflate() to decompress a chunk at a time, and then you don't need to know the uncompressed size ahead of time.
You should also use the deflate() functions for compression. Then you can keep the stream open, and keep compressing until you're done. Then you will have a single zlib stream.
i want to know, is there a possibility to find out where in the response Stream the header ends?
The background of the question is as following, i am using sockets in c to get content from a website, the content is encoded in gzip. I would like to read the content directly from stream and encode the gzip content with zlib. But how do i know the gzip content started and the http header is finished.
I roughly tried two ways which are giving me some, in my opinion, strange results. First, i read in the whole stream, and print it out in terminal, my http header ends with "\r\n\r\n" like i expected, but the secound time, i just retrieve the response once to get the header and then read the content with while loop, here the header ends without "\r\n\r\n".
Why? And which way is the right way to read in the content?
I'll just give you the code so you could see how i'm getting the response from server.
//first way (gives rnrn)
char *output, *output_header, *output_content, **output_result;
size_t size;
FILE *stream;
stream = open_memstream (&output, &size);
char BUF[BUFSIZ];
while(recv(socket_desc, BUF, (BUFSIZ - 1), 0) > 0)
{
fprintf (stream, "%s", BUF);
}
fflush(stream);
fclose(stream);
output_result = str_split(output, "\r\n\r\n");
output_header = output_result[0];
output_content = output_result[1];
printf("Header:\n%s\n", output_header);
printf("Content:\n%s\n", output_content);
.
//second way (doesnt give rnrn)
char *content, *output_header;
size_t size;
FILE *stream;
stream = open_memstream (&content, &size);
char BUF[BUFSIZ];
if((recv(socket_desc, BUF, (BUFSIZ - 1), 0) > 0)
{
output_header = BUF;
}
while(recv(socket_desc, BUF, (BUFSIZ - 1), 0) > 0)
{
fprintf (stream, "%s", BUF); //i would just use this as input stream to zlib
}
fflush(stream);
fclose(stream);
printf("Header:\n%s\n", output_header);
printf("Content:\n%s\n", content);
Both give the same result printing them to terminal, but the secound one should print out some more breaks, at least i expect, because they get lost splitting the string.
I am new to c, so i might just oversee some easy stuff.
You are calling recv() in a loop until the socket disconnects or fails (and writing the received data to your stream the wrong way), storing all of the raw data into your char* buffer. That is not the correct way to read an HTTP response, especially if HTTP keep-alives are used (in which case no disconnect will occur at the end of the response). You must follow the rules outlined in RFC 2616. Namely:
Read until the "\r\n\r\n" sequence is encountered. This terminates the response headers. Do not read any more bytes past that yet.
Analyze the received headers, per the rules in RFC 2616 Section 4.4. They tell you the actual format of the remaining response data.
Read the remaining data, if any, per the format discovered in #2.
Check the received headers for the presence of a Connection: close header if the response is using HTTP 1.1, or the lack of a Connection: keep-alive header if the response is using HTTP 0.9 or 1.0. If detected, close your end of the socket connection because the server is closing its end. Otherwise, keep the connection open and re-use it for subsequent requests (unless you are done using the connection, in which case do close it).
Process the received data as needed.
In short, you need to do something more like this instead (pseudo code):
string headers[];
byte data[];
string statusLine = read a CRLF-delimited line;
int statusCode = extract from status line;
string responseVersion = extract from status line;
do
{
string header = read a CRLF-delimited line;
if (header == "") break;
add header to headers list;
}
while (true);
if ( !((statusCode in [1xx, 204, 304]) || (request was "HEAD")) )
{
if (headers["Transfer-Encoding"] ends with "chunked")
{
do
{
string chunk = read a CRLF delimited line;
int chunkSize = extract from chunk line;
if (chunkSize == 0) break;
read exactly chunkSize number of bytes into data storage;
read and discard until a CRLF has been read;
}
while (true);
do
{
string header = read a CRLF-delimited line;
if (header == "") break;
add header to headers list;
}
while (true);
}
else if (headers["Content-Length"] is present)
{
read exactly Content-Length number of bytes into data storage;
}
else if (headers["Content-Type"] begins with "multipart/")
{
string boundary = extract from Content-Type header;
read into data storage until terminating boundary has been read;
}
else
{
read bytes into data storage until disconnected;
}
}
if (!disconnected)
{
if (responseVersion == "HTTP/1.1")
{
if (headers["Connection"] == "close")
close connection;
}
else
{
if (headers["Connection"] != "keep-alive")
close connection;
}
}
check statusCode for errors;
process data contents, per info in headers list;
using rapidjson, how can i encode a number to json format? i have 123.321, i want to convert it to "123.321", then save it in a file. later using json convert it back to 123.321.
i don't want to use Document which saves as like "{"tag":"value"}", i want it to be saved as just a "value", then be converted back.
i have the following code to convert number to "number":
Value v(123);
StringBuffer mybuffer;
Writer<StringBuffer> mywriter(mybuffer);
v.Accept(mywriter);
const char* myjson = mybuffer.GetString();
how to convert it back? is the following a solution? i don't want to use handler!
Reader reader;
StringStream ss(myjson);
reader.Parse(ss, handler);
thanks for the upcoming helps.
I just found the answer after digging into google:
to encode a number to json using rapidjson:
Value v(123.321);
StringBuffer mybuffer;
Writer<StringBuffer> mywriter(mybuffer);
v.Accept(mywriter);
const char* myjson = mybuffer.GetString();
now myjson has "123.321" as its value.
then to decode myjson to a number:
Document d;
d.Parse(myjson); // myjson is "123.321"
assert(d.IsNumber());
value = d.GetDouble(); // now the value is 123.321
as simple as it gets.
I have been working on wrapping ENet into a set of easy to use functions for a few weeks now and seem to have a bit of an issue.
I have a std::stringstream and am attempting to send the contents to a remote machine using ENet then reconstruct the std::stringstream on the remote machine.
The reason I need to use a std::stringstream is due to the fact that I'm serializing my data with the Cereal Serialization Library which requires a stream.
With Azoth's help he has identified that I need to be using std::istringstream and std::ostringstream. Previously I was only using std::stringstream which was causing an exception.
However now an exception is being thrown within Cereal at portable_binary.hpp line 156:
throw Exception("Failed to read " + std::to_string(size) + " bytes from input stream! Read " + std::to_string(readSize));
Here's what I'm doing:
void Send(ENetHost* Host)
{
std::ostringstream SData;
{
cereal::PortableBinaryOutputArchive Archive(SData);
Archive(PacketData);
}
std::string Out = SData.str();
ENetPacket* Packet = enet_packet_create(Out.c_str(), Out.size(), ENET_PACKET_FLAG_RELIABLE);
enet_host_broadcast(Host, 0, Packet);
}
A Cereal Portable Binary Data Archive is constructed to hold a single vector.
The std::ostringstream is sent off to the host using ENet.
This part seems to work okay, I can print the information out before and after and it appears to be the same, albeit some weird symbols, but they print out the same on both ends.
Now a std::istringstream is created on the host with the data we received.
NetPacket(enet_uint8 const* Data)
{
std::istringstream SData(reinterpret_cast<char const*>(Data));
{
cereal::PortableBinaryInputArchive Archive(SData);
Archive(PacketData);
}
}
At this point I receive the exception at line:
Archive(PacketData)
I have a feeling the data is being changed somehow when it's sent through ENet and/or I'm not pulling the data out of the std::ostringstream correctly and/or not putting the data back into the std::istringstream correctly.
Thank you very much for your time I greatly appreciate it.
Disclaimer: I'm not familiar with enet.
You are getting this error because you aren't constructing the std::stringstream properly upon receiving the packet. A send/receive pair should look something like:
my_send_function()
{
std::ostringstream os;
{
cereal::PortableBinaryOutputArchive ar(os);
ar( whatever_needs_to_be_serialized );
} // the binary archives will flush their output
// immediately, but it's better to
// use cereal archives in an RAII matter all the time
std::string data = os.str();
create_packet(data.c_str(), data.size());
// send out
}
And then on the receiving end, something like this:
my_receive_function( uint8_t const * data ) // data came from some packet
{
MyDataType d;
std::istringstream is(reinterpet_cast<char const *>(data));
// this is safe to do since we generated the data using c_str(), which added
// a null terminator to the data
{
cereal::PortableBinaryInputArchive ar(is);
ar( d );
}
}
The basic idea here: use cereal and some ostringstream to generate a string (which is really just an array of bytes), send those raw bytes over the network, pull them into an istringstream, and then have cereal parse that.
I'm using the Botan library for encryption within Qt. I have it working to where I can encrypt and decrypt from one file to another, but I'm trying to alter it to encrypt from a file to a QDomDocument (encrypted file will just be an XML file), and decrypt back from a QDomDocument to a file.
This is what I have so far for the actual encryption (filePlainText/fileEnc are just txt file paths).
std::ifstream in(filePlainText.c_str(),std::ios::binary);
std::ofstream out(fileEnc.c_str(),std::ios::binary);
Pipe pipe(get_cipher("AES-256/CBC",key,iv,ENCRYPTION),new DataSink_Stream(out));
pipe.start_msg();
in >> pipe;
pipe.end_msg();
out.flush();
out.close();
in.close();
DataSink_Stream accepts a ofsteam or ostream. So I figure I need to use an ostream when decrypting from file to variable. But how can I store the contents of the ostream into something I can feed into a QDomDocument?
Then for encrypting back into a file, use an istream into an ofstream, but how can I take feed the QDomDocument content into an istream?
QDomDocument can be read from and written to a QByteArray and you can read from / write to a std::string with std::ostringstream / std::istringstream.
So if you combine these, you would have something like:
// before the encoding
const QByteArray & buffer = document.toByteArray(-1);
std::istringstream in(std::string(buffer.data(), buffer.size()));
... // encoding
And for the decoding part:
// before the decoding
std::ostringstream out;
... // decoding
// after the decoding
const std::string & buffer = out.str();
document.setContent(QByteArray(buffer.c_str(), buffer.size()));