After executing the code below, I see that the line
encoded = QString(DicResourceByteArray.toBase64().data());
gets too much RAM. Approximately 60MB.
How can I release it from memory?
Dic_block DicBlock;
qint64 Count;
int pos, len;
QByteArray DicResourceByteArray;
QDataStream out(&DicResourceByteArray, QIODevice::WriteOnly);
QString encoded;
while(DicInstance.readBlock(DicBlock))
{
if(DicBlock.type == 2)
{
pos = 0;
len = (unsigned char)DicBlock.data[pos++];
std::string filename( DicBlock.data+pos, len );
pos += len;
out.writeRawData(DicBlock.data + pos, DicBlock.length - pos);
encoded = QString(DicResourceByteArray.toBase64().data());
QString strQuery = QString("INSERT INTO Dictionary_Resources([FileName], [ImageBasedOn64]) values('%1', '%2')").arg(QString::fromStdString(filename), encoded);
query->exec(strQuery);
delete encoded;
}
}
delete query;
db.close();
//...
DicInstance.close();
First thing: grab the data array with .toBase64().constData(), this avoids a possible copy of your data.
Second thing: Move the declaration of QString encoded; into the if-block, this ensures, that after the if-block memory gets released.
Third thing: remove the delete encoded;! (Astonishing that it compiles as encoded is not a pointer).
You don't need delete encoded, the QString is going to be automatically deleted (and released from memory) at the end of your block.
You're doing things very inefficiently by copying the contents of encoded into strQuery. Bind it as a value in your query instead.
Related
I serialize the file via the code beneath, and send it over winsocks, this works fine with textfiles, but when I tried to send a jpg, the string contains \0 as some of the character elements, so the sockets only send part of the string, thinking \0 is the end, i was considering replacing \0 with something else, but say i replace it with 'xx', then replace it back on the other end, what if the file had natural occurrences of 'xx' that get lost? Sure I could make a large, unlikely sequence, but that bloats the file.
Any help appreciated.
char* read_file(string path, int& len)
{
std::ifstream infile(path);
infile.seekg(0, infile.end);
size_t length = infile.tellg();
infile.seekg(0, infile.beg);
len = length;
char* buffer = new char[len]();
infile.read(buffer, length);
return buffer;
}
string load_to_buffer(string file)
{
char* img;
int ln;
img = read_file(file, ln);
string s = "";
for (int i = 1; i <= ln; i++){
char c = *(img + i);
s += c;
}
return s;
}
Probably somewhere in your code (that isn't seen in the code you have posted) you use strlen() or std::string::length() to send the data, and/or you use std::string::c_str() to get the buffer. This results in truncated data because these functions stop at \0.
std::string is not good to handle binary data. Use std::vector<char> instead, and remove the new[] stuff.
I have been using the excellent rapidxml library to read and use information from XML files to hold cutscene information for a game I am programming in C++. I have run into an odd problem,
I start by loading the XML file into a rapidxml::xmldocument<>* from std::ifstream* XMLFile
std::stringstream buffer; //Create a string buffer to hold the loaded information
buffer << XMLFile->rdbuf(); //Pass the ifstream buffer to the string buffer
XMLFile->close(); //close the ifstream
std::string content(buffer.str()); //get the buffer as a string
buffer.clear();
cutScene = new rapidxml::xml_document<>;
cutScene->parse<0>(&content[0]);
root = cutScene->first_node();
my cutscene xml file is made up of "parts" and at the beginning I want to load all of those parts (which are all xml_nodes) into a vector
//Load parts
if (parts->size() == 0) {
rapidxml::xml_node<>* partNode = root->first_node("part");
parts->push_back(partNode);
for (int i = 1; i < numParts; i++) {
parts->push_back(partNode->next_sibling());
printf("name of part added at %i: %s.\n", i, parts->at(i)->name());
}
}
That last line prints "name of part added at 1: part" to the console.
The problem is for some reason, whenever I try to access the vector and print the same name of that same specific part not as a part of this method, the name can be accessed but is just a random string of letters and numbers. It seems that for some reason rapidxml is deleting everything after my load method is complete. I am still new to posting on stackoverflow so if you need more information just ask, thanks!
Rapidxml is in-situ xml parser. It alters the original string buffer (contentin your case) to format null-terminated tokens such as element and attribute names. Secondly, the lifespan of tree nodes referenced byparts items is defined by xml_document (currscene) instance.
Keep currscene and content instances together with the vector, this will keep the vector items alive as well.
e.g:
struct SceneData
{
std::vector<char> content;
rapidxml::xml_document<> cutScene;
std::vector<rapidxml::xml_node<>*> parts;
bool Parse(const std::string& text);
};
bool SendData::Parse(const std::string& text)
{
content.reserve(text.length() + 1);
content.assign(text.begin(), text.end());
content.push_back('\0');
parts.clear();
try
{
cutScene.parse<0>(content.data());
}
catch(rapidxml::parse_error & err)
{
return false;
}
// Load parts.
rapidxml::xml_node<>* root = cutScene.first_node();
rapidxml::xml_node<>* partNode = root->first_node("part");
parts->push_back(partNode);
for (int i = 1; i < numParts; i++) {
parts->push_back(partNode->next_sibling());
//printf("name of part added at %i: %s.\n", i, parts->at(i)->name());
}
return true ;
}
EDITED
The parser expects a sequence of characters terminated by '\0 as input. Since a buffer referenced by &string[0] is not guaranteed to be null-terminated, it is recommended to copy the string content into std::vector<char>.
Starting from a QByteArray, I'd like to search "\n" char inside my QByteArray and join all the characters from the beginning up to "\n" and save them in a QString; after that, I'd pass to the following bytes up to the next "\n" and save these into a new QString
QByteArray MyArray= (all data from my previous process);
quint16 ByteArrayCount = MyArray.count(); // number of bytes composing MyArray
quint16 mycounter;
QString myString;
while (mycounter < ByteArrayCount)
{
if(MyArray[mycounter] != "\n")
myString.append(MyArray[mycounter]);
mycounter++;
}
This to append all bytes preceeding a new line; my problem is how to evaluate MyArray[counter], since I'm not able to check every byte when the counter increases.
Solution?
You could save yourself the trouble and simply:
QString s(myArray);
QStringList resultStrings = s.split('\n');
This will give you a list of strings split for every new line character, which is what you sound like you want to do.
Also, not to belabor the point, but you don't initialize your counter, and you really should ;)
Here is simple example of using function hello
QString str = "ooops\nhello mama\n daddy cool";
QByteArray bta;
bta.append(str);
for(quint16 index = bta.indexOf('\n');
index != -1;
index = bta.indexOf('\n', index+1)) {
/**
* Do something with index
**/
}
But according to your question there is not so clear when you say that you "not able to check every byte". If you know diapasons of available mem, you can use raw data with:
const char * ptr = MyArray.constData();
and use custom validators:
while(ptr){
if(valid(ptr) && ptr == '\n') {
/**
* do something ...
**/
}
ptr++;
}
ow and also in C/C++:
"\n" != 'n'
because "\n" - is const C string(char[2]) containing \n and EOF('\0')
and '\n' - is just simple C char;
In my app I read a string field from a file in local (not Unicode) charset.
The field is a 10 bytes, the remainder is filled with zeros if the string < 10 bytes.
char str ="STRING\0\0\0\0"; // that was read from file
QByteArray fieldArr(str,10); // fieldArr now is STRING\000\000\000\000
fieldArr = fieldArr.trimmed() // from some reason array still containts zeros
QTextCodec *textCodec = QTextCodec::codecForLocale();
QString field = textCodec->ToUnicode(fieldArr).trimmed(); // also not removes zeros
So my question - how can I remove trailing zeros from a string?
P.S. I see zeros in "Local and Expressions" window while debuging
I'm going to assume that str is supposed to be char const * instead of char.
Just don't go over QByteArray -- QTextCodec can handle a C string, and it ends with the first null byte:
QString field = textCodec->toUnicode(str).trimmed();
Addendum: Since the string might not be zero-terminated, adding storage for a null byte to the end seems to be impossible, and making a copy to prepare for making a copy seems wasteful, I suggest calculating the length ourselves and using the toUnicode overload that accepts a char pointer and a length.
std::find is good for this, since it returns the ending iterator of the given range if an element is not found in it. This makes special-case handling unnecessary:
QString field = textCodec->toUnicode(str, std::find(str, str + 10, '\0') - str).trimmed();
Does this work for you?
#include <QDebug>
#include <QByteArray>
int main()
{
char str[] = "STRING\0\0\0\0";
auto ba = QByteArray::fromRawData(str, 10);
qDebug() << ba.trimmed(); // does not work
qDebug() << ba.simplified(); // does not work
auto index = ba.indexOf('\0');
if (index != -1)
ba.truncate(index);
qDebug() << ba;
return 0;
}
Using fromRawData() saves an extra copy. Make sure that the str
stays around until you delete the ba.
indexOf() is safe even if you have filled the whole str since
QByteArray knows you only have 10 bytes you can safely access. It
won't touch 11th or later. No buffer overrun.
Once you removed extra \0, it's trivial to convert to a QString.
You can truncate the string after the first \0:
char * str = "STRING\0\0\0\0"; // Assuming that was read from file
QString field(str); // field == "STRING\0\0\0\0"
field.truncate(field.indexOf(QChar::Null)); // field == "STRING" (without '\0' at the end)
I would do it like this:
char* str = "STRING\0\0\0\0";
QByteArray fieldArr;
for(quint32 i = 0; i < 10; i++)
{
if(str[i] != '\0')
{
fieldArr.append(str[i]);
}
}
QString can be constructed from a char array pointer using fromLocal8Bit. The codec is chosen the same way you do manually in your code.
You need to set the length manually to 10 since you say you have no guarantee that an terminating null byte is present.
Then you can use remove() to get rid of all null bytes. Caution: STRI\0\0\0\0NG will also result in STRING but you said that this does not happen.
char *str = "STRING\0\0\0\0"; // that was read from file
QString field = QString::fromLocal8Bit(str, 10);
field.remove(QChar::Null);
Well, here again, this time the compiler is showing me a memory error (leak):
otest(18015,0xacae32c0) malloc: * error for object 0x194e734:
incorrect checksum for freed object - object was probably modified
after being freed.
* set a breakpoint in malloc_error_break to debug
I'm using clang with ARC activated and STL, this is a C++ file (.cpp).
What I found: If I comment the "delete" line it runs without problems. It makes me wonder who is freeing my allocated memory (cStr).
Btw. This code takes a query string (arg=abc&arg2=asdb) and returns a map whit those values.
static map<string, string> getDecodedQueryString( string qs ) {
map<string, string> r;
qs = qs+"&";
char key[100], value[100], * cStr, *ptr;
cStr = new char[ qs.length() ];
memcpy( cStr, qs.c_str(), url.length()-1);
cStr[qs.length()]=0;
ptr = strtok(cStr, "&");
while ( ptr != NULL && sscanf( ptr, "%[^=]=%[^&]", &key, &value ) == 2) {
r[key]=value;
ptr = strtok(NULL, "&");
}
delete [] cStr; //leaking?
return r;
}
Thanks
The problem is likely in these lines:
cStr = new char[ qs.length() ];
memcpy( cStr, qs.c_str(), url.length()-1);
cStr[qs.length()]=0;
Even without the memcpy() (which may have its own problems due to the length of url, as I mentioned above in a comment), the cStr[qs.length()] = 0 writes one byte past the end of the buffer.
If your compiler has strdup() available (it's nonstandard, but most do), you can replace this with:
cStr = strdup(qs.c_str());
// ...
free(cStr);
This saves you from having to mess around with manually allocating bytes, copying them, null terminating in the right place, etc.
Str = new char[ qs.length() ];
...
cStr[qs.length()]=0;
That write is invalid, it's one past the end of cStr. If the allocator stored a checksum right after the allocation, and checks it at delete time, you've just stomped on it.
Something along the lines of below will do the same thing.
std::stringstream ss(qs);
std::string temp;
std::string key;
std::string value;
while(std::getline(ss, temp, '&')) {
std::stringstream equal(temp);
std::getline(equal, key, '=');
std::getline(equal, value, '=');
r[key] = value;
}