c++ find and handle each lines is slower than php - c++

Hello i created a program to handle a config file line by checking each lines and get the config blocks but for first time i made it with php and the speed was amazing. we have some blocks like this
Block {
}
php program can read each line and detect about 50,000 of this blocks in just 1 second after that i went to c++ to create my program in c++ but i saw a very very bad problem. my program was too slow (read 50,000 of this blocks in 55 seconds) while my php codes was exactly the same of c++ codes (in action and activity). php was 55x faster than c++ while the codes are the same.
this is my code in php
const PATH = "conf.txt";
if(!file_exists(PATH)) die("path_not_found");
if(!is_readable((PATH))) die("path_not_readable");
$Lines = explode("\r\n", file_get_contents(PATH));
class Block
{
public $Name;
public $Keys = array();
public $Blocks = array();
}
function Handle(& $Lines, $Start, & $Return_block, & $End_on)
{
for ($i = $Start; $i < count($Lines); $i++)
{
while (trim($Lines[$i]) != "")
{
$Pos1 = strpos($Lines[$i], "{");
$Pos2 = strpos($Lines[$i], "}");
if($Pos1 !== false && ($Pos2 === false || $Pos2 > $Pos1)) // Detect { in less position
{
$thisBlock = new Block();
$thisBlock->Name = trim(substr($Lines[$i], 0, $Pos1));
$Lines[$i] = substr($Lines[$i], $Pos1 + 1);
Handle($Lines, $i, $thisBlock, $i);
$Return_block->Blocks[] = $thisBlock;
}
else { // Detect } in less position than {
$Lines[$i] = substr($Lines[$i], $Pos2 + 1);
$End_on = $i;
return;
}
}
}
}
$DefaultBlock = new Block();
Handle($Lines, 0, $DefaultBlock, $NullValue);
$OutsideKeys = $DefaultBlock->Keys;
$Blocks = $DefaultBlock->Blocks;
echo "Found (".count($OutsideKeys).") keys and (".count($Blocks).") blocks.<br><br>";
and this is my code in C++
string Trim(string & s)
{
auto wsfront = std::find_if_not(s.begin(), s.end(), [](int c) {return std::isspace(c); });
auto wsback = std::find_if_not(s.rbegin(), s.rend(), [](int c) {return std::isspace(c); }).base();
return (wsback <= wsfront ? std::string() : std::string(wsfront, wsback));
}
class Block
{
private:
string Name;
vector <Block> Blocks;
public:
void Add(Block & thisBlock) { Blocks.push_back(thisBlock); }
Block(string Getname = string()) { Name = Getname; }
int Count() { return Blocks.size(); }
};
void Handle(vector <string> & Lines, size_t Start, Block & Return, size_t & LastPoint, bool CheckEnd = true)
{
for (size_t i = Start; i < Lines.size(); i++)
{
while (Trim(Lines[i]) != "")
{
size_t Pos1 = Lines[i].find("{");
size_t Pos2 = Lines[i].find("}");
if (Pos1 != string::npos && (Pos2 == string::npos || Pos1 < Pos2)) // Found {
{
string Name = Trim(Lines[i].substr(0, Pos1));
Block newBlock = Block(Name);
Lines[i] = Lines[i].substr(Pos1 + 1);
Handle(Lines, i, newBlock, i);
Return.Add(newBlock);
}
else { // Found }
Lines[i] = Lines[i].substr(Pos2 + 1);
return;
}
}
}
}
int main()
{
string Cont;
___PATH::GetFileContent("D:\\conf.txt", Cont);
vector <string> Lines = ___String::StringSplit(Cont, "\r\n");
Block Return;
size_t Temp;
// The problem (low handle speed) start from here not from including or split
Handle(Lines, 0, Return, Temp);
cout << "Is(" << Return.Count() << ")" << endl;
return 0;
}
as you can see, this codes are exactly the same in action but i don't know why php handling in this code is 55x faster than my c++ codes. you can create a txt file and create about 50,000 of this block's
Block {
}
and test it yourself. please help me to fix this. i am really confused (same codes but not same performance
php = 50,000 blocks and detect in 1 second
c++ = 50,000 blocks and detect in 55 seconds (and maybe more) !
i have no problem in my program design. because i got my performance completely on php but my problem is on c++ that is 55x slower than php in same code action !
i am using (visual studio 2017) to compile this program (c++)

First, "code" is singular, not plural.
C++ is a very different language than php. It is not "the same code", and it is nowhere near the same in action.
For example, these two lines:
Block newBlock = Block(Name);
Return.Add(newBlock);
First create a Block on the stack, and then call Block's copy constructor to make another one inside the vector. You then throw away the stack object.
Also, vectors guarantee that they are contiguous, so as you add new Blocks via your Add method, vector will occasionally stop, allocate another chunk of memory (twice as big as the last one, iirc), copy everything over to that new chunk, and then free the old one. Either preallocate the vector (via vector::reserve()), or consider using something like a deque that doesn't guarantee continuity in memory if you don't need that property.
I also don't know what ___String::StringSplit does, but you are almost certain to have the same vector growth problem in reading your file.

Culprit is in these 2 lines:
Handle(Lines, i, newBlock, i);
Return.Add(newBlock);
Let's say you have 5 levels of 1 block each. What Happens on bottom one? You copy one instance of block. What happens on level 4? You copy 2 blocks (parent and its child). So for level 5 you make 15 copies - 1+2+3+4+5. Look at this diagram:
Handle level1 copies 5 blocks (`Return`->level4->level3->level4->level5)
Handle level2 copies 4 blocks (`Return`->level3->level4->level5)
Handle level3 copies 3 blocks (`Return`->level4->level5
Handle level4 copies 2 blocks (`Return`->level5)
Handle level5 copies 1 block (`Return`)
Formula is:
S = ( N + N^2 ) / 2
so for levels 20 you would do 210 copies and so on.
Suggestion is to use move semantics to avoid this copy:
// change method Add to this
void Add(Block thisBlock) { Blocks.push_back(std::move(thisBlock)); }
// and change this call
Return.Add( std::move( newBlock ) );
Or allocate blocks dynamically using smart pointers

Out of simple curiousity, try this Trim implementation instead:
void _Trim(std::string& result, const std::string& s) {
const auto* ptr = s.data();
const auto* left = ptr;
const auto* end = s.data() + s.size();
while (ptr < end && std::isspace(*ptr)) {
++ptr;
}
if (ptr == end) {
result = "";
return;
}
left = ptr;
while (end > left && std::isspace(*(end-1))) {
--end;
}
result = std::string(left, end);
}
std::string Trim(const std::string& s) {
// Not sure if RVO would fire for direct implementation of _Trim here
std::string result;
_Trim(result, s);
return result;
}
And another optimization:
void Add(Block& thisBlock) {
Blocks.push_back(std::move(thisBlock));
}
// Don't use thisBlock after call to this function. It is
// far from being pretty but it should avoid *lots* of copies.
I wonder if you'll get better result. Pls let me know.

Related

How can I optimize Astar for vast empty spaces?

I am creating a game with a 3D grid for flying entities, So I have a lot of points and connections in the air where there aren't any obstructions. I didn't want to decrease the resolution of my grid so I thought I could just skip over chunks (or empties as I call them) of the Astar map while they're not containing any obstructions, and I modified Godot's Astar algorithm to do this.
Unfortunately this ended up being slower than looping through points one at a time due to the way I implemented this modification, which needs to loop through all the edge points of an empty.
2D representation of how one edge point of an empty connects to all other edge points:
This ends up looping through a larger number of points than letting the A* algorithm work it's way through the empty.
So I'm sorta stumped on how to make this more efficient while still preserving the most optimal path.
I could potentially narrow down what faces of the empty should be scanned over by first comparing the center points of all 8 faces of the empty (as my grid consists of hexagonal prisms). Or maybe I should somehow use the face center points of the empty's faces exclusively instead of all edge points.
I mainly want to know if anyone has worked on an issue like this before, and if so what would be the recommended solution?
Here is the astar loop for reference:
bool AStar::_solve(Point *begin_point, Point *end_point, int relevant_layers) {
pass++;
//make sure parallel layers are supported
// or if *relevant_layers is 0 then use all points
bool supported = relevant_layers == 0 || (relevant_layers & end_point->parallel_support_layers) > 0;
if (!end_point->enabled || !supported) {
return false;
}
bool found_route = false;
Vector<Point *> open_list;
SortArray<Point *, SortPoints> sorter;
begin_point->g_score = 0;
begin_point->f_score = _estimate_cost(begin_point->id, end_point->id);
open_list.push_back(begin_point);
while (!open_list.empty()) {
Point *p = open_list[0]; // The currently processed point
if (p == end_point) {
found_route = true;
break;
}
sorter.pop_heap(0, open_list.size(), open_list.ptrw()); // Remove the current point from the open list
open_list.remove(open_list.size() - 1);
p->closed_pass = pass; // Mark the point as closed
//if the point is part of an empty, look through all of the edge points of said empty (as to skip over any points within the empty).
OAHashMap<int, Point*> connections;
PoolVector<Empty*> enabled_empties;
int size = p->empties.size();
PoolVector<Empty*>::Read r = p->empties.read();
for (int i = 0; i < size; i++) {
Empty* e = r[i];
supported = relevant_layers == 0 || (relevant_layers & e->parallel_support_layers) > 0;
//if the empty is enabled and the end point is not within the empty
if (e->enabled && supported && !end_point->empties.has(e)) {
enabled_empties.append(e);
//can travel to any edge point
for (OAHashMap<int, Point*>::Iterator it = e->edge_points.iter(); it.valid; it = e->edge_points.next_iter(it)) {
int id = *it.key;
Point* ep = *(it.value);
ep->is_neighbour = false;
//don't connect to the same point
if (id != p->id && (i == 0 || !connections.has(id))) {
connections.set(id, ep);
}
}
}
}
//add neighbours to connections
for (OAHashMap<int, Point*>::Iterator it = p->neighbours.iter(); it.valid; it = p->neighbours.next_iter(it)) {
int id = *it.key;
Point* np = *(it.value);// The neighbour point
np->is_neighbour = true;
//don't need to check for duplicate point connections if no empties
if (size == 0 || !connections.has(id)) {
//don't add points within enabled empties since they're meant to be skipped over
if (np->empties.size() > 0 && !np->on_empty_edge) {
bool in_enabled_empty = false;
PoolVector<Empty*>::Read r1 = np->empties.read();
for (int i = 0; i < np->empties.size(); i++) {
if (enabled_empties.has(r1[i])) {
in_enabled_empty = true;
break;
}
}
if (!in_enabled_empty) {
connections.set(id, np);
}
}
else {
connections.set(id, np);
}
}
}
for (OAHashMap<int, Point *>::Iterator it = connections.iter(); it.valid; it = connections.next_iter(it)) {
Point *e = *(it.value); // The neighbour point
//make sure parallel layers are supported
// or if *relevant_layers is 0 then use all points
supported = relevant_layers == 0 || (relevant_layers & e->parallel_support_layers) > 0;
if (!e->enabled || e->closed_pass == pass || !supported) {
continue;
}
real_t tentative_g_score = p->g_score + _compute_cost(p->id, e->id) * e->weight_scale;
bool new_point = false;
if (e->open_pass != pass) { // The point wasn't inside the open list.
e->open_pass = pass;
open_list.push_back(e);
new_point = true;
} else if (tentative_g_score >= e->g_score) { // The new path is worse than the previous.
continue;
}
e->prev_point = p;
e->prev_point_connected = e->is_neighbour;
e->g_score = tentative_g_score;
e->f_score = e->g_score + _estimate_cost(e->id, end_point->id);
if (new_point) { // The position of the new points is already known.
sorter.push_heap(0, open_list.size() - 1, 0, e, open_list.ptrw());
} else {
sorter.push_heap(0, open_list.find(e), 0, e, open_list.ptrw());
}
}
}
return found_route;
}
Note: I'm still not exactly sure what the sorter does.
the entire code can be seen here in a_star.cpp and a_star.h
Edit:
if anyone wants to reference or use this, I've modified the Astar code to add user-defined octants and to use a user-defined straight line function (they are user-defined so they can work with any type of grid) to be used between octants when possible to further decrease runtime, and it works very well in terms of speed. Though the pathing is not optimal, especially when adding a lot of obstacles/restricting the available positions.

Using a thread pool to parallelize a function makes it slower: why?

I am working a on database than runs on top on RocksDB. I have a find function that takes a query in parameter, iterates over all documents in the database, and returns the documents that match the query. I want to parallelize this function so the work is spread on multiple threads.
To achieve that, I tried to use ThreadPool: I moved the code of the loop in a lambda, and added a task to the thread pool for each document. After the loop, each result is processed by the main thread.
Current version (single thread):
void
EmbeDB::find(const bson_t& query,
DocumentPtrCallback callback,
int32_t limit,
const bson_t* projection)
{
int32_t count = 0;
bson_error_t error;
uint32_t num_query_keys = bson_count_keys(&query);
mongoc_matcher_t* matcher = num_query_keys != 0
? mongoc_matcher_new(&query, &error)
: nullptr;
if (num_query_keys != 0 && matcher == nullptr)
{
callback(&error, nullptr);
return;
}
bson_t document;
rocksdb::Iterator* it = _db->NewIterator(rocksdb::ReadOptions());
for (it->SeekToFirst(); it->Valid(); it->Next())
{
const char* bson_data = (const char*)it->value().data();
int bson_length = it->value().size();
std::vector<char> decrypted_data;
if (encryptionEnabled())
{
decrypted_data.resize(bson_length);
bson_length = decrypt_data(bson_data, bson_length, decrypted_data.data(), _encryption_method, _encryption_key, _encryption_iv);
bson_data = decrypted_data.data();
}
bson_init_static(&document, (const uint8_t*)bson_data, bson_length);
if (num_query_keys == 0 || mongoc_matcher_match(matcher, &document))
{
++count;
if (projection != nullptr)
{
bson_error_t error;
bson_t projected;
bson_init(&projected);
mongoc_matcher_projection_execute_noop(
&document,
projection,
&projected,
&error,
NULL
);
callback(nullptr, &projected);
}
else
{
callback(nullptr, &document);
}
if (limit >= 0 && count >= limit)
{
break;
}
}
}
delete it;
if (matcher)
{
mongoc_matcher_destroy(matcher);
}
}
New version (multi-thread):
void
EmbeDB::find(const bson_t& query,
DocumentPtrCallback callback,
int32_t limit,
const bson_t* projection)
{
int32_t count = 0;
bool limit_reached = limit == 0;
bson_error_t error;
uint32_t num_query_keys = bson_count_keys(&query);
mongoc_matcher_t* matcher = num_query_keys != 0
? mongoc_matcher_new(&query, &error)
: nullptr;
if (num_query_keys != 0 && matcher == nullptr)
{
callback(&error, nullptr);
return;
}
auto process_document = [this, projection, num_query_keys, matcher](const char* bson_data, int bson_length) -> bson_t*
{
std::vector<char> decrypted_data;
if (encryptionEnabled())
{
decrypted_data.resize(bson_length);
bson_length = decrypt_data(bson_data, bson_length, decrypted_data.data(), _encryption_method, _encryption_key, _encryption_iv);
bson_data = decrypted_data.data();
}
bson_t* document = new bson_t();
bson_init_static(document, (const uint8_t*)bson_data, bson_length);
if (num_query_keys == 0 || mongoc_matcher_match(matcher, document))
{
if (projection != nullptr)
{
bson_error_t error;
bson_t* projected = new bson_t();
bson_init(projected);
mongoc_matcher_projection_execute_noop(
document,
projection,
projected,
&error,
NULL
);
delete document;
return projected;
}
else
{
return document;
}
}
else
{
delete document;
return nullptr;
}
};
const int WORKER_COUNT = std::max(1u, std::thread::hardware_concurrency());
ThreadPool pool(WORKER_COUNT);
std::vector<std::future<bson_t*>> futures;
bson_t document;
rocksdb::Iterator* db_it = _db->NewIterator(rocksdb::ReadOptions());
for (db_it->SeekToFirst(); db_it->Valid(); db_it->Next())
{
const char* bson_data = (const char*)db_it->value().data();
int bson_length = db_it->value().size();
futures.push_back(pool.enqueue(process_document, bson_data, bson_length));
}
delete db_it;
for (auto it = futures.begin(); it != futures.end(); ++it)
{
bson_t* result = it->get();
if (result)
{
count += 1;
if (limit < 0 || count < limit)
{
callback(nullptr, result);
}
delete result;
}
}
if (matcher)
{
mongoc_matcher_destroy(matcher);
}
}
With simple documents and query, the single-thread version processes 1 million documents in 0.5 second on my machine.
With the same documents and query, the multi-thread version processes 1 million documents in 3.3 seconds.
Surprisingly, the multi-thread version is way slower. Moreover, I measured the execution time and 75% of the time is spent in the for loop. So basically the line futures.push_back(pool.enqueue(process_document, bson_data, bson_length)); takes 75% of the time.
I did the following:
I checked the value of WORKER_COUNT, it is 6 on my machine.
I tried to add futures.reserve(1000000), thinking that maybe the vector re-allocation was at fault, but it didn't change anything.
I tried to remove the dynamic memory allocations (bson_t* document = new bson_t();), it didn't change the result significantly.
So my question is: is there something that I did wrong for the multi-thread version to be that slower than the single-thread version?
My current understanding is that the synchronization operations of the thread pool (when tasks are enqueued and dequeued) are simply consuming the majority of the time, and the solution would be to change the data-structure. Thoughts?
Parallelization has overhead.
It takes around 500 nanoseconds to process each document in the single-threaded version. There's a lot of bookkeeping that has to be done to delegate work to a thread-pool (both to delegate the work, and to synchronize it afterwards), and all that bookkeeping could very well require more than 500 nanoseconds per job.
Assuming your code is correct, then the bookkeeping takes around 2800 nanoseconds per job. To get a significant speedup from parallelization, you're going to want to break the work into bigger chunks.
I recommend trying to process documents in batches of 1000 at a time. Each future, instead of corresponding to just 1 document, will correspond to 1000 documents.
Other optimizations
If possible, avoid unnecessary copying. If something gets copied a bunch, see if you can capture it by reference instead of by value.

Copy only necessary objects from PDF file

I've got a huge PDF file with more than 100 pages and I want to separate them to single PDF files (containing only one page each). Problem is, that PoDoFo does not copy just the page, but the whole document because of the references (and so each of the 100 PDF files have same size as the 100-page PDF). A relevant mailing list post can be found, unfortunately there is no solution provided.
In source code of function InsertPages there is explanation:
This function works a bit different than one might expect.
Rather than copying one page at a time - we copy the ENTIRE document
and then delete the pages we aren't interested in.
We do this because
1) SIGNIFICANTLY simplifies the process
2) Guarantees that shared objects aren't copied multiple times
3) offers MUCH faster performance for the common cases
HOWEVER: because PoDoFo doesn't currently do any sort of "object
garbage collection" during a Write() - we will end up with larger
documents, since the data from unused pages will also be in there.
I have tried few methods to copy only relevant objects, but each of them failed.
Copy all pages and remove irrelevant ones
Use XObject wrapping: FillXObjectFromDocumentPage and FillXObjectFromExistingPage
Copy object by object
Use RenumberObjects with bDoGarbageCollection = true
but none of them worked out. Does anybody have an idea or working solution for this problem?
The only solution is to use another PDF library. Or wait for garbage collection to be implemented.
The problem is stated in the quote you mentioned:
> during a Write() - we will end up with larger documents, since the
> data from unused pages will also be in there.
This means podofo always puts the entire PDF content in your file, no matter what. The whole PDF is there, you just don't see parts of it.
Dennis from the podofo support sent me an working example of optimized version of InsertPages function which is actually fixing page references and decreases document size significantly!
void PdfMemDocument::InsertPages2(const PdfMemDocument & rDoc, std::vector<int> pageNumbers)
{
std::unordered_set<PdfObject*> totalSet;
std::vector<pdf_objnum> oldObjNumPages;
std::unordered_map<pdf_objnum, pdf_objnum> oldObjNumToNewObjNum;
std::vector<PdfObject*> newPageObjects;
// Collect all dependencies from all pages that are to be copied
for (int i = 0; i < pageNumbers.size(); ++i) {
PdfPage* page = rDoc.GetPage(pageNumbers[i]);
if (page) {
oldObjNumPages.push_back(page->GetObject()->Reference().ObjectNumber());
std::unordered_set<PdfObject*> *set = page->GetPageDependencies();
totalSet.insert(set->begin(), set->end());
delete set;
}
}
// Create a new page object for every copied page from the old document
// Copy all objects the pages depend on to the new document
for (auto it = totalSet.begin(); it != totalSet.end(); ++it) {
unsigned int length = static_cast<unsigned int>(GetObjects().GetSize() + GetObjects().GetFreeObjects().size());
PdfReference ref(static_cast<unsigned int>(length+1), 0);
PdfObject* pObj = new PdfObject(ref, *(*it));
pObj->SetOwner(&(GetObjects()));
if ((*it)->HasStream()) {
PdfStream *stream = (*it)->GetStream();
pdf_long length;
char* buf;
stream->GetCopy(&buf, &length);
PdfMemoryInputStream inputStream(buf, length);
pObj->GetStream()->SetRawData(&inputStream, length);
free(buf);
}
oldObjNumToNewObjNum.insert(std::pair<pdf_objnum, pdf_objnum>((*it)->Reference().ObjectNumber(), length+1));
GetObjects().push_back(pObj);
newPageObjects.push_back(pObj);
}
// In all copied objects, fix the object numbers so they are valid in the new document
for (auto it = newPageObjects.begin(); it != newPageObjects.end(); ++it) {
FixPageReferences(GetObjects(), *it, oldObjNumToNewObjNum);
}
// Insert the copied pages into the pages tree
for (auto it = oldObjNumPages.begin(); it != oldObjNumPages.end(); ++it) {
PdfObject* pageObject = GetObjects().GetObject(PdfReference(oldObjNumToNewObjNum[(*it)], 0));
PdfPage *page = new PdfPage(pageObject, std::deque<PdfObject*>());
GetPagesTree()->InsertPage(GetPageCount() - 1, page);
}
}
std::unordered_set<PdfObject *>* PdfPage::GetPageDependencies() const
{
std::unordered_set<PdfObject *> *set = new std::unordered_set<PdfObject *>();
const PdfObject* pageObj = GetObject();
if (pageObj) {
PdfVecObjects* objects = pageObj->GetOwner();
if (objects) {
set->insert((PdfObject*)pageObj);
objects->GetObjectDependencies2(pageObj, *set);
}
}
return set;
}
// Optimized version of PdfVecObjects::GetObjectDependencies
void PdfVecObjects::GetObjectDependencies2(const PdfObject* pObj, std::unordered_set<PdfObject*> &refMap) const
{
// Check objects referenced from this object
if (pObj->IsReference())
{
PdfObject* referencedObject = GetObject(pObj->GetReference());
if (referencedObject != NULL && refMap.count(referencedObject) < 1) {
(refMap).insert((PdfObject *)referencedObject); // Insert referenced object
GetObjectDependencies2((const PdfObject*)referencedObject, refMap);
}
}
else {
// Recursion
if (pObj->IsArray())
{
PdfArray::const_iterator itArray = pObj->GetArray().begin();
while (itArray != pObj->GetArray().end())
{
GetObjectDependencies2(&(*itArray), refMap);
++itArray;
}
}
else if (pObj->IsDictionary())
{
TCIKeyMap itKeys = pObj->GetDictionary().GetKeys().begin();
while (itKeys != pObj->GetDictionary().GetKeys().end())
{
if ((*itKeys).first != PdfName("Parent")) {
GetObjectDependencies2((*itKeys).second, refMap);
}
++itKeys;
}
}
}
}
void FixPageReferences(PdfVecObjects& objects, PdfObject* pObject, std::unordered_map<pdf_objnum, pdf_objnum>& oldNumToNewNum) {
if( !pObject)
{
PODOFO_RAISE_ERROR( ePdfError_InvalidHandle );
}
if( pObject->IsDictionary() )
{
TKeyMap::iterator it = pObject->GetDictionary().GetKeys().begin();
while( it != pObject->GetDictionary().GetKeys().end() )
{
if ((*it).first != PdfName("Parent")) {
FixPageReferences(objects, (*it).second, oldNumToNewNum);
}
++it;
}
}
else if( pObject->IsArray() )
{
PdfArray::iterator it = pObject->GetArray().begin();
while( it != pObject->GetArray().end() )
{
FixPageReferences(objects, &(*it), oldNumToNewNum),
++it;
}
}
else if( pObject->IsReference() )
{
//PdfObject* referencedObj = objects.GetObject(pObject->GetReference());
pdf_objnum oldnum = pObject->GetReference().ObjectNumber();
pdf_objnum newnum = oldNumToNewNum[oldnum];
if (!newnum) throw new std::exception("No new object number for old object number");
*pObject = PdfReference(newnum, 0);
}
}

std::list copy to std::vector skipping elements

I've run across a rather bizarre exception while running C++ code in my objective-C application. I'm using libxml2 to read an XSD file. I then store the relevant tags as instances of the Tag class in an std::list. I then copy this list into an std::vector using an iterator on the list. However, every now and then some elements of the list aren't copied to the vector. Any help would be greatly appreciated.
printf("\n length list = %lu, length vector = %lu\n",XSDFile::tagsList.size(), XSDFile::tags.size() );
std::list<Tag>::iterator it = XSDFile::tagsList.begin();
//result: length list = 94, length vector = 0
/*
for(;it!=XSDFile::tagsList.end();++it)
{
XSDFile::tags.push_back(*it); //BAD_ACCESS code 1 . . very bizarre . . . . 25
}
*/
std::copy (XSDFile::tagsList.begin(), XSDFile::tagsList.end(), std::back_inserter (XSDFile::tags));
printf("\n Num tags in vector = %lu\n", XSDFile::tags.size());
if (XSDFile::tagsList.size() != XSDFile::tags.size())
{
printf("\n length list = %lu, length vector = %lu\n",XSDFile::tagsList.size(), XSDFile::tags.size() );
//result: length list = 94, length vector = 83
}
I've found the problem. The memory was corrupted causing the std::list to become corrupted during the parsing of the XSD. I parse the XSD using a function start_element.
xmlSAXHandler handler = {0};
handler.startElement = start_element;
I used malloc guard in xcode to locate the use of freed memory. It pointed to the line:
std::strcpy(message, (char*)name);
So I removed the malloc (actually commented in the code) and it worked. The std::vector now consistently copies all 94 entries of the list. If anyone has an explanation as to why this worked that would be great.
static void start_element(void * ctx, const xmlChar *name, const xmlChar **atts)
{
// int len = strlen((char*)name);
// char *message = (char*)malloc(len*sizeof(char));
// std::strcpy(message, (char*)name);
if (atts != NULL)
{
// atts[0] = type
// atts[1] = value
// len = strlen((char*)atts[1]);
// char *firstAttr = (char*)malloc(len*sizeof(char));
// std::strcpy(firstAttr, (char*)atts[1]);
if(strcmp((char*)name, "xs:include")==0)
{
XSDFile xsd;
xsd.ReadXSDTypes((char*)atts[1]);
}
else if(strcmp((char*)name, "xs:element")==0)
{
doElement(atts);
}
else if(strcmp((char*)name, "xs:sequence")==0)
{
//set the default values
XSDFile::sequenceMin = XSDFile::sequenceMax = 1;
if (sizeof(atts) == 4)
{
if(strcmp((char*)atts[3],"unbounded")==0)
XSDFile::sequenceMax = -1;
int i = 0;
while(atts[i] != NULL)
{
//atts[i] = name
//atts[i+i] = value
std::string name((char*)atts[i]);
std::string value((char*)atts[i+1]);
if(name=="minOccurs")
XSDFile::sequenceMin = (atoi(value.c_str()));
else if(name=="maxOccurs")
XSDFile::sequenceMax = (atoi(value.c_str()));
i += 2;
}
}
}
}
//free(message);
}

Boost Test Case Giving Mixed Results

I'm currently practising the Test-Driven Development style of programming as well as trying to memorize some useful idioms and exception safety rules I've learned. I've used an old programming assignment from semester one to simply go nuts, use stuff where it shouldn't be used to get a feel for them, learn their respective pros and cons. You know, practising.
I've been coding with the Boost_Unit_Test framework, and it's been going good so far(only testing framework I've tried). Except here in this test:
BOOST_AUTO_TEST_CASE(CopyConstructor)
{
Field *field = Field::EmptyField();
PushValsToField(&field, 5);
Field *field2 = new Field(*field);
BOOST_REQUIRE_EQUAL(field->Size(), field2->Size());
BOOST_REQUIRE_EQUAL((*field)[0], (*field2)[0]);
BOOST_REQUIRE_EQUAL((*field)[1], (*field2)[1]);
BOOST_REQUIRE_EQUAL((*field)[2], (*field2)[2]);
BOOST_REQUIRE_EQUAL((*field)[3], (*field2)[3]);
BOOST_REQUIRE_EQUAL((*field)[4], (*field2)[4]);
// Error with BOOST_EQUAL_REQUIRE_COLLECTIONS
BOOST_REQUIRE_EQUAL_COLLECTIONS(&(*field)[0], &(*field)[5],
&(*field2)[0], &(*field)[5]);
delete field;
delete field2;
}
I'm not sure why, but the collections test fails on the last compare ([5]). However, the other tests pass. Why is the last test failing?
Also, any style guides or pointers would be greatly appreciated, but not in the scope of the question.
Error:
fatal error in "CopyConstructor":
critical check { &(*field)[0], &(*field)[5] } == { &(*field2)[0], &(*field)[5] } failed.
Collections size mismatch: 5 != 1073731817
Useful Information and Code Snippets
void PushValsToField(Field **field, int numPushes)
{
for (int i(1); i <= numPushes; ++i)
(*field)->Push_Back(i*10);
}
Constructors
Field *Field::EmptyField()
{
return new Field();
}
Field::Field()
: v_(new ElemType[10000]), vused_(0), vsize_(10000)
{}
Field::Field(const Field& other)
: v_(h::NewCopy(other.v_,
other.vsize_,
other.vsize_)),
vused_(other.vused_),
vsize_(other.vsize_)
{}
// Also available as a const& version
int& Field::operator[](int index) throw(const char *)
{
if (index < 0 || index > vused_)
throw "Index out of bounds.";
else
return v_[index];
}
Copy function
template <class T>
T *NewCopy( const T* src,
size_t srcSize,
size_t destSize)
{
assert( destSize >= srcSize );
T *dest = new T[destSize];
try
{
std::copy(src, (src + srcSize), dest);
}
catch(...)
{
delete[] dest;
throw;
}
return dest;
}
If your size is 5, aren't your valid indices 0-4? 5 would be out of range. I think you have an error in your operator[] bounds check:
if (index < 0 || index > vused_)
... should be ...
if (index < 0 || index >= vused_)
BOOST_REQUIRE_EQUAL((*field)[0], (*field2)[0]); // one
BOOST_REQUIRE_EQUAL((*field)[1], (*field2)[1]); // two
BOOST_REQUIRE_EQUAL((*field)[2], (*field2)[2]); // three
BOOST_REQUIRE_EQUAL((*field)[3], (*field2)[3]); // four
BOOST_REQUIRE_EQUAL((*field)[4], (*field2)[4]); // five
BOOST_REQUIRE_EQUAL((*field)[5], (*field2)[5]); // wait, six?
Count the number of elements you are checking, zero through five... It gives you six. But your collection is supposed to have five elements.