I'm working on Solaris 5.8, C++, using the Json parser.
The problem is: while parsing a file of size greater than 700 MB, the process crashes with core dump error.
It roughly occurs at below code point -
int printbuf_memappend(struct printbuf *p, char *buf, int size)
{
char *t;
if(p->size - p->bpos <= size)
{
int new_size = json_max(p->size * 2, p->bpos + size + 8);
if (!(t = realloc(p->buf, new_size)))
return -1;
p->size = new_size;
p->buf = t;
}
memcpy(p->buf + p->bpos, buf, size); // CORE DUMP HERE
p->bpos += size;
p->buf[p->bpos]= '\0';
return size;
}
Could you please help to identify the problem? The core dump file contain only the data being copied. Can increase of RAM be a solution ? Or do I need to limit the file size to 700MB ?
If crash happened in memcpy, you have two variants
something wrong with input or output.
To test the second variant add memset after realloc:
int new_size = json_max(p->size * 2, p->bpos + size + 8);
if (!(t = realloc(p->buf, new_size)))
return -1;
p->size = new_size;
p->buf = t;
memset(p->buf + p->bpos, 0, size);
On Linux (depend on configuration) possible to allocate not existing virtual memory.
The real allocation happens after the first usage. May the same happens on your Solaris? relloc return ok, but system really have no enought memory? memset should give answer to this question.
Related
I'm using MVS 2010, C++ program without using CRT.
Implementation of finding extension of file:
LPWSTR findExtension(LPCWSTR fileName)
{
int pos = findchr(fileName, L".");
if (pos != -1) {
int lenght = lstrlenW(fileName);
wchar_t* extension = (wchar_t*)HeapAlloc(GetProcessHeap(), NULL, lenght - pos + 1);
for (int i = 0; i < lenght - pos; i++)
{
extension[i] = fileName[pos + 1 + i];
}
extension[lenght - pos] = 0;
LPWSTR ret = extension;
return ret;
}
}
There are problems. Sometimes it can crash application. And memory leak.
How to fix this issues?
Your problem is:
int lenght = lstrlenW(fileName);
lenght is the number of wide characters not the number of bytes. So you allocate a too small buffer here:
wchar_t* extension = (wchar_t*)HeapAlloc(GetProcessHeap(), NULL, lenght - pos + 1);
You need to use this instead:
wchar_t* extension = (wchar_t*)HeapAlloc(GetProcessHeap(), NULL, sizeof(wchar_t) * (lenght - pos + 1));
HeapAlloc() function will allocate memory on heap. When you're not using the heap memory block you need to free it. You can use:
BOOL HeapFree( HANDLE hHeap,DWORD dwFlags,_Frees_ptr_opt_ LPVOID lpMem);
Even though you're in a {....} module, heap will not behave just like the stack that it will automatically FREE its memory. The more you reuse the function the more it will allocate memory to the heap and your private memory size(see task manager) will grow bigger and bigger. And by the time the OS can't give you memory it will suddenly crash.
Not freeing heap memory after using is bad if your program is running a long period (worst case if infinite time) and always perform processes.
I was trying to get a simple demo quickfix program to run on solaris, namely http://www.codeproject.com/Articles/429147/The-FIX-client-and-server-implementation-using-Qui prior to getting it to do what I want it to.
unfortunately in main the application gives a bus error when
FIX::SocketInitiator initiator( application, storeFactory, settings, logFactory);
is called
examine the core dump with gdb and I see
(gdb) where
#0 FIX::SessionFactory::create (this=0xffbfee90, sessionID=#0x101fe8, settings=#0x100e34)
at FieldConvertors.h:113
#1 0xff2594ac in FIX::Initiator::initialize (this=0xffbff108) at stl_tree.h:246
#2 0xff25b270 in Initiator (this=0xffbff108, application=#0xffbff424,
messageStoreFactory=#0xffbff1c4, settings=#0xffbff420, logFactory=#0xffbff338)
at Initiator.cpp:61
#3 0xff25f8a8 in SocketInitiator (this=0xffbff108, application=#0xffbff3c8,
factory=#0xffbff388, settings=#0xffbff408, logFactory=#0xffbff338) at SocketInitiator.cpp:52
#4 0x0004a900 in main (argc=2, argv=0xffbff4c4) at BondsProClient.cpp:42
So I look in FieldConverters.h and we have the code
inline char* integer_to_string( char* buf, const size_t len, signed_int t )
{
const bool isNegative = t < 0;
char* p = buf + len;
*--p = '\0';
unsigned_int number = UNSIGNED_VALUE_OF( t );
while( number > 99 )
{
unsigned_int pos = number % 100;
number /= 100;
p -= 2;
*(short*)(p) = *(short*)(digit_pairs + 2 * pos);
}
if( number > 9 )
{
p -= 2;
*(short*)(p) = *(short*)(digit_pairs + 2 * number); //LINE 113 bus error line
}
else
{
*--p = '0' + char(number);
}
if( isNegative )
*--p = '-';
return p;
}
Looking at this I'm actually not surprised this crashes. It's de-referencing a char* pointer passed to the function as a short, without checking the alignment, which can't be known. This is illegal to any C / C++ standard and since the sparc processor can't perform an unaligned memory access, the thing obviously crashes. Am I being really thick here, or is this a stone cold bug of massive proportions in the quickfix headers? quickfix IS (according to their website) supposed to compile and be usable on solaris sparc. Does anyone know of any work around for this? The option of edit thew header to sprintf springs to mind, as does aligning some things. Or is the a red herring with something different causing an unaligned buffer?
If it's crashing due to misaligned loads/stores then you could replace lines such as:
*(short*)(p) = *(short*)(digit_pairs + 2 * number);
with a safer equivalent using memcpy:
memcpy((void *)p, (const void *)(digit_pairs + 2 * number), sizeof(short));
I have a program that generates files containing random distributions of the character A - Z. I have written a method that reads these files (and counts each character) using fread with different buffer sizes in an attempt to determine the optimal block size for reads. Here is the method:
int get_histogram(FILE * fp, long *hist, int block_size, long *milliseconds, long *filelen)
{
char *buffer = new char[block_size];
bzero(buffer, block_size);
struct timeb t;
ftime(&t);
long start_in_ms = t.time * 1000 + t.millitm;
size_t bytes_read = 0;
while (!feof(fp))
{
bytes_read += fread(buffer, 1, block_size, fp);
if (ferror (fp))
{
return -1;
}
int i;
for (i = 0; i < block_size; i++)
{
int j;
for (j = 0; j < 26; j++)
{
if (buffer[i] == 'A' + j)
{
hist[j]++;
}
}
}
}
ftime(&t);
long end_in_ms = t.time * 1000 + t.millitm;
*milliseconds = end_in_ms - start_in_ms;
*filelen = bytes_read;
return 0;
}
However, when I plot bytes/second vs. block size (buffer size) using block sizes of 2 - 2^20, I get an optimal block size of 4 bytes -- which just can't be correct. Something must be wrong with my code but I can't find it.
Any advice is appreciated.
Regards.
EDIT:
The point of this exercise is to demonstrate the optimal buffer size by recording the read times (plus computation time) for different buffer sizes. The file pointer is opened and closed by the calling code.
There are many bugs in this code:
It uses new[], which is C++.
It doesn't free the allocated memory.
It always loops over block_size bytes of input, not bytes_read as returned by fread().
Also, the actual histogram code is rather inefficient, since it seems to loop over each character to determine which character it is.
UPDATE: Removed claim that using feof() before I/O is wrong, since that wasn't true. Thanks to Eric for pointing this out in a comment.
You're not stating what platform you're running this on, and what compile time parameters you use.
Of course, the fread() involves some overhead, leaving user mode and returning. On the other hand, instead of setting the hist[] information directly, you're looping through the alphabet. This is unnecessary and, without optimization, causes some overhead per byte.
I'd re-test this with hist[j-26]++ or something similar.
Typically, the best timing would be achieved if your buffer size equals the system's buffer size for the given media.
I just ran into a free(): invalid next size (fast) problem while writing a C++ program. And I failed to figure out why this could happen unfortunately. The code is given below.
bool not_corrupt(struct packet *pkt, int size)
{
if (!size) return false;
bool result = true;
char *exp_checksum = (char*)malloc(size * sizeof(char));
char *rec_checksum = (char*)malloc(size * sizeof(char));
char *rec_data = (char*)malloc(size * sizeof(char));
//memcpy(rec_checksum, pkt->data+HEADER_SIZE+SEQ_SIZE+DATA_SIZE, size);
//memcpy(rec_data, pkt->data+HEADER_SIZE+SEQ_SIZE, size);
for (int i = 0; i < size; i++) {
rec_checksum[i] = pkt->data[HEADER_SIZE+SEQ_SIZE+DATA_SIZE+i];
rec_data[i] = pkt->data[HEADER_SIZE+SEQ_SIZE+i];
}
do_checksum(exp_checksum, rec_data, DATA_SIZE);
for (int i = 0; i < size; i++) {
if (exp_checksum[i] != rec_checksum[i]) {
result = false;
break;
}
}
free(exp_checksum);
free(rec_checksum);
free(rec_data);
return result;
}
The macros used are:
#define RDT_PKTSIZE 128
#define SEQ_SIZE 4
#define HEADER_SIZE 1
#define DATA_SIZE ((RDT_PKTSIZE - HEADER_SIZE - SEQ_SIZE) / 2)
The struct used is:
struct packet {
char data[RDT_PKTSIZE];
};
This piece of code doesn't go wrong every time. It would crash with the free(): invalid next size (fast) sometimes in the free(exp_checksum); part.
What's even worse is that sometimes what's in rec_checksum stuff is just not equal to what's in pkt->data[HEADER_SIZE+SEQ_SIZE+DATA_SIZE] stuff, which should be the same according to the watch expressions from my debugging tools. Both memcpy and for methods are used but this problem remains.
I don't quite understand why this would happen. I would be very thankful if anyone could explain this to me.
Edit:
Here's the do_checksum() method, which is very simple:
void do_checksum(char* checksum, char* data, int size)
{
for (int i = 0; i < size; i++)
{
checksum[i] = ~data[i];
}
}
Edit 2:
Thanks for all.
I switched other part of my code from the usage of STL queue to STL vector, the results turn to be cool then.
But still I didn't figure out why. I am sure that I would never pop an empty queue.
The error you report is indicative of heap corruption. These can be hard to track down and tools like valgrind can be extremely helpful. Heap corruptions are often hard to debug with a simple debugger because the runtime error often occurs long after the actual corruption.
That said, the most obvious potential cause of your heap corruption, given the code posted so far, is if DATA_SIZE is greater than size. If that occurs then do_checksum will write beyond the end of exp_checksum.
Three immediate suggestions:
Check for size <= 0 (instead of "!size")
Check for size >= DATA_SIZE
Check for malloc returning NULL
Have you tried Valgrind?
Also, make sure to never send more than RDT_PKTSIZE as size to not_corrupt()
bool not_corrupt(struct packet *pkt, int size)
{
if (!size) return false;
if (size > RDT_PKTSIZE) return false;
/* ... */
Valgrind is good ... but validating all your inputs and checking all error conditions is even better.
Stepping through the code in the debugger isn't a bad idea, either.
I would also call "do_checksum (size)" (your actual size), instead of DATA_SIZE (presumably "maximum size").
DATA_SIZE is a macro defined the max length in my program so the size
should be less than DATA_SIZE
even if that is true, your logic only creates enough memory to hold size characters.
so you should call
do_checksum(exp_checksum, rec_data, size);
and, if you do not want to use std::string (which is fine), you should switch from malloc/free to new/delete when talking C++
Problem solved, thank you all for the help
I've got a bit of a problem here it's not something that's blowing my program up, but it's just bothering me that I can't fix it. I have a function reading in some data from a file, at the end of the execution, the stack around variable longGarbage is corrupted. I've looked around a bit and found that a possible cause is writing to invalid memory. I cleaned up some memory leaks that I had and the problem still persists. What's confusing me is that it happens when the function finishes executing, so it appears to be happening when the variable goes out of scope. Here's the code...
CHCF::CHCF(std::string fileName)
: PAKID("HVST84838672")
{
FILE * archive = fopen(fileName.c_str(), "rb");
std::string strGarbage = "";
unsigned int intGarbage = 0;
unsigned long longGarbage = 0;
unsigned char * data = 0;
char charGarbage = '0';
if (!archive)
{
fclose (archive);
return;
}
for (int i = 0; i < 12; i++)
{
fread(&charGarbage, 1, 1, archive);
strGarbage += charGarbage;
}
if (strGarbage != PAKID)
{
fclose(archive);
throw "Incorrect archive format";
}
strGarbage = "";
fread(&_gameID, sizeof(_gameID),1,archive);
fread(&_fileCount, sizeof(_fileCount),1,archive);
for (int i = 0; i < _fileCount; i++)
{
fread(&longGarbage, 8,1,archive); //file offset
fread(&intGarbage, 4, 1, archive);//fileName
for (int i = 0; i < intGarbage; i++)
{
fread(&charGarbage, 1, 1, archive);
strGarbage += charGarbage;
}
fread(&longGarbage, 8, 1, archive); //fileSize
fread(&intGarbage, 4, 1, archive); //fileType
data = new unsigned char[longGarbage];
for (long i = 0; i < longGarbage; i++)
{
fread(&charGarbage, 1, 1, archive);
data[i] = charGarbage;
}
switch ((FILETYPES)intGarbage)
{
case MAP:
_maps.append(strGarbage, new CFileData(strGarbage, FILETYPES::MAP, data, longGarbage));
break;
default:
break;
}
delete [] data;
data = 0;
strGarbage.clear();
longGarbage = 0;
}
fclose(archive);
} //error happens here
Here is the CFileData constructor:
CFileData::CFileData(std::string fileName, FILETYPES type, unsigned char *data, long fileSize)
{
_fileName = fileName;
_type = type;
_data = new unsigned char[fileSize];
for (int i = 0; i < fileSize; i++)
_data[i] = data[i];
}
Might I suggest std::vector instead of calling new and delete manually? Your code is not exception safe -- you leak if an exception is thrown.
fread(&longGarbage, 8, 1, archive); //fileSize Are you sure sizeof(long) is 8? I suspect it's 4. I believe on Linux boxes sometimes it's 8, but most everywhere else sizeof(long) is 4, and sizeof(long long) is 8.
What about any constructors on members of this class? They can corrupt the stack too.
What's happening is that something is writing to memory around or over the location of longGarbage which is causing the corruption.
You don't say what development environment you are using. One way to diagnose this would be to set a breakpoint that triggers when a specific memory location changes. Choose a memory location that overlaps the area of corruption and wait for it to trigger unexpectedly.
Another way to diagnose this would be to examine code that changes memory around or over longGarbage. That could be almost anything of course but likely candidates are modifications to 'data', modifications to 'intGarbage' and modifications to 'longGarbage' itself.
We can narrow things down even further because we can (usually) be fairly sure the assignment operator itself is safe. Code like data = new... isn't likely to be the culprit so really we need to focus on memory changes that involve taking the address of 'data', 'intGarbage' or 'longGarbage'. In particular memory changes that change more bytes than they should.
Several others have already pointed out that a long is probably not eight bytes in length. If you pass the wrong length to fread, the extra bytes retrieved have to go somewhere.
You are using a lot of magic numbers for data sizes, so I would check that first. In particular, I doubt that sizeof(unsigned long)==8 and sizeof(unsigned in)==4 in all possible circumstances. Refer to your compiler's documentation, but you should still be wary, as this is very likely to change from one compiler/platform to another.
Check for these bits:
fread(&longGarbage, 8,1,archive); //file offset
You also might want to use C++ <iostream> library instead of the C FILE* stuff for reading. It would allow for a much shorter version because you wouldn't need to close the file 3 times.
It seems from the other comments and the information provided that the issue is on the C++ side you should use either __int64 for a windows environment, or int64_t for cross platform.