I need to read a binary file containing several bytes and divide the contents into frames, each consisting of 535 bytes each. The number of frames present in the file is not known at runtime and thus I need to dynamically allocate memory for them. The code below is a snippet and as you can see, I'm trying to create a pointer to an array of bytes (uint8_t) and then increment into the next frame and so on, in the loop that reads the buffered data into the frames. How do I allocate memory at runtime and is this the best way to do the task? Please let me know if there is a more elegant solution. Also, how I manage the memory?
#include <cstdio>
using namespace std;
long getFileSize(FILE *file)
{
long currentPosition, endPosition;
currentPosition = ftell(file);
fseek(file, 0, 2);
endPosition = ftell(file);
fseek(file, currentPosition, 0);
return endPosition;
}
int main()
{
const char *filePath = "C:\Payload\Untitled.bin";
uint8_t *fileBuffer;
FILE *file = NULL;
if((file = fopen(filePath, "rb")) == NULL)
cout << "Failure. Either the file does not exist or this application lacks sufficient permissions to access it." << endl;
else
cout << "Success. File has been loaded." << endl;
long fileSize = getFileSize(file);
fileBuffer = new uint8_t[fileSize];
fread(fileBuffer, fileSize, 1, file);
uint8_t (*frameBuffer)[535];
for(int i = 0, j = 0; i < fileSize; i++)
{
frameBuffer[j][i] = fileBuffer[i];
if((i % 534) == 0)
{
j++;
}
}
struct frame {
unsigned char bytes[535];
};
std::vector<frame> frames;
Now your loop can simply read a frame and push it into frames. No explicit memory management needed: std::vector does that for you.
Related
void demodlg::printData(short* data)
{
FILE* pF;
char buf[50];
snprintf(buf, sizeof(buf), "%s\\%s\\%s%d.binary", "test", "data", "data", frameNum++);
pF = fopen(buf, "wb");
int lines = frameDescr->m_numLines;
int samples = frameDescr->m_pLineTypeDescr[0].m_numSamples;
int l, s;
fprintf(pF, "\t");
for (l = 0; l < lines; l++)
{
fprintf(pF, "%d\t", l);
}
fprintf(pF, "\n");
for (s = 0; s < samples; s++)
{
fprintf(pF, "%d)\t", s);
for (l = 0; l < lines; l++)
{
fprintf(pF, "%d\t", *(data + l * samples + s));
}
fprintf(pF, "\n");
}
fclose(pF);
}
I have the code snippet above which just takes in some data and then writes it out to a binary file. This function gets called about 20-30 times per second, so I'm trying to optimize it as much as possible. Each file that it writes to is about 1 MB in size. Ideally, I'd be able to write 20-30 MB per second. As of now, it's not at that rate.
Does anyone have any ideas on how I can optimize this further?
I originally was writing to a txt file before changing to a binary file, but the different isn't too noticeable, surprisingly.
Also, frameDescr gets updated for every frame so I believe I do need to get access to the lines and samples variables from inside, unfortunately.
I found this post to refer to (Writing a binary file in C++ very fast) but I'm not sure how I can apply it to mine.
Here is a short example of how I would write an array of data to a binary file and how I would read it back.
I do not understand the concept or purpose of lines in your code so I did not attempt to replicate it. If you do have additional data you need to write to allow it to be reconstructed when read I have placed comments to note where you could insert that code.
Keep in mind that the data when written as binary must be read the same way, so if you were writing the text in a particular format to consume it from another program then a binary file will not work for you unless you modify that other program or create an additional step to read the binary data and write the text format before consumption.
Assuming there is a speed advantage to writing the data as binary then adding an additional step to convert the binary data to text format is beneficial because you can do it offline when you're not trying to maintain a particular frame rate.
Normally since you tagged this c++ I would prefer manipulating the data in a vector and perhaps using c++ streams to write and read the data, but I tried to keep this as similar to your code as possible.
#include <cstdio>
#include <stdint.h>
const size_t kNumEntries = 128 * 1024;
void writeData(const char *filename, int16_t *data, size_t numEntries)
{
FILE *f = fopen(filename, "wb");
if (!f)
{
fprintf(stderr, "Error opening file: '%s'\n", filename);
return;
}
//If you have additional data that must be in the file write it here
//either as individual items that are mirrored in the reader,
//or using the pattern showm below for variable sized data.
//Write the number of entries we have to write to the file so the reader
//will know how much memory to allocate how many to read.
fwrite(&numEntries, sizeof(numEntries), 1, f);
//Write the actual data
fwrite(data, sizeof(*data), numEntries, f);
fclose(f);
}
int16_t* readData(const char *filename)
{
FILE *f = fopen(filename, "rb");
if (!f)
{
fprintf(stderr, "Error opening file: '%s'\n", filename);
return 0;
}
//If you have additional data to read, do it here.
//This code whould mirror the writing function.
//Read the number of entries in the file.
size_t numEntries;
fread(&numEntries, sizeof(numEntries), 1, f);
//Allocate memory for the entreis and read them into it.
int16_t *data = new int16_t[sizeof(int16_t) * numEntries];
fread(data, sizeof(*data), numEntries, f);
fclose(f);
return data;
}
int main()
{
int16_t *dataToWrite = new int16_t[sizeof(int16_t) * kNumEntries];
int16_t *dataRead = new int16_t[sizeof(int16_t) * kNumEntries];
for (int i = 0; i < kNumEntries; ++i)
{
dataToWrite[i] = i;
dataRead[i] = 0;
}
writeData("test.bin", dataToWrite, kNumEntries);
dataRead = readData("test.bin");
for (int i = 0; i < kNumEntries; ++i)
{
if (dataToWrite[i] != dataRead[i])
{
fprintf(stderr,
"Data mismatch at entry %d, : dataToWrite = %d, dataRead = %d\n",
i, dataToWrite[i], dataRead[i]);
}
}
delete[] dataRead;
return 0;
}
I have a server that sends raw binary data to print a "map" that a user must traverse through, however, I am having trouble clearing out my buffer after each line read and thus keep getting residual data printed at the end of the shorter lines. In the screenshot below you can see my output on the left, and what the output should be on the right. What is the best way to solve this? I feel like I am missing something but cant seem to find a solution.
And the code that is reading/printing this is below:
char* mapData = NULL;
string command = "command> ";
size_t dataSize = 0;
while(mapData != command.c_str()) {
unsigned char* buffer = (unsigned char*) &dataSize;
connection = read(mySocket, buffer, 8);
if(connection == -1 || connection < 0) {
cerr << "**Error: could not read text size" << endl;
return 1;
}
mapData = (char*)malloc(dataSize);
buffer = (unsigned char*) mapData;
while((connection = read(mySocket, buffer, dataSize)) != -1) {
if(connection == -1 || connection < 0) {
cerr << "**Error: could not read text size" << endl;
return 1;
}
if(dataSize != 1) {
cout << buffer;
}
free(buffer);
buffer = NULL;
}
}
You are ignoring the return value of read() to know how many bytes are in the buffer.
read() returns the actual number of bytes that were read, which may be fewer than you requested. So you need to call read() in a loop until you have read all of the bytes you are expecting, eg:
int readAll(int sock, void *buffer, size_t buflen)
{
unsigned char* pbuf = reinterpret_cast<unsigned char*>(buffer);
while (buflen > 0) {
int numRead = read(sock, pbuf, buflen);
if (numRead < 0) return -1;
if (numRead == 0) return 0;
pbuf += numRead;
buflen -= numRead;
}
return 1;
}
Also, after reading the buffer, you are treating it as if it were null-terminated, but it is not, which is why you get extra garbage in your output.
More importantly, mapData != command.c_str() will ALWAYS be true, so your while loop iterates indefinitely (until a socket error occurs), which is not what you want. You want the loop to end when you receive a "command> " string instead.
mapData is initially NULL, and c_str() NEVER returns NULL, so the loop ALWAYS iterates at least once.
Then you allocate and free mapData but don't reset it to NULL, so it is left pointing at invalid memory. Which doesn't really matter, since your while loop is just comparing pointers. c_str() will NEVER return a pointer to memory that mapData ever points to.
To end your loop correctly, you need to compare the contents of mapData after reading, not compare its memory address.
Try this instead:
char *mapData = NULL;
uint64_t dataSize = 0;
const string command = "command> ";
bool keepLooping = true;
do {
if (readAll(mySocket, &dataSize, sizeof(dataSize)) <= 0) {
cerr << "**Error: could not read text size" << endl;
return 1;
}
if (dataSize == 0)
continue;
mapData = new char[dataSize];
if (readAll(mySocket, mapData, dataSize) <= 0) {
cerr << "**Error: could not read text" << endl;
delete[] mapData;
return 1;
}
cout.write(mapData, dataSize);
keepLooping = (dataSize != command.size()) || (strncmp(mapData, command.c_str(), command.size()) != 0);
delete[] mapData;
}
while (keepLooping);
Alternatively:
string mapData;
uint64_t dataSize = 0;
const string command = "command> ";
do {
if (readAll(mySocket, &dataSize, sizeof(dataSize)) <= 0) {
cerr << "**Error: could not read text size" << endl;
return 1;
}
mapData.resize(dataSize);
if (dataSize > 0) {
if (readAll(mySocket, &mapData[0], dataSize) <= 0) {
cerr << "**Error: could not read text" << endl;
return 1;
}
cout << mapData;
}
}
while (mapData != command);
like #eozd pointed out, calling malloc and free in your loop is a bad idea since you use return statements. Your code may leak memory. You should ensure you call free before returns. Even better, you could declare your buffer outside of while loop, and use break instead of return, and call free if there was en error
Looking at your solution, it seems that the communication protocol involves sending data size first, followed by the actual data. How is data size written to the wire? You may need to convert it from network byte order.
To debug, you could print out the value of dataSize before every read to make sure that it is what you expect
You should clear the buffer too. Add:
memset(mapData, 0, dataSize);
after the malloc.
I am trying to read a binary file's data sadly opening in C++ is a lot different than in python for these things as they have byte mode. It seems C++ does not have that.
for (auto p = directory_iterator(path); p != directory_iterator(); p++) {
if (!is_directory(p->path()))
byte tmpdata;
std::ifstream tmpreader;
tmpreader.open(desfile, std::ios_base::binary);
int currentByte = tmpreader.get();
while (currentByte >= 0)
{
//std::cout << "Does this get Called?" << std::endl;
int currentByte = tmpreader.get();
tmpdata = currentByte;
}
tmpreader.close()
}
else
{
continue;
}
I want basically a clone of Python's methods of opening a file in 'rb' mode. To have to actual byte data of all of the contents (which is not readable as it has nonprintable chars even for C++. Most of which probably cant be converted to signed chars just because it contains zlib compressed data that I need to feed in my DLL to decompress it all.
I do know that in Python I can do something like this:
file_object = open('[file here]', 'rb')
turns out that replacing the C++ Code above with this helps. However fopen is depreciated but I dont care.
What the Code above did not do was work because I was not reading from the buffer data. I did realize later that fopen, fseek, fread, and fclose was the functions I needed for read bytes mode ('rb').
for (auto p = directory_iterator(path); p != directory_iterator(); p++) {
if (!is_directory(p->path()))
{
std::string desfile = p->path().filename().string();
byte tmpdata;
unsigned char* data2;
FILE *fp = fopen("data.d", "rb");
fseek(fp, 0, SEEK_END); // GO TO END OF FILE
size_t size = ftell(fp);
fseek(fp, 0, SEEK_SET); // GO BACK TO START
data2 = new unsigned char[size];
tmpdata = fread(data2, 1, size, fp);
fclose(fp);
}
else
{
continue;
}
int currentByte = tmpreader.get();
while (currentByte >= 0)
{
//std::cout << "Does this get Called?" << std::endl;
int currentByte = tmpreader.get();
//^ here!
You are declaring a second variable hiding the outer one. However, this inner one is only valid within the while loop's body, so the while condition checks the outer variable which is not modified any more. Rather do it this way:
int currentByte;
while ((currentByte = tmpreader.get()) >= 0)
{
I am trying to write a large binary file into a buffer in a C++ program. GDB always gets a segfault after trying to create a buffer the same size as the file read. It either fails on fclose(pf), rewind, or f(open) which leads me to believe that there is something wrong when I am trying to create the buffer. My code segment is as follows.
static int fileTransfer(struct mg_connection *conn, char * filename){
FILE *fp = fopen(filename, "r");
fseek(fp, 0, SEEK_END);
int size = ftell(fp);
char buf[size];
fclose(fp);
// This is an attempt to stop a segment fault from rewind.
fp = fopen(filename, "r");
conn->connection_param = (void *) fp;
size_t n = 0;
if(fp != NULL)
{
n = fread(buf, 1, sizeof(buf), fp);
mg_send_data(conn, buf, n);
if(n < sizeof(buf) || conn->wsbits != 0)
{
fclose(fp);
conn->connection_param = NULL;
}
}
return 1;
}
I have tried putting print statements in this code but they don't print to the console as they are running in a separate thread. Can someone give me some insight on why this segfault is happening, or some suggestions on how to make this code more efficient.
I should note that this code works properly on 1 and 10 MB files but not on anything larger.
never do this:
int size = ftell(fp);
char buf[size];
You are creating size on the STACK, not on the heap.... 100MB on the stack will not work.
AND... size must be a constant number, not a number coming from ftell(). I even don't know how it is compiling...
What you have to to is to allocate memory using malloc() or new operator.
static int fileTransfer(struct mg_connection *conn, char * filename){
FILE *fp = fopen(filename, "r");
fseek(fp, 0, SEEK_END);
int size = ftell(fp);
char * buf = new char[size]; // fix also here!
fclose(fp);
// This is an attempt to stop a segment fault from rewind.
fp = fopen(filename, "r");
conn->connection_param = (void *) fp;
size_t n = 0;
if(fp != NULL)
{
n = fread(buf, 1, size, fp); // fix also here!
mg_send_data(conn, buf, n);
if(n < size || conn->wsbits != 0)
{
fclose(fp);
conn->connection_param = NULL;
}
}
delete [] buf; // and you have to deallocate your buffer
return 1;
}
You are creating a buffer with automatic storage duration, which means it will be put on the stack by g++. The default stack size for any OS known to me is below 100 MB, meaning it will cause a segfault on system supporting them.
Try allocating your buffer with dynamic storage duration, which will place it on the heap.
What's going on is actually the namesake of this site! Basically, what is happening is your program is created it has a set amount of memory allocated for the stack.
When you create char buf[size], you are using a C99 feature called a variable length array (VLA). This allocates space on the stack for buf. However, buf is too large for the stack, so your program fails.
In order to fix this problem, you should use char * buf; and then do buf = malloc(size). This will place buf on the heap, which is larger than the stack. It also lets you check if you do not have enough memory, by checking if malloc() returns NULL. You need to be sure to free(buf) before you exit though!
As a side note, you can check how much space you have on the stack by using the ulimit -s command.
That seems like a lot to allocate on the stack. What if you put it on the heap instead?
char *buf = new char[size];
Use std::vector. Then you don't have the issues of stack space, or the other issue of writing non-standard C++ code:
#include <vector>
//...
static int fileTransfer(struct mg_connection *conn, char * filename)
{
FILE *fp = fopen(filename, "r");
fseek(fp, 0, SEEK_END);
int size = ftell(fp);
std::vector<char> buf(size);
fclose(fp);
fp = fopen(filename, "r");
conn->connection_param = (void *) fp;
size_t n = 0;
if(fp != NULL)
{
n = fread(&buf[0], 1, buf.size(), fp);
mg_send_data(conn, &buf[0], n);
if(n < buf.size() || conn->wsbits != 0)
{
fclose(fp);
conn->connection_param = NULL;
}
}
return 1;
}
I am trying to learn more about binary files, so I started with HexEdit, and I manually wrote a file and created a template for it. Here is my work:
Now, I started working on a console application in C++ Win32 to read the contents in that file and make them look friendly. Here is part my code:
typedef unsigned char BYTE;
long getFileSize(FILE *file)
{
long lCurPos, lEndPos;
lCurPos = ftell(file);
fseek(file, 0, 2);
lEndPos = ftell(file);
fseek(file, lCurPos, 0);
return lEndPos;
}
int main()
{
const char *filePath = "D:\\Applications\\ColorTableApplication\\file.clt";
BYTE *fileBuf; // Pointer to our buffered data
FILE *file = NULL; // File pointer
if ((file = fopen(filePath, "rb")) == NULL)
printf_s("Could not open specified file\n");
else {
printf_s("File opened successfully\n");
printf_s("Path: %s\n", filePath);
printf_s("Size: %d bytes\n\n", getFileSize(file));
}
long fileSize = getFileSize(file);
fileBuf = new BYTE[fileSize];
fread(fileBuf, fileSize, 1, file);
for (int i = 0; i < 100; i++){
printf("%X ", fileBuf[i]);
}
_getch();
delete[]fileBuf;
fclose(file); // Almost forgot this
return 0;
}
(I provided that much code because I want to be clear, to help you get the idea about what I am trying to do)
First of all, I need to get the first 14 bytes and write them in the console as text, and then, in a for I need to write something like this for each color:
black col_id = 1; R = 00; G = 00; B = 00;
red col_id = 2; R = FF; G = 00; B = 00;
etc...
How can I read and translate these bytes?
It is correct as you have it to write out the 14 bytes.
a technique is to create a struct with the layout of your records, then cast e.g. (C-style)
typedef struct
{
char name[10];
long col_id;
unsigned char R;
unsigned char G;
unsigned char B;
} rec;
rec* Record = (rec*)(fileBuf + StartOffsetOfRecords);
now you can get the contents of the first record
Record->name, ...
getting next record is just a matter of moving Record forward
++Record;
You could also have a struct for the header to make it more convenient to pickout the number of records, it is good to use stdint.h in order to get well defined sizes. also to pack structures on byte boundary to make sure no padding is done by the compiler i.e. #pragma pack(1) at the top of your source.
typedef struct
{
char signature[14];
uint32_t tableaddress;
uint32_t records;
} header;
typedef struct
{
char name[10];
uint32_t col_id;
unsigned char R;
unsigned char B;
unsigned char G;
} rec;
so instead when you read you could do like this
header Header;
rec* Record;
fread(&Header,sizeof(header),1,file);
fread(fileBuf,1,fileSize,file);
Record = (rec*)(fileBuf); // first record can be accessed through Record