Segmentation fault when trying to create a buffer of 100MB - c++

Question

I am trying to write a large binary file into a buffer in a C++ program. GDB always gets a segfault after trying to create a buffer the same size as the file read. It either fails on fclose(pf), rewind, or f(open) which leads me to believe that there is something wrong when I am trying to create the buffer. My code segment is as follows.
static int fileTransfer(struct mg_connection *conn, char * filename){
FILE *fp = fopen(filename, "r");
fseek(fp, 0, SEEK_END);
int size = ftell(fp);
char buf[size];
fclose(fp);
// This is an attempt to stop a segment fault from rewind.
fp = fopen(filename, "r");
conn->connection_param = (void *) fp;
size_t n = 0;
if(fp != NULL)
{
n = fread(buf, 1, sizeof(buf), fp);
mg_send_data(conn, buf, n);
if(n < sizeof(buf) || conn->wsbits != 0)
{
fclose(fp);
conn->connection_param = NULL;
}
}
return 1;
}
I have tried putting print statements in this code but they don't print to the console as they are running in a separate thread. Can someone give me some insight on why this segfault is happening, or some suggestions on how to make this code more efficient.
I should note that this code works properly on 1 and 10 MB files but not on anything larger.

Answer 1

never do this:
int size = ftell(fp);
char buf[size];
You are creating size on the STACK, not on the heap.... 100MB on the stack will not work.
AND... size must be a constant number, not a number coming from ftell(). I even don't know how it is compiling...
What you have to to is to allocate memory using malloc() or new operator.
static int fileTransfer(struct mg_connection *conn, char * filename){
FILE *fp = fopen(filename, "r");
fseek(fp, 0, SEEK_END);
int size = ftell(fp);
char * buf = new char[size]; // fix also here!
fclose(fp);
// This is an attempt to stop a segment fault from rewind.
fp = fopen(filename, "r");
conn->connection_param = (void *) fp;
size_t n = 0;
if(fp != NULL)
{
n = fread(buf, 1, size, fp); // fix also here!
mg_send_data(conn, buf, n);
if(n < size || conn->wsbits != 0)
{
fclose(fp);
conn->connection_param = NULL;
}
}
delete [] buf; // and you have to deallocate your buffer
return 1;
}

Answer 2

You are creating a buffer with automatic storage duration, which means it will be put on the stack by g++. The default stack size for any OS known to me is below 100 MB, meaning it will cause a segfault on system supporting them.
Try allocating your buffer with dynamic storage duration, which will place it on the heap.

Answer 3

What's going on is actually the namesake of this site! Basically, what is happening is your program is created it has a set amount of memory allocated for the stack.
When you create char buf[size], you are using a C99 feature called a variable length array (VLA). This allocates space on the stack for buf. However, buf is too large for the stack, so your program fails.
In order to fix this problem, you should use char * buf; and then do buf = malloc(size). This will place buf on the heap, which is larger than the stack. It also lets you check if you do not have enough memory, by checking if malloc() returns NULL. You need to be sure to free(buf) before you exit though!
As a side note, you can check how much space you have on the stack by using the ulimit -s command.

Answer 4

That seems like a lot to allocate on the stack. What if you put it on the heap instead?
char *buf = new char[size];

Answer 5

Use std::vector. Then you don't have the issues of stack space, or the other issue of writing non-standard C++ code:
#include <vector>
//...
static int fileTransfer(struct mg_connection *conn, char * filename)
{
FILE *fp = fopen(filename, "r");
fseek(fp, 0, SEEK_END);
int size = ftell(fp);
std::vector<char> buf(size);
fclose(fp);
fp = fopen(filename, "r");
conn->connection_param = (void *) fp;
size_t n = 0;
if(fp != NULL)
{
n = fread(&buf[0], 1, buf.size(), fp);
mg_send_data(conn, &buf[0], n);
if(n < buf.size() || conn->wsbits != 0)
{
fclose(fp);
conn->connection_param = NULL;
}
}
return 1;
}