reading .wav and using http post to transfer contents with esp32 esp8266 - c++

Question

Background:
I have a .wav file saved on an SD card. I would like to transfer that file to a server using my esp32. I am using node red to handle the server side activities.
Method Employed:
open the file in binary mode.
evaluate the size of the file
decide on a max upload size and allocate a buffer
Read the file and store to the buffer.
use http post to send data to the server.
if file is too large to send in a single buffer then divide the file up and send multiple http posts.
Problem:
I can successfully send text files. when I try to send .wav files the size of the sent wave file increases and the file is corrupted. Analyzing the file is difficult as its not all text, what I have done is open the file in notepad++ to see if I can spot anything. Everything should be the same in theory but several characters are coming up as blank squares in the transferred file and some are coming up as the exact same.
Analysis/Theory:
I am quite lost as to what the issue is. My leading theory is that a wave file is written in int16_t but in order to post the data it needs to be * uint8_t, maybe when the casting of the int16 to a uint8 data is lost, I looked at trying to change a int16_t into two int8_t bytes as done here https://stackoverflow.com/a/53374797/14050333 but had no luck, maybe I'm jumping to conclusions. Any help would be hugely appreciated!
Code:
Full code used to sell text files.
void loop()
{
WiFiClient client;
Serial.println("starting file upload");
IPAddress host(192, 168, 0, 37);
int port = 1880;
if (!client.connect(host, port))
{ // check connection to host if untrue internet connection could be down
Serial.println("couldn't connect to host");
}
HTTPClient http;
const char* serverName = "http://192.168.0.37:1880/sensor_file";
http.begin(client, serverName);
char *fname = "/sdcard/test_text.txt";
FILE *fp = fopen(fname, "rb"); // read in bytes
//get file size
fseek(fp, 0, SEEK_END); //send file pointer to end of file
int file_size = ftell(fp); //get end position of file
fseek(fp, 0, SEEK_SET); //send pointer back to start
int max_upload_size = 10; // array size, larger = less uploads but too large can cause memory issues
int num_of_uploads = file_size / max_upload_size; // figure out how many evenly sized upload chunks we need
int num_of_uploads_mod = file_size % max_upload_size; //find out size of remaining upload chunk if needed
int i;
//upload file in even chunks
if (num_of_uploads > 0)
{
char buff1[max_upload_size+1] = {}; // array to save file too. add 1 for end of array symbol '\n'
for (i = 0; i < num_of_uploads; i++)
{
fread(buff1, sizeof(buff1)-1, 1, fp); // -1 as don't want to count the '\n'
http.addHeader("File_name", "test file"); //header to say what the file name is
int httpResponseCode = http.POST((uint8_t *)buff1, sizeof(buff1)-1); //send data. Datatype is (uint8_t *)
}
}
//upload any remaining data
if (num_of_uploads_mod > 0)
{
int remainder = file_size - num_of_uploads * max_upload_size;
char buff2[remainder+1] = {};
fread(buff2, sizeof(buff2)-1, 1, fp); //read from file and store to buff2
http.addHeader("File_name", "test file");
int httpResponseCode = http.POST((uint8_t *)buff2, sizeof(buff2)-1); //send buff2 to server
}
http.end(); // Close connection
delay(10 * 1000);
}
Adjustments made for .wav files
int remainder = file_size - num_of_uploads * max_upload_size;
int16_t buff2[remainder+1] = {};
fread(buff2, sizeof(buff2)-1, 1, fp); //remainder
http.addHeader("File_name", "test file");
int httpResponseCode = http.POST((uint8_t *)buff2, sizeof(buff2)-1);

Answer 1

Its working!
There were 2 main issues with the code as outlined by heap underrun. The first issue is that I was reading in the wav file as int16_t the correct datatype to use was uint8_t.
Why are you using an array of int16_t-type elements as a buffer? You are reading a file in binary mode, so be it .wav, .jpg, .ttf, or anything else, it's just a sequence of bytes (uint8_t, not int16_t) anyway. Another thing, fread() expects the size of each object to read as the second parameter and the number of objects to read as the third parameter, so, in case of objects being bytes, first define buffer as uint8_t buff1[max_upload_size] = {}; (no need for +1/-1 games), and then fread(buff1, sizeof *buff1, sizeof buff1 / sizeof *buff1, fp);. The same for buff2. –
heap underrun
The second issue was that I did not include a header in the post stream specifying the content type. As it wasn't needed for the text file and when writing the file in node-red it lets you choose the encoding. I didn't think I would need it, however as it turns out I needed to add:
http.addHeader("Content-Type", "application/octet-stream");
Below is the working code for the file upload section:
if (num_of_uploads > 0)
{
uint8_t buff1[max_upload_size] = {};
for (i = 0; i < num_of_uploads; i++)
{
fread(buff1, sizeof *buff1, sizeof buff1 / sizeof *buff1, fp);
http.addHeader("File_name", "test file"); //header to say what the file name is
http.addHeader("Content-Type", "application/octet-stream");
int httpResponseCode = http.POST(buff1, sizeof(buff1));
}
}
if (num_of_uploads_mod > 0)
{
int remainder = file_size - num_of_uploads * max_upload_size;
uint8_t buff2[remainder] = {};
fread(buff2, sizeof *buff2, sizeof buff2 / sizeof *buff2, fp);
http.addHeader("File_name", "test file");
http.addHeader("Content-Type", "application/octet-stream");
int httpResponseCode = http.POST(buff2, sizeof(buff2));
}
On a slightly interesting side note out of curiosity I tried running the above code but with
uint16_t buff1[max_upload_size] = {};
and
http.POST((uint8_t) buff1, sizeof(buff2));
The file uploaded but the size was 2x what it should be, curiously however the file wasn't corrupted, and played the audio as it was recorded. Just thought that was interesting.
I'll close out this answer as the original question was successfully answered. Again thank you for the help, I've been at this literally weeks and you solved my problems in hours!