I have been having some issues with trying to read data out of a binary file that is structured so that the header is the first 1024, or 4069 bytes, and the payloads are all regular blocks after that. Below is a model of the code that I have been using:
Header:
#pragma once
#include <stdio.h>
#include <iostream>
#include <fstream>
class BinaryFile
{
public:
long MagicNumber;
long HeaderSize;
long PayloadSize;
//... about 20 other header details
char* Padding; // unused area of the header that is reserved for future use
std::ifstream BinaryFileStream;
BinaryFile();
int Open_Binary_File(const char* path);
int Load_Payload_Into_Buffer(int payload_index, void* buffer);
};
And the C++ code that I have tried to use:
#include "BinaryFile.h"
BinaryFile:BinaryFile()
{
MagicNumber = 0;
HeaderSize = 1024;
PayloadSize = 62830080;
// ... all the other header items, initalized with default values
std:ifstream BinaryFileStream;
Padding[360];
}
int BinaryFile::Open_Binary_File(const char* path) // This is the one that I would like to be using
{
BinaryFileStream.open(path, std::ifstream::binary);
BinaryFileStream.read((char*)&MagicNumber, (size_t) 4);
BinaryFileStream.read((char*)&HeaderSize, (size_t) 4);
// ... The rest of the header is placed into the object
BinaryFileStream.read((char*)&Padding, (size_t) 360);
// The file pointer should now be 1024 bytes into the file, at the end of the header, and at the start of the first payload
return 0;
}
int Load_Payload_Into_Buffer(int payload_index, void* buffer)
{
buffer = malloc(PayloadSize);
size_t offset = HeaderSize + static_cast<long long>(frame_index) * PayloadSize;
BinaryFileStream.seekg(offset, BinaryFileStream.beg);
BinaryFileStream.read((char*)buffer, PayloadSize);
return 0;
Error:
return 1;
}
Below are some variations that I have tried:
int BinaryFile::Open_Binary_File(const char* path)
{
BinaryFileStream.open(path, std::ifstream::binary);
BinaryFileStream.read((char*)&MagicNumber, (size_t) 4);
BinaryFileStream.read((char*)&HeaderSize, (size_t) 4);
// ... The rest of the header is placed into the object
BinaryFileStream.read((char*)&Padding, (size_t) 360);
BinaryFileStream.clear();
BinaryFileStream._Seekbeg.seekg((size_t)0, BinaryFileStream._Seekbeg);
BinaryFileStream.sync();
// The file pointer should now be 0 bytes into the file, at the start of the header
return 0;
}
int BinaryFile::Open_Binary_File(const char* path)
{
BinaryFileStream.open(path, std::ifstream::binary);
BinaryFileStream.read((char*)&MagicNumber, (size_t) 4);
BinaryFileStream.read((char*)&HeaderSize, (size_t) 4);
// ... The rest of the header is placed into the object
BinaryFileStream.read((char*)&Padding, (size_t) 360);
BinaryFileStream.clear();
BinaryFileStream.seekg((size_t)-1024);
BinaryFileStream.sync();
// The file pointer should now be 0 bytes into the file, at the start of the header
return 0;
}
The issue that I am running into to is, the payload that is being returned to the buffer contains some of the next payload. The setting of the file pointer doesn't seem to be working the way that I expect it to. i.e. if I say
BinaryFileStream._Seekbeg.seekg(position, BinaryFileStream._Seekbeg)
I expect the pointer to return to the start of the file, and then seek along the number of bytes that I have said in position. Is there a different way to do this? or is there something that I am missing?
Turns out, what I needed to do was multiply the HeaderSize by CHAR_BIT so it would look something like this:
int Load_Payload_Into_Buffer(int payload_index, void* buffer)
{
buffer = malloc(PayloadSize);
size_t offset = HeaderSize*CHAR_BIT + static_cast<long long>(frame_index) * PayloadSize;
BinaryFileStream.seekg(offset, BinaryFileStream.beg);
BinaryFileStream.read((char*)buffer, PayloadSize);
return 0;
Error:
return 1;
}
And what I thought was a portion of the next payload , was in fact the same payload, but split at some arbitrary point, and stacked next to it. So the payload was split approx. 3/4 of the way through, and it was then re-arranged, so that the original last quarter was now at in the first position, and the original first 3/4 was now in the 2nd position.
Hopefully that makes some sense, and will help someone in the future!
Related
I am still learning Cpp, so please advise if I am misunderstanding here.
Using an ESP32, I am trying to read / write files to Flash / FFat. This is the method I have created which should read a file from flash and load it into PSRAM:
unsigned char* storage_read(char* path) {
File file = FFat.open(path);
if(!file) {
Serial.println("no file");
return 0x00;
}
int count = file.size();
unsigned char* buffer = (unsigned char*)ps_malloc(count);
Serial.printf("Bytes: %d\n", count);
Serial.printf("Count: %d\n", sizeof(buffer));
for (int i = 0; i < count; i++) {
buffer[i] = (unsigned char)file.read();
}
file.close();
return buffer;
}
The problem is that I get the contents of my b64 data file, with the addition of several extra bytes of data globbed on the end.
Calling the method with:
Serial.printf("Got: %s", storage_read("/frame/testframe-000.b64"));
I get the output:
Bytes: 684
Count: 4
Got: <myb64string> + <68B of garbage>
Why would sizeof not be returning the proper size?
What would be the proper way of loading this string into a buffer?
Why would sizeof not be returning the proper size?
That's because sizeof() has a very specific function (not very intuitive). It is used - compile time - to query the size of the data type passed to it. Calling sizeof(buffer) returns the size, in bytes, of the type of variable buffer. It's an unsigned char*, so a 4-byte memory address. So that's what you get.
What would be the proper way of loading this string into a buffer?
What I noticed is that you're expecting to load string data from your file, but you don't explicitly terminate it with a zero byte. As you probably know, all C strings must be terminated with a zero byte. Data that you load from the file most likely doesn't have one (unless you took extra care to add it while saving). So when you read a string from a file sized N bytes, allocate a buffer of N+1 bytes, load the file into it and terminate it with a zero. Something like this:
unsigned char* storage_read(char* path) {
File file = FFat.open(path);
if(!file) {
Serial.println("no file");
return 0x00;
}
int count = file.size();
unsigned char* buffer = (unsigned char*)ps_malloc(count + 1); //< Updated
Serial.printf("Bytes: %d\n", count);
Serial.printf("Count: %d\n", sizeof(buffer));
for (int i = 0; i < count; i++) {
buffer[i] = (unsigned char)file.read();
}
buffer[count] = 0; //< Added
file.close();
return buffer;
}
And since you're returning a heap-allocated buffer from your function, take extra care to remember to delete it in caller when finished. This line in your code will leak the memory:
Serial.printf("Got: %s", storage_read("/frame/testframe-000.b64"));
I am reading a collection file (20 or so small files in one) with fread_s and the content is being written in a struct. Like 99% of the times it reads the data correctly, but one time, at always the same position it seems to ignore the byte size of the element size parameter and just reads 500 or so bytes until it aborts and reports a feof error. The thing is, it doesn't even write the last three bytes of the int to the struct.
When I remove the checks, and let it continue reading, it will read normal again, like nothing happened.
I observed that the _Placeholder variable in the file pointer gets changed to a different value, and then back again, but I guess its just the eof error getting packed in there.
#pragma pack(push, 1)
struct fileHeader {
__int32 typeID;
bool isGFX;
char filename[8];
__int32 offset;
};
#pragma pack(pop)
#define HEADERSIZE 68
#define FILEHEADERSIZE 17
....
FILE *file;
fopen_s(&file, filename.c_str(), "r");
for (int i = 0; i < header.files - 1; i++) {
fseek(file, HEADERSIZE + i * FILEHEADERSIZE, 0);
fileHeader headerFile;
memset(&headerFile, 0, FILEHEADERSIZE);
int oldPointer = ftell(file); //118
int d = fread_s(&headerFile, FILEHEADERSIZE, FILEHEADERSIZE, 1, file); //returns 0
int newPointer = ftell(file); //630
int e = errno; //0
int ea = ferror(file); //0
int ef = feof(file); //1
//getting used here in a function
}
headerFile = {typeID=17 isGFX=true filename=0x00fefd05 "CURSORR" offset = 164} - offset should be 6820
Like Jonathan Leffler said in the comments, the mistake was, that I didn't read in binary mode. a simple change fopen_s(&file, filename.c_str(), "r"); to fopen_s(&file, filename.c_str(), "rb"); fixed the problem.
I am trying to copy a WAV sound in C. the original file is a 2 seconds file, but I want to replicate the data in the destination file several times, so that it plays longer. For example, if I copy it 3 times, it should play for 6 seconds... right?
But for some reason, even though the destination file is bigger than the original file, it still plays for 2 seconds...
Can anyone help please?
Here is my code:
#include <iostream>
#include <stdio.h>
#include <stdlib.h>
using namespace std;
typedef struct header_file
{
char chunk_id[4];
int chunk_size;
char format[4];
char subchunk1_id[4];
int subchunk1_size;
short int audio_format;
short int num_channels;
int sample_rate;
int byte_rate;
short int block_align;
short int bits_per_sample;
char subchunk2_id[4];
int subchunk2_size;
} header;
typedef struct header_file* header_p;
int main()
{
FILE * infile = fopen("../files/man1_nb.wav","rb"); // Open wave file in read mode
FILE * outfile = fopen("../files/Output.wav","wb"); // Create output ( wave format) file in write mode
int BUFSIZE = 2; // BUFSIZE can be changed according to the frame size required (eg: 512)
int count = 0; // For counting number of frames in wave file.
short int buff16[BUFSIZE]; // short int used for 16 bit as input data format is 16 bit PCM audio
header_p meta = (header_p)malloc(sizeof(header)); // header_p points to a header struct that contains the wave file metadata fields
int nb; // variable storing number of byes returned
if (infile)
{
fread(meta, 1, sizeof(header), infile); // Read only the header
fwrite(meta,1, sizeof(*meta), outfile); // copy header to destination file
int looper = 0; // number of times sound data is copied
for(looper=0; looper <2; looper++){
while (!feof(infile))
{
nb = fread(buff16,1,BUFSIZE,infile); // Reading data in chunks of BUFSIZE
count++; // Incrementing Number of frames
fwrite(buff16,1,nb,outfile); // Writing read data into output file
}
fseek(infile, 44, SEEK_SET); // Go back to end of header
}
}
fclose(infile); fclose(outfile);
return 0;
}
Both of your read and write code parts are wrong.
wav files have RIFF format and consists of tlv chunks. Each chunk consists of header and data. Typically wav file consists of 3 chunks: format chunk with FOURCC code, format chunk with PCMWAVEFORMAT struct and data chunk with sound data. Also since size of each chunk is limited by 32 bit of length holding field, large files are constructed by concatenating wav files together.
You need to parse file chunk-by-chunk, and write into destination chunk-by-chunk, updating headers accordingly.
When you change size of your data you'll need to update output header as well.
long total_bytes_written_to_outfile = ftell(outfile);
// correct chunk_size and subchunk2_size just before closing outfile:
fseek(outfile, 0, SEEK_SET);
int size = total_bytes_written_to_outfile - sizeof(*meta);
meta->chunk_size = sizeof(header) - sizeof(meta->chunk_id) - sizeof(meta->chunk_size) + size;
meta->subchunk2_size = size;
fwrite(meta, 1, sizeof(*meta), outfile);
fclose(outfile);
Also, to make sure you are reading correct file check that meta->chunk_size == file size of man1_nb.wav - 8
I need to create a custom reading callback function that can read contents of a file in the form of a std::string into a uint8_t * buf. I tried multiple different methods found around the internet and on stackoverflow but sometimes it works and other the the program infinitely loops or stops execution half way.
I have no problems with amr/3gp files but all wav/pcm files are causing some problems for some reason. All I know its something to do with the reading function I have so far.
Ideally I would like to be able to give the program any type of file and then it converts it.
This is how I am calling the readCallback function from the code:
//create the buffer
uint8_t * avio_ctx_buffer = NULL;
//allocate space for the buffer using ffmpeg allocation method
avio_ctx_buffer = (uint8_t *) av_malloc(avio_ctx_buffer_size);
//Allocate and initialize an AVIOContext for buffered I/O.
//audio variable contains the contents of the audio file
avio_ctx = avio_alloc_context(avio_ctx_buffer, avio_ctx_buffer_size,0, &audio, &readCallback, NULL, NULL);
Here is the callback function that works on some types of files:
static int readCallback(void* opaque, uint8_t * buf, int buf_size){
std::string * file =static_cast<std::string *>(opaque);
if(file->length() == 0){
return AVERROR_EOF; //if we reach to the end of the string, return
// return End of file
}
// Creating a vector of the string size
std::vector<uint8_t> array(file->length());
//Copying the contents of the string into the vector
std::copy(file->begin(),file->end(),array.begin());
//Copying the vector into buf
std::copy(array.begin(),array.end(),buf);
return file->length();
}
After tyring some stuff for awhile, I got a solution using std::stringstream and it works well with several formats I tested with so far: 3gp/amr,wav/pcm,mp3.
Here a the snippet of code:
//Create a string stream that contains the audio
std::stringstream audio_stream(audio);
//create the buffer
uint8_t * avio_ctx_buffer = NULL;
//allocate space for the buffer using ffmpeg allocation method
avio_ctx_buffer = (uint8_t *) av_malloc(avio_ctx_buffer_size);
//Allocate and initialize an AVIOContext for buffered I/O.
//Pass the stringstream audio_stream
avio_ctx = avio_alloc_context(avio_ctx_buffer, avio_ctx_buffer_size,0,&audio_stream, &readCallback, NULL, NULL);
The callback function:
static int readFunction1(void* opaque, uint8_t * buf, int buf_size){
//Cast the opaque pointer to std::stringstream
std::stringstream * me =static_cast<std::stringstream *>(opaque);
//If we are at the end of the stream return FFmpeg's EOF
if(me->tellg() == buf_size){
return AVERROR_EOF;
}
// Read the stream into the buf and cast it to char *
me->read((char*)buf, buf_size);
//return how many characters extracted
return me->tellg();
}
Is there an way to set the start position from an char pointer, which was an file in memory.
I need to read the extract size bytes of data from the stream, and copy them to the supplied data address and return the number of bytes read.
This was my memset tryout, i try to do someting like fread with an file in memory.
Hope someone can help me.
typedef signed long long Int64; // osx for example
Int64 FileStream::read(void* data, Int64 size)
{
// make sure, that size was not > as filename
Int64 wanted_buffer = currentposition + size;
if (wanted_buffer > memfile->GetSize())
size = memfile->GetSize() - currentposition;
// tryout with memcpy
memcpy(data, currentposition + memfile->GetBuffer(), size);
currentposition += size;
// like
// return std::fread(data, 1, static_cast<std::size_t>(size), m_file);
return currentposition;
}
If you're able to use the c FILE struct you can use fseek on a FILE *.