I am trying to read a binary file using fread() and Rcpp seems to be able to read the file, given ftell() returns the proper size. When I try to print the first byte, it either returns a ☐ or nothing at all. Then RStudio crashes. This code runs perfectly fine in VSCode, but not through Rcpp.
This is how I am trying to read the file.
inline void readFile(string filePath){
//read a file using the C fopen function and store to fileData
FILE* file = fopen(filePath.c_str(), "rb");
if (file == NULL){Rcpp::stop("Cannot open file");}
//find size and print it to console
fseek(file, 0, SEEK_END);
int sizeOfFile = ftell(file);
if (sizeOfFile < 1 || sizeOfFile == NULL){Rcpp::stop("Bad File size");}
Rcpp::Rcout << "File size: " << sizeOfFile << endl;
fileData = (char*)malloc(sizeof(char*)*sizeOfFile);
rewind(file);
fread(fileData, sizeOfFile, 1, file);
fclose(file);
arrayPointer = fileData;
end = fileData + sizeOfFile;
if(arrayPointer == NULL){Rcpp::stop("arrayPointer is null");}
Rcpp::Rcout << "ArrayPointer: " << *(uint8_t*)&arrayPointer[0] << endl; //crashes here
// Rcpp::Rcout << "File size: " << sizeOfFile << endl;
}
If I comment out where I print the first value in arrayPointer then program crashes in the next line after I call this function.
const_array_iterator(string filePath) {
//set up the iterator
readFile(filePath);
//read first 28 bytes of fileData put it into params -> metadata
uint32_t params[7]; //Crashes here too
memcpy(¶ms, arrayPointer, 28);
arrayPointer+=32; //first delimitor is 4 bytes
Rcpp::Rcout << "Copied params" << endl;
magicByteSize = params[0];
rowType = params[1];
nRows = params[2];
colType = params[3];
nCols = params[4];
valueWidth = params[5];
oldIndexType = params[6];
memcpy(&value, arrayPointer, valueWidth);
arrayPointer += valueWidth;
memcpy(&newIndexWidth, arrayPointer, 1);
arrayPointer++; //this should make it point to first index
}
My R code
library(Rcpp)
library(RcppClock)
library(RcppEigen)
sourceCpp("src\\playground.cpp") # The file the previous code blocks belong to
iteratorBenchmark(10, 10, 5.0)
This code is for use in a custom iterator, and I think it will work fine if these issues are fixed. I tried using an ifstream, but ran into similar issues. I have tried running this on linux as well as windows (WSL), but neither seem to work. I know the file is being read, as ftell() returns the correct amount of bytes. The data just seems to not be read properly from the file.
Related
I am very new to socket programming, and i am trying to send over TCP connection but getting few errors.
here is my code
FILE* File;
char* Buffer;
unsigned long Size;
File = fopen("C:\\test.zip", "rb");
if (!File)
{
printf("Error while readaing the file\n");
return;
}
// file size 1
fseek(File, 0, SEEK_END);
Size = ftell(File);
fseek(File, 0, SEEK_SET);
Buffer = new char[Size];
fread(Buffer, Size, 1, File);
char cSize[MAX_PATH];
sprintf(cSize, "%i", Size);
cout << "MAX PATH " << MAX_PATH<<endl;
cout << "cSize: " << cSize << endl;
fclose(File);
`
So this to find the size of my file. most of the code i am trying it out from other questions in here but it didnt solve my problem.
'
my send and recv:
unsigned long filechunk = 1025;
unsigned long byteSent = 0;
unsigned long bytesToSend = 0;
send(Sub, cSize, MAX_PATH, 0); // File size to client
while (byteSent < Size) {
if ((Size - byteSent) >= filechunk) {
bytesToSend = filechunk;
}
else {
bytesToSend = Size - byteSent;
}
if (send(Sub, Buffer + byteSent, bytesToSend, 0)) {
std::cout << "Sent: ";
}
byteSent += bytesToSend;
std::cout << "Size : "<<Size<<" BytesSent : "<<byteSent<<" Bytes to send: " << bytesToSend << std::endl;
system("pause");
on the client side:
int Size;
char* Filesize = new char[5000000]; // is there a better way? my sfiles size are unknown but at least 50mb
if (recv(Socket, Filesize, 5000000, 0)) // File size
{
Size = atoi((const char*)Filesize);
printf("File size: %d\n", Size);
}
char* Buffer = new char[Size];
FILE* File;
File = fopen("test.zip", "wb"); //start copying from the server, creating the file first.
std::string convert;
long conv;
std::cout << "Size: " << Size << std::endl;
int total=Size;
int byteRecv = 0;
int recvCheck;
int bytes = 1025;
//getting the file
while (byteRecv < Size ) {
recvCheck = recv(Socket, Buffer, bytes, 0);
if (recvCheck >0) // File
{
fwrite(Buffer, 1, byteRecv, File);
std::cout << "Recieved:" << byteRecv << std::endl;
Size -= byteRecv;
byteRecv += byteRecv;
std::cout << "Error: " << WSAGetLastError();
}
else {
std::cout << "Error: " << WSAGetLastError();
total += 1; // the loop often get into infinit loop so i force it in case of this error.
if (total > 3) {
break;
}
}
}
fclose(File);
So, i know it is not very efficient and i am not sure if there are similar questions as i have been digging in here for a few weeks now.
-is there a better way i can make a char*[]? as i dont know the size of the files i want to send yet.
- does ftell() and sifeof() work the same way?
-when i check for the size i recved from the server it is alays wrong. Ex: server file: 32633513, recv size: 3263
-most of the code i have taken from other problems and combined it. if you see anything that is not needed do tell me so i take notes of that.
There is a lot of wrong things but that may correct your problem at first:
On the client side replace (your are both decrementing the total count of bytes and the received ones with the wrong value):
Size -= byteRecv;
byteRecv += byteRecv;
with:
byteRecv += recvCheck; // actualizes the count of received bytes
The other problem is your buffer size. Never try to get an entire file in memory, this is nonsense in general; files are usually managed chunks by chunks. As you are reading at most 1025 bytes in each loop, then only use a buffer of size 1025, you don't need more. Same for reading and writing...
I have a set of public and private keys that I want to use in my code. Both keys are stored in files, but the public key is stored in a text file in which the first line has "id : {a number}" and then from line two starts tha public key.
I tried to write a function to parse the file and return an RSA pointer (RSA *). It works by reading the first line of the file and ignoring it, just to move the file pointer, and then I use pub_key=PEM_read_RSAPublicKey(fp,NULL,NULL,NULL); to fill pub_key and return it.
This is my function call pub_key = parse_pubkey_from_common_file("encryption/1.id"); and this is my function:
RSA *parse_pubkey_from_common_file(char *filepath)
{
RSA *pub_key=NULL;
FILE *fp=NULL;
size_t len = 0;
size_t nread = 0;
char *line = NULL;
fp = fopen(filepath, "rt");
bool first=true;
while ((int)(nread = getline(&line, &len, fp)) != -1)
{
line[nread-1] = '\0'; // get rid of the "\n"
std::cout << "line = " <<line << "length = "<<nread<< std::endl;
if (first)
break;
}
free(line);
pub_key = PEM_read_RSAPublicKey(fp,NULL,NULL,NULL);
if (pub_key!=NULL)
{
fprintf(stderr,"Public key read.\n");
}
BIO * keybio = BIO_new(BIO_s_mem());
RSA_print(keybio, pub_key, 0);
char buffer [1024];
std::string res = "";
while (BIO_read (keybio, buffer, 1024) > 0)
{
std::cout << buffer;
}
BIO_free(keybio); //appropriate free "method"
fclose(fp);
std::cout << "------------------ from the original file ------------------" << '\n';
fp = fopen("encryption/public.pem", "rt");
pub_key = PEM_read_RSAPublicKey(fp,NULL,NULL,NULL);
if (pub_key!=NULL)
{
fprintf(stderr,"Public key read.\n");
}
keybio = BIO_new(BIO_s_mem());
RSA_print(keybio, pub_key, 0);
res = "";
while (BIO_read (keybio, buffer, 1024) > 0)
{
std::cout << buffer;
}
BIO_free(keybio); //appropriate free "method"
unsigned int cipher_size=RSA_size(pub_key);
std::cout << "cipher_size is: "<< cipher_size << '\n';
fclose(fp);
return pub_key;
}
From "std::cout ---- from the original file -----" and below is code that does the same thing but with the original (.pem) file with the public key.
When I run the function with this part commented out I can not use correctly the returned RSA *. But when this runs and overwrites the above variable, I can use it correctly later with no problems.
I also tried to print what I read to find any differences but there were not any.
Here are the files I use and a screenshot of the execution:
the first file accessed during the execution with id: {a number} in it
the second file un-edited .pem, with the public key:
last, the execution output:
It might be something obvious that I miss but, I cannot figure out why this happens.
Any help will be much appreciated.
Even though the text file to which I saved all the samples contains (possibly) proper samples, the sound file generated using the same set of data contains only the noise. The code responsible for writing the wav file:
void Filter::generateFrequencySound()
{
SNDFILE * outfile;
SF_INFO sfinfo;// = {0};
memset (&sfinfo, 0, sizeof (sfinfo)) ;
//preparing output file
sfinfo.format = SF_FORMAT_WAV | SF_FORMAT_PCM_16;
sfinfo.channels = 1;
sfinfo.samplerate = 44100;
std::cout << "Trying to save samples to a file" << std::endl;
const char* path = "FilterInFrequency.wav";
outfile = sf_open(path, SFM_WRITE, &sfinfo);
if(!(outfile))
{
std::cout << "Failed to create output file" << std::endl;
sf_perror(outfile);
return;
}
unsigned long savedSamples = sf_write_double( outfile,
outputOfFrequencyFiltration,
bufferSize);
if(savedSamples > bufferSize)
{
std::cout << "Failed to save all samples into outflie. Number of sampels " << savedSamples << std::endl;
sf_close(outfile);
return;
}
sf_close(outfile);
QSound::play("FilterInFrequency.wav");
}
The code responsible for writing samples into a text file:
QFile file("finalResult_1.txt");
if(!file.open(QIODevice::WriteOnly))
{
std::cout << "something went wrong";
exit(16);
}
QTextStream outstream(&file);
for(unsigned long i = 0; i < bufferSize; i++)
{
QString line = QString::number(outputOfFrequencyFiltration[i]);
outstream << line << "\n";
}
file.close();
Comparison of divergence between wav and plotted text file can be seen in the attached image. The plots have been created using the same amount of data (~20500 samples- ~10% of the output file). The file size is same for both plots.
What could be the possible reason for the differences?
textfile
wavfile
I'm working on what will be a pretty large and complicated system and trying to make sure it's as watertight as possible right from the start. Whilst running some memory checks, I noticed something odd when using stringstreams: they don't always seem to release all the memory when they get deleted/go out of scope.
I've tried searching the internet for answers, but most are old (so possibly out of date) and/or are more concerned with refreshing the contents than releasing the memory, so I've not really been able to tell if it's a known issue or a common mistake I'm making.
I've written a simple test to show what's going on:
#include <iostream>
#include <string>
#include <sstream>
#include <fstream>
using namespace std;
float getMemUsage(int& pid)
{
if (pid < 0)
pid = getpid();
char buf[30];
snprintf(buf, 30, "/proc/%u/statm", (unsigned)pid);
FILE* pf = fopen(buf, "r");
if (pf)
{
unsigned size; // total program size
//unsigned resident;// resident set size
//unsigned share;// shared pages
//unsigned text;// text (code)
//unsigned lib;// library
//unsigned data;// data/stack
//unsigned dt;// dirty pages (unused in Linux 2.6)
fscanf(pf, "%u" /* %u %u %u %u %u"*/, &size/*, &resident, &share, &text, &lib, &data*/);
fclose(pf);
return size/1024.0;
}
else
return -1.0;
}
int main(int argc, char* argv[])
{
if (argc < 2)
cerr << "no file specified\n";
ifstream file;
file.open(argv[1]);
int pid = -1;
const float memUseAtStart = getMemUsage(pid);
{
float memUseBefore = getMemUsage(pid);
stringstream sstream;
float memUseAfter = getMemUsage(pid);
cerr << "\tMemory use change after stringstream declaration: " << memUseAfter - memUseBefore << endl;
memUseBefore = getMemUsage(pid);
filebuf* pbuf = file.rdbuf();
memUseAfter = getMemUsage(pid);
cerr << "\tMemory use change after getting file buffer: " << memUseAfter - memUseBefore << endl;
memUseBefore = getMemUsage(pid);
sstream << pbuf;
memUseAfter = getMemUsage(pid);
cerr << "\tMemory use change after copying file contents: " << memUseAfter - memUseBefore << endl;
memUseBefore = getMemUsage(pid);
sstream.clear();
sstream.str( string() );
memUseAfter = getMemUsage(pid);
cerr << "\tMemory use change after 'clearing': " << memUseAfter - memUseBefore << endl;
}
cerr << "Overall memory use change: " << getMemUsage(pid) - memUseAtStart << endl;
file.close();
return 0;
}
Which gives me the following output when called with a file larger than around 32K:
Memory use change after stringstream declaration: 0
Memory use change after getting file buffer: 0
Memory use change after copying file contents: 0.0322266
Memory use change after 'clearing': 0
Overall memory use change: 0.00195312
I'm running on Linux (SL6.6) and compiling with gcc 4.1.2 (though I've also tried clang and ICC with similar results).
Obviously, it's not a huge leak; it's just a little annoying that I can't make it completely tidy... Is there something I can/should do to release the memory manually? Or is it just something weird (with my setup and/or stringstream itself) I'll have to live with?
NB The intended use for the stringstream is to read in some file contents above and then parse them line by line; I would try using istringstream but I couldn't figure out how to set its value from the ifstream...
Thanks!
Next iteration of my question:
Thank you for your inputs, it has helped me to understand a little bit more about the Frame and inputSamples utility.
I’ve done modifications to my source code with the new knowledge you’ve given me. But I still have problems, so I might not have understood fully what you meant.
Here is my OpenFile function, sorry for the name but I’ll refactor later; when it’ll work =)
//-----------------------------------------------------------------------------
/*
This Function Open a File containing the Audio, Binary, Data.
*///___________________________________________________________________________
const short* OpenFile(const char* fileName, long& fileSize, WavFormat* wav)
{
// ouvre le fichier
ifstream file;
file.open((char*)fileName, ios::binary|ios::in);
if (file.good())
{
// Read the WAV's Header
wav = CheckWavHeader(file, wav);
cout << "chunkID: " << wav->chunkID <<'\n';
cout << "chunkSize: " << wav->chunkSize <<'\n';
cout << "format: " << wav->format <<'\n';
cout << "subChunk1ID: " << wav->subChunk1ID <<'\n';
cout << "subChunk1Size: " << wav->subChunk1Size <<'\n';
cout << "audioFormat: " << wav->audioFormat <<'\n'; // audioFormat == 1, alors PCM 16bits
cout << "numChannels: " << wav->numChannels <<'\n';
cout << "sampleRate: " << wav->sampleRate <<'\n';
cout << "byteRate: " << wav->byteRate <<'\n';
cout << "blockAlign: " << wav->blockAlign <<'\n';
cout << "bitsPerSample: " << wav->bitsPerSample <<'\n';
cout << "subChunk2ID: " << wav->subChunk2ID <<'\n';
cout << "subChunk2Size: " << wav->subChunk2Size <<'\n';
// Get the file’s size
file.seekg(0L, ios::end);
fileSize = ((long)file.tellg() - DATA_POS);
file.seekg(DATA_POS, ios::beg); // back to the data.
// Read the Data into the Buffer
uint nbSamples = fileSize / sizeof(short);
short* inputArray = new short[nbSamples];
file.read((char*)inputArray, fileSize);
// Close the file and return the Data
file.close();
return (const short*)inputArray;
}
else
{
exit(-1);
}
}
I’m opening the file, checking its size, create a short buffer and read the wav’s data into the short buffer and finally I return it.
In the main, for now I commented the G711 decoder.
When I run the application, the faacEncOpen gives me 2048 for inputSamples (it’s logic since I have 2 channels in the Wav’s file for a FRAME_LEN of 1024).
So if I understood correctly, 1 Frame == 2048 samples for my application. So for each Frame I call the faacEncEncode, I give the tmpInputBuffer that is a buffer of the same size as inputSamples at the inputBuffer[i * inputSamples] index.
//-----------------------------------------------------------------------------
/*
The Main entry Point of the Application
*///_____________________________________________________________________________
int main()
{
// Get the File's Data
WavFormat* wav = new WavFormat;
long fileSize;
const short* fileInput = OpenFile("audioTest.wav", fileSize, wav);
// G711 mu-Law Decoder
//MuLawDecoder* decoder = new MuLawDecoder();
//short* inputBuffer = decoder->MuLawDecode_shortArray((byte*)fileInput, (int)nbChunk);
short* inputBuffer = (short*)fileInput;
// Info for FAAC
ulong sampleRate = wav->sampleRate;
uint numChannels = wav->numChannels;
ulong inputSamples;
ulong maxOutputBytes;
// Ouvre l'Encodeur et assigne la Configuration.
faacEncHandle hEncoder = faacEncOpen(sampleRate, numChannels, &inputSamples, &maxOutputBytes);
faacEncConfigurationPtr faacConfig = faacEncGetCurrentConfiguration(hEncoder);
faacConfig->inputFormat = FAAC_INPUT_16BIT;
faacConfig->bitRate = 64000;
int result = faacEncSetConfiguration(hEncoder, faacConfig);
/*Input Buffer and Output Buffer*/
byte* outputBuffer = new byte[maxOutputBytes];
int nbBytesWritten = 0;
Sink* sink = new Sink();
uint nbFrame = fileSize / inputSamples;
int32_t* tmpInputBuffer = new int32_t[inputSamples];
for (uint i = 0; i < nbFrame; i++)
{
strncpy((char*)tmpInputBuffer, (const char*)&inputBuffer[i * inputSamples], inputSamples);
nbBytesWritten = faacEncEncode(hEncoder, tmpInputBuffer, inputSamples, outputBuffer, maxOutputBytes);
cout << 100.0 * (float)i / nbFrame << "%\t nbBytesWritten = " << nbBytesWritten << "\n";
if (nbBytesWritten > 0)
{
sink->AddAACStream(outputBuffer, nbBytesWritten);
}
}
sink->WriteToFile("output.aac");
// Close AAC Encoder
faacEncClose(hEncoder);
// Supprimer tous les pointeurs
delete sink;
//delete decoder;
delete[] fileInput;
//delete[] inputBuffer;
delete[] outputBuffer;
delete[] tmpInputBuffer;
system("pause");
return 0;
}
When the output Data is Dumped into an .acc file (as RAW AAC), I use the application mp4muxer.exe to create an .mp4 file to listen to the final converted sound. But the sound is not good at all...
I'm wondering if there is something I'm not seeing or do not unserstand that I should.
Thank you in advance for your useful inputs.
Each call to faacEncEncode encodes inputSamples samples, not just one. Your main loop should read that many samples from the WAV file into the input buffer, then call faacEncEncode once for that buffer, and finally write the output buffer to the AAC file.
It's possible that I've misunderstood what you're doing (if so, it would be useful to know: (1) What's the OpenFile function you're calling, and does it (despite its name) actually read the file as well as opening it? (2) How is inputBuffer set up?) but:
faacEncEncode expects to be given a whole frame's worth of samples. A frame is the number of samples you got passed back in inputSamples when you called faacEncOpen. (You can give it less than a whole frame if you've reached the end of the input, of course.)
So you're getting 460 and 539 bytes for each of two frames -- not for 16 bits in each case. And it looks as if your input-data pointers are actually offset by only one sample each time, so you're handing it badly overlapping frames. (And the wrong number of them; nbChunk is not the number of frames you have.)