I am stuck since 2 days with a seemingly simple calculation.
But I just don't get it.
I am encoding an audio file with a compressing algorithm.
The entire audio file is separated into "chunks" of 960 bytes.
Each chunk is compressed to 60 bytes.
My uncompressed file is 1480320 bytes long.
My encoded file is 46320 bytes long.
Something seems to be wrong.
I tried to calculate the theoretic uncompressed file size from the file size of the encoded audio.
Here is how the file is encoded:
short *m_in;
short *m_out;
unsigned char *m_data;
unsigned char *m_fbytes;
int m_max_frame_size;
int m_frame_size;
int m_sampling_rate;
int m_max_payload_bytes;
int m_bitrate_bps;
int m_iByteLen1FrameEncoded;
int m_iByteLen1FrameDecoded;
m_sampling_rate=48000;
m_max_frame_size = 960*6;
m_max_payload_bytes=1500;
m_bitrate_bps= 24000;
m_iByteLen1FrameEncoded=60;
m_iByteLen1FrameDecoded=960;
m_in = (short*)malloc(m_max_frame_size*sizeof(short));
m_out = (short*)malloc(m_max_frame_size*sizeof(short));
m_data = (unsigned char*)calloc(m_max_payload_bytes,sizeof(char));
m_fbytes = (unsigned char*)malloc(m_iByteLen1FrameDecoded*sizeof(short));
FILE *fin= fopen(uPathInput.c_str(), "rb");
FILE *fout=fopen(uPathOutput.c_str(), "wb");
int curr_read=0;
int stop=0;
while (!stop)
{
int err;
err = fread(m_fbytes, sizeof(short), 960, fin);
curr_read = err;
for(int i=0;i<curr_read;i++)
{
opus_int32 s;
s=m_fbytes[2*i+1]<<8|m_fbytes[2*i];
s=((s&0xFFFF)^0x8000)-0x8000;
m_in[i]=s;
}
if (curr_read < 960)
{
for (int i=curr_read;i<960;i++)
{
m_in[i] = 0;
}
stop = 1;
}
//iLen will always return 60, so I guess the 960 bytes are compressed to 60 bytes, right?
int iLen = opus_encode(m_enc, m_in, m_iByteLen1FrameDecoded, m_data, m_max_payload_bytes);
if (fwrite(m_data, 1, iLen, fout) !=iLen)
{
fprintf(stderr, "Error writing.\n");
}
}
fclose(fin);
fclose(fout);
}
The compression ratio seems to be
960/60 = 16
So I calculated 46320 bytes * 16.
But that gets me to 741120 bytes.
And that doesn't fit. I expected it to be 1480320 bytes.
I am trying to find the error in my calculation, but I just don't manage.
Does anybody see where I went wrong?
Thank you very much for any help!
Right, to expand on my comments. The problem is found here:
fread(m_fbytes, sizeof(short), 960, fin);
You're reading 960 shorts, which should be 2 bytes wide, so you're really reading 1920 bytes. If opus_encode() returns the compressed size in bytes, that would make the compression ration 32 as Robert observed.
I'd also simplify the code processing the chunks:
size_t ITEM_SIZE = sizeof(short);
int ITEM_COUNT = 960;
// fread should first return a short item count, then zero
size_t shorts_read = 0;
while (shorts_read = fread(m_fbytes, ITEM_SIZE, ITEM_COUNT, fin)) {
size_t i = 0;
for (; i<read; i++) {
opus_int32 s;
// etc.
}
for (; i < ITEM_COUNT; i++) {
m_in[i] = 0;
}
// opus_encode() etc
}
You get rid of the useless stop flag and a level of nesting, and the construct is idiomatic for "read until you can't." (See this SO question.)
I retract what I mentioned about the code being hokey, I thought fread returns the bytes read, not the items read.
Related
This question already has answers here:
Zlib deflated input is larger than original input string of chars?
(3 answers)
Closed 5 years ago.
I'm now trying to understand how deflate and inflate work
Here is a simple program with dummyFields struct.
// Here i give a total memory size for the output buffer used by deflate func
#define CHUNK 16384
struct dummyFields
{
long a;
char b;
long c;
float d;
float e;
float f;
float g;
float h;
char i;
unsigned int j;
};
Bytef *dataOriginal = (Bytef*)malloc( sizeof(dummyFields) );
Bytef *dataCompressed = (Bytef*)malloc( CHUNK );
z_stream s
s.zalloc = Z_NULL;
s.zfree = Z_NULL;
s.opaque = Z_NULL;
deflateInit(&s, Z_DEFAULT_COMPRESSION);
s.avail_out = CHUNK;
s.next_out = dataCompressed;
int compressSize = 0;
int decompSize = 0;
dummyFields para;
// set all values equals to 0
memset( ¶, 0, sizeof(dummyFields) );
//Inserts value in struct fields
para.a = 31272;
para.b = 'z';
para.c = 66.54;
para.e = 123;
para.f = 66.54;
.
.
para.j = 123;
//copy this values in a Bytef* elements
memcpy( dataOriginal, ¶, sizeof(dummyFields));
s.avail_in = sizeof(dummyFields);
s.next_in = dataOriginal;
int response = deflate(&s, Z_FINISH);
//don't get any errors here
if( res == Z_STREAM_END ){
compressSize = CHUNK - s.avail_out;
}
}
deflateEnd(&s);
//here I get 45 byte but actual struct sizeof is 40.
printf("Total bytes after compress %d\n",compressSize);
// Trying to get back my data
Bytef *decomp = (Bytef*)malloc( sizeof(Particle) );
z_stream s_inflate;
s_inflate.zalloc = Z_NULL;
s_inflate.zfree = Z_NULL;
s_inflate.opaque = Z_NULL;
inflateInit(&s_inflate);
// data i want to get at the next inflate
s_inflate.avail_in = spaceUsed;
s_inflate.next_in = dataCompressed;
s_inflate.avail_out = sizeof(dummyFields);
s_inflate.next_out = decomp;
int response = inflate( &s_inflate, Z_NO_FLUSH );
if( res == Z_STREAM_END ){
decompSize = CHUNK - s.avail_out;
}
//Here I got 40 bytes which is correct beacuse actual struct size is 40
printf("Total bytes after compress %d\n",decompSize);
inflateEnd( &s_inflate );
dummyFields data;
memset( &data, 0, sizeof(data) );
memcpy( &data, decomp, sizeof(data));
when I tried to back my data from the inflate response I get actual values(whic h is correct). Deflate and Inflate functions work fine.
When i try to find the size of(sizeof(dummyFields)) the struct it give me 40 bytes
Problem
Actual size of struct is 40 when I compress the data it give me 45
bytes how it is possible ?
My requirement data is 30 to 40 bytes is there any another library
which will compress the data 10 to 20 bytes(when I give 30 to 40 bytes)?
Is there some way to GUARANTEE that the output compressed data will
be SMALLER than the input data?
Note
When I increase number of struct fields or size of data 40 to 100 bytes compression result is Ok.
When I decrease number of fields or size 100 bytes to 40 bytes compression result is not Good
There is no GUARANTEE that output compressed data will be SMALLER. Compression implies some overhead for storing some structural information describing the data being packed (like dictionary in primitive case) that allows to represent the data using less space. You may get a larger compressed output for a single compressed structure, but you most likely will get a much smaller compressed output for an array of these structures (especially when they are not very different from each other).
If your compressed output turns out to be larger that uncompressed then just store an uncompressed version.
I am loading a 10GB file into memory and I find that even if I strip away any extra overhead and store the data in nothing but an array it still takes up 53 GB of ram. This seems crazy to me since I am converting some of the text data to longs which take up less room and convert the rest to char * which should take up the same amount of room as a text file. I have about 150M rows of data in the file I am trying to load. Is there any reason why this should take up so much ram when I load it the way I do below?
There are three files here a fileLoader class and its header file and a main that simply runs them.
To answer some questions:
OS is UBUNTU 12.04 64bit
This is on a machien with 64GB of RAM and an SSD hd that I have providing 64GB of swap space for RAM
I am loading all of the data at once becuase of the need for speed. It is critical for the application. All sorting, indexing, and lots of the data intensive work runs on the GPU.
The other reason is that loading all of the data at once made it much simpler for me to write the code. I dont have to worry about indexed files, and mappings to locations in another file for example.
Here is the header file:
#ifndef FILELOADER_H_
#define FILELOADER_H_
#include <iostream>
#include <fstream>
#include <fcntl.h>
#include <stdlib.h>
#include <string.h>
#include <string>
class fileLoader {
public:
fileLoader();
virtual ~fileLoader();
void loadFile();
private:
long long ** longs;
char *** chars;
long count;
long countLines(std::string inFile);
};
#endif /* FILELOADER_H_ */
Here is the CPP file
#include "fileLoader.h"
fileLoader::fileLoader() {
// TODO Auto-generated constructor stub
this->longs = NULL;
this->chars = NULL;
}
char ** split(char * line,const char * delim,int size){
char ** val = new char * [size];
int i = 0;
bool parse = true;
char * curVal = strsep(&line,delim);
while(parse){
if(curVal != NULL){
val[i] = curVal;
i++;
curVal = strsep(&line,delim);
}else{
parse = false;
}
}
return val;
}
void fileLoader::loadFile(){
const char * fileName = "/blazing/final/tasteslikevictory";
std::string fileString(fileName);
//-1 since theres a header row and we are skipinig it
this->count = countLines(fileString) -1;
this->longs = new long long*[this->count];
this->chars = new char **[this->count];
std::ifstream inFile;
inFile.open(fileName);
if(inFile.is_open()){
std::string line;
int i =0;
getline(inFile,line);
while(getline(inFile,line)){
this->longs[i] = new long long[6];
this->chars[i] = new char *[7];
char * copy = strdup(line.c_str());
char ** splitValues = split(copy,"|",13);
this->longs[i][0] = atoll(splitValues[4]);
this->longs[i][1] = atoll(splitValues[5]);
this->longs[i][2] = atoll(splitValues[6]);
this->longs[i][3] = atoll(splitValues[7]);
this->longs[i][4] = atoll(splitValues[11]);
this->longs[i][5] = atoll(splitValues[12]);
this->chars[i][0] = strdup(splitValues[0]);
this->chars[i][1] = strdup(splitValues[1]);
this->chars[i][2] = strdup(splitValues[2]);
this->chars[i][3] = strdup(splitValues[3]);
this->chars[i][4] = strdup(splitValues[8]);
this->chars[i][5] = strdup(splitValues[9]);
this->chars[i][6] = strdup(splitValues[10]);
i++;
delete[] splitValues;
free(copy);
}
}
}
fileLoader::~fileLoader() {
// TODO Auto-generated destructor stub
if(this->longs != NULL){
delete[] this->longs;
}
if(this->chars != NULL){
for(int i =0; i <this->count;i++ ){
free(this->chars[i]);
}
delete[] this->chars;
}
}
long fileLoader::countLines(std::string inFile){
int BUFFER_SIZE = 16*1024;
int fd = open(inFile.c_str(), O_RDONLY);
if(fd == -1)
return 0;
/* Advise the kernel of our access pattern. */
posix_fadvise(fd, 0, 0, 1); // FDADVICE_SEQUENTIAL
char buf[BUFFER_SIZE + 1];
long lines = 0;
while(size_t bytes_read = read(fd, buf, BUFFER_SIZE))
{
if(bytes_read == (size_t)-1)
return 0;
if (!bytes_read)
break;
for(char *p = buf; (p = (char*) memchr(p, '\n', (buf + bytes_read) - p)); ++p)
++lines;
}
return lines;
}
Here is the file with my main function:
#include "fileLoader.h"
int main()
{
fileLoader loader;
loader.loadFile();
return 0;
}
Here is an example of the data that I am loading:
13|0|1|1997|113|1|4|12408012|C9FF921CA04ADA3D606BF6DAC4A0B092|SEMANAL|66C5E828DC69F857ADE060B8062C923E|113|1
14|0|1|1997|113|1|5|12408012|C9FF921CA04ADA3D606BF6DAC4A0B092|SEMANAL|66C5E828DC69F857ADE060B8062C923E|113|1
15|0|1|1997|113|1|6|12408012|C9FF921CA04ADA3D606BF6DAC4A0B092|SEMANAL|66C5E828DC69F857ADE060B8062C923E|113|1
16|0|1|1997|113|1|7|12408012|C9FF921CA04ADA3D606BF6DAC4A0B092|SEMANAL|66C5E828DC69F857ADE060B8062C923E|113|1
17|0|1|1997|113|1|8|12408012|C9FF921CA04ADA3D606BF6DAC4A0B092|SEMANAL|66C5E828DC69F857ADE060B8062C923E|113|1
18|0|1|1997|113|1|9|12408012|C9FF921CA04ADA3D606BF6DAC4A0B092|SEMANAL|66C5E828DC69F857ADE060B8062C923E|113|1
19|0|1|1997|113|1|10|12408012|C9FF921CA04ADA3D606BF6DAC4A0B092|SEMANAL|66C5E828DC69F857ADE060B8062C923E|113|1
20|0|1|1997|113|1|11|12408012|C9FF921CA04ADA3D606BF6DAC4A0B092|SEMANAL|66C5E828DC69F857ADE060B8062C923E|113|1
21|0|1|1997|113|1|12|12408012|C9FF921CA04ADA3D606BF6DAC4A0B092|SEMANAL|66C5E828DC69F857ADE060B8062C923E|113|1
9|0|1|1997|113|1|13|12408012|C9FF921CA04ADA3D606BF6DAC4A0B092|SEMANAL|66C5E828DC69F857ADE060B8062C923E|113|1
27|0|1|1992|125|1|1|10183|9EF534D2CF74B24AC28CBD9BE937A412|SEMANAL|375CCE505F5353CCDE85D4E84A9888D8|125|1
28|0|1|1992|125|1|2|10183|9EF534D2CF74B24AC28CBD9BE937A412|SEMANAL|375CCE505F5353CCDE85D4E84A9888D8|125|1
29|0|1|1992|125|1|3|10183|9EF534D2CF74B24AC28CBD9BE937A412|SEMANAL|375CCE505F5353CCDE85D4E84A9888D8|125|1
30|0|1|1992|125|1|4|10183|9EF534D2CF74B24AC28CBD9BE937A412|SEMANAL|375CCE505F5353CCDE85D4E84A9888D8|125|1
31|0|1|1992|125|1|5|10183|9EF534D2CF74B24AC28CBD9BE937A412|SEMANAL|375CCE505F5353CCDE85D4E84A9888D8|125|1
32|0|1|1992|125|1|6|10183|9EF534D2CF74B24AC28CBD9BE937A412|SEMANAL|375CCE505F5353CCDE85D4E84A9888D8|125|1
33|0|1|1992|125|1|7|10183|9EF534D2CF74B24AC28CBD9BE937A412|SEMANAL|375CCE505F5353CCDE85D4E84A9888D8|125|1
34|0|1|1992|125|1|8|10183|9EF534D2CF74B24AC28CBD9BE937A412|SEMANAL|375CCE505F5353CCDE85D4E84A9888D8|125|1
35|0|1|1992|125|1|9|10183|9EF534D2CF74B24AC28CBD9BE937A412|SEMANAL|375CCE505F5353CCDE85D4E84A9888D8|125|1
36|0|1|1992|125|1|10|10183|9EF534D2CF74B24AC28CBD9BE937A412|SEMANAL|375CCE505F5353CCDE85D4E84A9888D8|125|1
37|0|1|1992|125|1|11|10183|9EF534D2CF74B24AC28CBD9BE937A412|SEMANAL|375CCE505F5353CCDE85D4E84A9888D8|125|1
38|0|1|1992|125|1|12|10183|9EF534D2CF74B24AC28CBD9BE937A412|SEMANAL|375CCE505F5353CCDE85D4E84A9888D8|125|1
39|0|1|1992|125|1|13|10183|9EF534D2CF74B24AC28CBD9BE937A412|SEMANAL|375CCE505F5353CCDE85D4E84A9888D8|125|1
40|0|1|1992|125|1|14|10183|9EF534D2CF74B24AC28CBD9BE937A412|SEMANAL|375CCE505F5353CCDE85D4E84A9888D8|125|1
41|0|1|1992|125|1|15|10183|9EF534D2CF74B24AC28CBD9BE937A412|SEMANAL|375CCE505F5353CCDE85D4E84A9888D8|125|1
10|0|1|1996|126|1|1||||||
You are allocating nine chunks of memory for each line, so you are allocating a total of 1350 million pieces of memory. These allocations have a certain overhead, usually at least twice the size of a pointer, possibly even more. On a 64 bit machine, that is already 16 bytes, so you get 21.6 GB of overhead.
In addition to that, you get the overhead of heap fragmentation and alignment: Even if you only ever store a string in it, the allocator has to align the memory allocations so that you can store the largest possible values in it without triggering misalignment. Alignment may depend on the vector unit of your CPU, which can require very significant alignments, 16 byte alignment not being uncommon.
Doing the calculation with 16 bytes allocation overhead and 16 bytes alignment, we get allocations of 43.2 GB without the original data. With the original data this calculation is already very close to your measurement.
Each of those objects and strings you create has individual memory management overhead. So you load the string "0" from column 2, depending on your memory manager, it probably takes between two and four full words (could be more). Call it 16 to 32 bytes of storage to hold a one byte string. Then you load the "1" from column 3. And so on.
i'm recording sound and encoding to mp3 with ffmpeg lib. then decode the mp3 data right away, play the decode data, but it's sounds so delayed.
here are the codes:
the function encode first parameter accepts the raw pcm data, len = 44100.
encode parameters:
cntx_->channels = 1;
cntx_->sample_rate = 44100;
cntx_->sample_fmt = 6;
cntx_->channel_layout = AV_CH_LAYOUT_MONO;
cntx_->bit_rate = 8000;
err_ = avcodec_open2(cntx_, codec_, NULL);
vector<unsigned char> encode(unsigned char* encode_data, unsigned int len)
{
vector<unsigned char> ret;
AVPacket avpkt;
av_init_packet(&avpkt);
unsigned int len_encoded = 0;
int data_left = len / 2;
int miss_c = 0, i = 0;
while (data_left > 0)
{
int sz = data_left > cntx_->frame_size ? cntx_->frame_size : data_left;
mp3_frame_->nb_samples = sz;
mp3_frame_->format = cntx_->sample_fmt;
mp3_frame_->channel_layout = cntx_->channel_layout;
int needed_size = av_samples_get_buffer_size(NULL, 1,
mp3_frame_->nb_samples, cntx_->sample_fmt, 1);
int r = avcodec_fill_audio_frame(mp3_frame_, 1, cntx_->sample_fmt, encode_data + len_encoded, needed_size, 0);
int gotted = -1;
r = avcodec_encode_audio2(cntx_, &avpkt, mp3_frame_, &gotted);
if (gotted){
i++;
ret.insert(ret.end(), avpkt.data, avpkt.data + avpkt.size);
}
else if (gotted == 0){
miss_c++;
}
len_encoded += needed_size;
data_left -= sz;
av_free_packet(&avpkt);
}
return ret;
}
std::vector<unsigned char> decode(unsigned char* data, unsigned int len)
{
std::vector<unsigned char> ret;
AVPacket avpkt;
av_init_packet(&avpkt);
avpkt.data = data;
avpkt.size = len;
AVFrame* pframe = av_frame_alloc();
while (avpkt.size > 0){
int goted = -1;av_frame_unref(pframe);
int used = avcodec_decode_audio4(cntx_, pframe, &goted, &avpkt);
if (goted){
ret.insert(ret.end(), pframe->data[0], pframe->data[0] + pframe->linesize[0]);
avpkt.data += used;
avpkt.size -= used;
avpkt.dts = avpkt.pts = AV_NOPTS_VALUE;
}
else if (goted == 0){
avpkt.data += used;
avpkt.size -= used;
avpkt.dts = avpkt.pts = AV_NOPTS_VALUE;
}
else if(goted < 0){
break;
}
}
av_frame_free(&pframe);
return ret;
}
Suppose it's the 100th call to encode(data, len), this "frame" would appear in 150th or later in the decode call, the latency is not acceptable. It seems the mp3lame encoder would keep the sample data for later use, but not my desire.
I don't know what is going wrong. Thank you for any information.
today i debug the code again and post some detail:
encode: each pcm sample frame len = 23040 ,which is 10 times of mp3 frame size, each time call encode only output 9 frames, this output cause decode output 20736 samples, 1 frame(2304 bytes) is lost, and the sound is noisy.
if the mp3 or mp2 encode is not suitable for real time voice transfer, which encoder should i choose?
Suppose it's the 100th call to encode(data, len), this "frame" would appear in 150th or later in the decode call, the latency is not acceptable.
Understand how the codec works and adjust your expectations accordingly.
MP3 is a lossy codec. It works by converting your time domain PCM data to the frequency domain. This conversion alone requires time (because frequency components do not exist in any instant of time... they can only exist over a period of time). At a simple level, it then uses a handful of algorithms to determine what spectral information to keep, and what to throw away. Each MP3 frame is hundreds of samples long in duration. (576 is as low as you can typically go. Twice that is a typical number.)
Now that you have the minimum time for creating frames, MP3 also uses what is called a bit reservoir. If a complex passage requires more bandwidth, it borrows unused bandwidth from neighboring frames. To facilitate this, a buffer of many frames is required.
On top of all the codec work, FFmpeg itself has buffering (for detection of input and what not), and there are buffers in your pipes to and from FFmpeg. I would imagine the codec itself may also employ general buffering on the input and output.
Finally, you're decoding the stream and playing it back, which means that most of the same kinds of buffers used in encoding are now used for decoding. And, we're not even talking about the several hundred milliseconds of latency you have for getting the audio data through a sound card and out the analog to your speaker.
You have an unrealistic expectation, and while it is possible to tweak some things to reduce latency (such as disabling the bit reservoir), it will result in a poor quality stream and will not be of low latency anyway.
I have an information retrieval and storage course project which for the first part I have to find the optimum buffer size for reading big files from the hard disk. our t.a says with increasing the buffer size up to a certain point (usually 4 bytes) the reading speed will increase but after that it decreases. but with my code below, it just increases no matter the buffer size or the file size (I have tested it on 100 mb). from what I know buffering only makes sense in parallel asynchronous processes (like threads) and that expectation for the buffer size-reading speed curve should hold true when the file is defragmented and\or the cost of looking up file directory and addresses(for the disk) is significant enough, so is the problem related to my code or the way ifstream handles things or maybe those conditions just don't hold up here?
ifstream in("D:ISR\\Articles.dat", std::ifstream::binary);
if(in)
{
in.seekg(0, in.end);
int length = in.tellg();
length = 100 * 1024 * 1024;
int bufferSize = 2;
int blockSize = 1024;//1kB
int numberOfBlocks = length / blockSize;
if(length % blockSize > 0) numberOfBlocks++;
clock_t t;
double time;
for(int i = 0; i < 5; i++)
{
in.seekg(0, in.beg);
int position = 0;
int bufferPosition;
char* streamBuffer = new char[bufferSize];
in.rdbuf()->pubsetbuf(streamBuffer, bufferSize);
t = clock();
for(int i = 0; i < numberOfBlocks; i++)
{
char* buffer = new char[blockSize];
bufferPosition = 0;
while(bufferPosition < blockSize && position < length)
{
in.read(buffer + bufferPosition, bufferSize);
position += bufferSize;
bufferPosition += bufferSize;
}
delete[] buffer;
}
t = clock() - t;
time = double(t) / CLOCKS_PER_SEC;
cout << "Buffer size : " << bufferSize << " -> Total time in seconds : " << time << "\n";
bufferSize *= 2;
}
what I know buffering only makes sense in parallel asynchronous
processes
No! No! Buffering make sense in many situations. A common situation is I/O. If you increase the size of read/write buffer. Operating system can touch the I/O device less.
And it can read/write larger blocks in each operation. Then, the performance gets better.
Choose buffer size in 2^n: 128, 512, 1024,... otherwise it can decrease the performance.
it just increases no matter the buffer size or the file size
The above statement does not hold true. Since you measure your program repeatedly, the successive result will be better than the previous ones due to the benefits of system cache. In fact, you access the file content from system cache instead of hard disk. BUT after the buffer size overs a threshold, the reading performance WILL decrease. Thanks to Richard Steven's chapter 3 in APUE 2nd, you can find the detailed and extensive experiments of reading & writing buffers.
I have a program that generates files containing random distributions of the character A - Z. I have written a method that reads these files (and counts each character) using fread with different buffer sizes in an attempt to determine the optimal block size for reads. Here is the method:
int get_histogram(FILE * fp, long *hist, int block_size, long *milliseconds, long *filelen)
{
char *buffer = new char[block_size];
bzero(buffer, block_size);
struct timeb t;
ftime(&t);
long start_in_ms = t.time * 1000 + t.millitm;
size_t bytes_read = 0;
while (!feof(fp))
{
bytes_read += fread(buffer, 1, block_size, fp);
if (ferror (fp))
{
return -1;
}
int i;
for (i = 0; i < block_size; i++)
{
int j;
for (j = 0; j < 26; j++)
{
if (buffer[i] == 'A' + j)
{
hist[j]++;
}
}
}
}
ftime(&t);
long end_in_ms = t.time * 1000 + t.millitm;
*milliseconds = end_in_ms - start_in_ms;
*filelen = bytes_read;
return 0;
}
However, when I plot bytes/second vs. block size (buffer size) using block sizes of 2 - 2^20, I get an optimal block size of 4 bytes -- which just can't be correct. Something must be wrong with my code but I can't find it.
Any advice is appreciated.
Regards.
EDIT:
The point of this exercise is to demonstrate the optimal buffer size by recording the read times (plus computation time) for different buffer sizes. The file pointer is opened and closed by the calling code.
There are many bugs in this code:
It uses new[], which is C++.
It doesn't free the allocated memory.
It always loops over block_size bytes of input, not bytes_read as returned by fread().
Also, the actual histogram code is rather inefficient, since it seems to loop over each character to determine which character it is.
UPDATE: Removed claim that using feof() before I/O is wrong, since that wasn't true. Thanks to Eric for pointing this out in a comment.
You're not stating what platform you're running this on, and what compile time parameters you use.
Of course, the fread() involves some overhead, leaving user mode and returning. On the other hand, instead of setting the hist[] information directly, you're looping through the alphabet. This is unnecessary and, without optimization, causes some overhead per byte.
I'd re-test this with hist[j-26]++ or something similar.
Typically, the best timing would be achieved if your buffer size equals the system's buffer size for the given media.