I'm recently working on ID3v2.4.0.
Reading 2.4.0 document, i found a particular part that i can't understand - sync-safe integer.
Why does the ID3v2 use this method?
Of course, i know why the ID3v2 uses Unsynchronization scheme, which is used to keep MPEG decoder from considering ID3 tag as a MPEG sync data.
But what i couldn't understand is that why sync-safe integer instead of Unsynchronization Scheme (= inserting $00).
Is there any reason why they adopt sync-safe integer when expressing tag size instead of inserting $00?
These two method result in completely same effect.
ID3v2 document says that the size of unsynchronized data is not known in advance.
But that statement does not make sense.
If tag data is stored in buffer, one can know the size of unsynchronized data after simply replacing the problematic character with $FF 00.
Is there anyone who can help me?
I would presume for simplicity, and the unsynch/synch scheme only makes sense when used on an mpeg file.
It is trivial to read in the four bytes and convert them to a regular integer:
// pseudo code
uint32_t size;
file.read( &size, sizeof(uint32_t) );
size = (size & 0x0000007F) |
( (size & 0x00007F00) >> 1 ) |
( (size & 0x007F0000) >> 2 ) |
( (size & 0x7F000000) >> 3 );
If they used the same unsynch scheme as frame data you would need to read each byte separately, look for the FF00 pattern, and reconstruct the integer byte by byte. Also, if the ‘size’ field in the header could be a variable number of bytes, due to unsynch bytes being inserted, the entire header would be a variable number of bytes. Simpler for them to say 'the header is always 10 bytes in size and it looks like this...'.
ID3v2 document says that the size of unsynchronized data is not known in advance. But that statement does not make sense. If tag data is stored in buffer, one can know the size of unsynchronized data after simply replacing the problematic character with $FF 00.
You are correct, it doesn't make sense. The size written in the id3v2 header and frame headers is the size after unsynchronisation, if any, was applied. However, it is permissible to write frame data without unsynching as id3v2 may be used for tagging files other than mp3, where the concept of unsynch/synch makes no sense. I think what section 6.2 was trying to say is 'regardless of whether this is an mp3 file, or a frame is written unsynched/synched, the frame size is always written in a mpeg synch-safe manner'.
ID3v2.4 frames can have the ‘Data Length Indicator’ flag set in the frame header, in which case you can find out how big a buffer is after synchronisation. Refer to section 4.1.2 of the spec.
Is there anyone who can help me?
Some helpful advice from someone who has written a conforming id3v2 tag reader: Don't try make sense of the spec. It surely was written by madmen and sadists. Just looking at it again is giving me nightmares.
Related
I have tried to make a simple wav writer. I wanted to do this so that I could read in a wav file (using a pre-existing wav reader), resample the audio data then write the resampled data to another wav file. Input files could be 16 bitsPerSample or 32 bitsPerSample and I wanted to save the resampled audio with the same number of bitsPerSample.
The writer is working but there a couple of things I don't understand to do with endianness and I was hoping someone may be able to help me?
I previously had no experience of reading or writing binary files. I began by looking up the wav file format online and tried to write the data following the correct format. At first the writing wasn't working but I then found out that wav files are little-endian and it was trying to make my file writer consistent with this that brought up the majority of my problems.
I have got the wav writer to work now (by way of a test whereby I read in a wav file and checked I could write the unsampled audio and reproduce the exact same file) however there are a couple of points I am still unsure on to do with endianness and I was hoping someone may be able to help me?
Assuming the relevant variables have already been set here is my code for the wav writer:
// Write RIFF header
out_stream.write(chunkID.c_str(),4);
out_stream.write((char*)&chunkSize,4);
out_stream.write(format.c_str());
// Write format chunk
out_stream.write(subchunk1ID.c_str(),4);
out_stream.write((char*)&subchunk1Size,4);
out_stream.write((char*)&audioFormat,2);
out_stream.write((char*)&numOfChannels,2);
out_stream.write((char*)&sampleRate,4);
out_stream.write((char*)&byteRate,4);
out_stream.write((char*)&blockAlign,2);
out_stream.write((char*)&bitsPerSample,2);
// Write data chunk
out_stream.write(subchunk2ID.c_str(),4);
out_stream.write((char*)&subchunk2Size,4);
// Variables for writing 16 bitsPerSample data
std::vector<short> soundDataShort;
soundDataShort.resize(numSamples);
char theSoundDataBytes [2];
// soundData samples are written as shorts if bitsPerSample=16 and floats if bitsPerSample=32
switch( bitsPerSample )
{
case (16):
// cast each of the soundData samples from floats to shorts
// then save the samples in little-endian form (requires reversal of byte-order of the short variable)
for (int sample=0; sample < numSamples; sample++)
{
soundDataShort[sample] = static_cast<short>(soundData[sample]);
theSoundDataBytes[0] = (soundDataShort[sample]) & 0xFF;
theSoundDataBytes[1] = (soundDataShort[sample] >> 8) & 0xFF;
out_stream.write(theSoundDataBytes,2);
}
break;
case (32):
// save the soundData samples in binary form (does not require change to byte order for floats)
out_stream.write((char*)&soundData[0],numSamples);
}
The questions that I have are:
In the soundData vector why does the endianness of a vector of shorts matter but the vector of floats doesn't? In my code I have reversed the byte order of the shorts but not the floats.
Originally I tried to write the shorts without reversing the byte order. When I wrote the file it ended up being half the size it should have been (i.e. half the audio data was missing, but the half that was there sounded correct), why would this be?
I have not reversed the byte order of the shorts and longs in the other single variables which are essentially all the other fields that make up the wav file e.g. sampleRate, numOfChannels etc but this does not seem to affect the playing of the wav file. Is this just because media players do not use these fields (and hence I can't tell that I have got them wrong) or is it because the byte order of these variables does not matter?
In the soundData vector why does the endianness of a vector of shorts matter but the vector of floats doesn't? In my code I have reversed the byte order of the shorts but not the floats.
Actually, if you take a closer look at your code, you will see that you are not reversing the endianness of your shorts at all. Nor do you need to, on Intel CPUs (or on any other low-endian CPU).
Originally I tried to write the shorts without reversing the byte order. When I wrote the file it ended up being half the size it should have been (i.e. half the audio data was missing, but the half that was there sounded correct), why would this be?
I have no idea without seeing the code but I suspect that some other factor was in play.
I have not reversed the byte order of the shorts and longs in the other single variables which are essentially all the other fields that make up the wav file e.g. sampleRate, numOfChannels etc but this does not seem to affect the playing of the wav file. Is this just because media players do not use these fields (and hence I can't tell that I have got them wrong) or is it because the byte order of these variables does not matter?
These fields are in fact very important and must also be little-endian, but, as we have seen, you don't need to swap those either.
before = new unsigned char[mSizeNeeded*4];
uLong value = compressBound(mSizeNeeded*4);
after = new unsigned char[value];
compress(after, &value, before, mSizeNeeded*4);
fwrite(&after, 1, value, file);
'before' has a bunch of audio data stored into it and I am trying to compress it and store it into 'after'. I then write it into a file. The file is the same size as the original file, it also contains the same data that was in before (as far as I can tell).
Compress also returns OK so I know that the compression is not failing.
Okay, so it looks like my only problem is somewhere in the compression (I think). I am able to run compress and then I can uncompress and get the correct data out. Also, it is writing into the file and fwrite returns 561152 but the count (value) is 684964. So it looks like something is wrong with fwrite. I looked more carefully and the after data is different than the before data.
561152 is the same size as the original audio data in a .wav file that I have (stripped of the .wav headers of course).
Based on your original text:
fwrite (&before, ...
I am trying to compress it and store it into 'after'. I then write it into a file.
I think not. You are writing the original data to the file, you should probably be writing after instead.
The other thing you should get in the habit of doing is checking return values from functions that you care about. In other words, compress() will tell you if a problem occurs yet you seem to be totally ignoring the possibility.
Similarly, fwrite() also uses its return value to indicate whether it was successful or not. Since you haven't included the code showing how that's set up, this is also a distinct possibility. In particular fwrite is under no obligation to write your entire block to the file in one hit (device may be full, etc), that's why it has a return value, so you can detect and adjust for that situation. Often, a better option than:
fwrite (&after, 1, value, file);
is:
fwrite (&after, value, 1, file);
since the latter will always give you one for a fully successful write, something else for a failure of some description.
That would be my first step in establishing where the problem lies.
On top of that, there are numerous other (generally-applicable) methods you can use to track down the issue, such as:
outputting all variables after they change or are set (like the return values of functions, after, before, value and so on).
delete the output file before running your program, to ensure it's created afresh.
run the code through a debugger so you can see what's happening under the covers.
clearing after to all zero bytes (or a known pattern) to ensure you don't get stale data in there.
And, as a final approach (given that the zlib source code is freely available), you can also modify (or debug into) it so that you can clearly see what's going on under the covers.
I am trying to debug a problem with some code that uses zlib 1.2.8. The problem is that this larger project can make archives, but runs into Z_DATA_ERROR header problems when trying to extract that archive.
To do this, I wrote a small test program in C++ that compresses ("deflates") a specified regular file, writes the compressed data to a second regular file, and extracts ("inflates") to a third regular file, one line at a time. I then diff the first and third files to make sure I get the same bytes.
For reference, this test project is located at: https://github.com/alexpreynolds/zlib-test and compiles under Clang (and should also compile under GNU GCC).
My larger question is how to deal with header data correctly in my larger project.
In my first test scenario, I can set up compression machinery with the following code:
z_error = deflateInit(this->z_stream_ptr, ZLIB_TEST_COMPRESSION_LEVEL);
Here, ZLIB_TEST_COMPRESSION_LEVEL is 1, to provide best speed. I then run deflate() on the z_stream pointer until there is nothing left that comes out of compression.
To extract these bytes, I can use inflateInit():
int ret = inflateInit(this->z_stream_ptr);
So what is the header format, in this case?
In my second test scenario, I set up the deflate machinery like so:
z_error = deflateInit2(this->z_stream_ptr,
ZLIB_TEST_COMPRESSION_LEVEL,
ZLIB_TEST_COMPRESSION_METHOD,
ZLIB_TEST_COMPRESSION_WINDOW_BITS,
ZLIB_TEST_COMPRESSION_MEM_LEVEL,
ZLIB_TEST_COMPRESSION_STRATEGY);
These deflate constants are, respectively, 1 for level, Z_DEFLATED for method, 15+16 or 31 for window bits, 8 for memory level, and Z_DEFAULT_STRATEGY for strategy.
The former inflateInit() call does not work; instead, I must use inflateInit2() and specify a modified window bits value:
int ret = inflateInit2(this->z_stream_ptr, ZLIB_TEST_COMPRESSION_WINDOW_BITS + 16);
In this case, the window bits value is not 31 as in the deflateInit2() call, but 15+32 or 47.
If I use 31 (or any other value than 47), then I get a Z_DATA_ERROR on subsequent inflate() calls. That is, if I use the same window bits for the inflateInit2() call:
int ret = inflateInit2(this->z_stream_ptr, ZLIB_TEST_COMPRESSION_WINDOW_BITS);
Then I get the following error on attempting to inflate():
Error: inflate to stream failed [-3]
Here, -3 is the same as Z_DATA_ERROR.
According to the documentation, using 31 with deflateInit2() should write a gzip header and trailer. Thus, 31 on the following inflateInit2() call should be expected to be able to extract the header information.
Why is the modified value 47 working, but not 31?
My test project is mostly similar to the example code on the zlib site, with the exception of the extraction/inflation code, which inflates one z_stream chunk at a time and parses the output for newline characters.
Is there something special about running inflate() only when a new buffer of extracted data is asked for — like header information going missing between inflate() calls — as opposed to running the whole extraction in one pass, as in the zlib example code?
My larger debugging problem is looking for a robust way to extract a chunk of zlib-compressed data only on request, so that I can extract data one line at a time, as opposed to getting the whole extracted file. Something about the way I am handling the zlib format parameter seems to be messing me up, but I can't figure out why or how to fix this.
deflateInit() and inflateInit(), as well as deflateInit2() and inflateInit2() with windowBits in 0..15 all process zlib-wrapped deflate data. (See RFC 1950 and RFC 1951.)
deflateInit2() and inflateInit2() with negative windowBits in -1..-15 process raw deflate data with no header or trailer. deflateInit2() and inflateInit2() with windowBits in 16..31, i.e. 16 added to 0..15, process gzip-wrapped deflate data (RFC 1952). inflateInit2() with windowBits in 32..47 (32 added to 0..15) will automatically detect either a gzip or zlib header (but not raw deflate data), and decompress accordingly.
Why is the modified value 47 working, but not 31?
31 does work. I did not try to look at your code to debug it.
Is there something special about running inflate() only when a new
buffer of extracted data is asked for — like header information going
missing between inflate() calls — as opposed to running the whole
extraction in one pass, as in the zlib example code?
I can't figure out what you're asking here. Perhaps a more explicit example would help. The whole point of inflate() is to decompress a chunk at a time.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
I am little bit confused on binary files, i know the datas are stored in chunks in binary files, and from my knowledge through experimenting i found that if we had a struct with member variables like this:
struct student{
int Roll_No;
char Name[10];
}
Then after updating the variables with contents, and saving it in a binary file the binary file is of 14 bytes, 10 bytes of char and 4 of int, so if we analyze the file in a hexeditor the file has 4 bytes reserved for Roll_no and 10 bytes reserved for Name in which the filled contents are filled and others can be seen as dots in the file, i mean if we create a program with struct/class like above, and after saving contents to the file, the file's size is the same as we created structure, i mean 4 of int and 10 of char, so from my knowledge if i created a new image format eg. (Dot).MyIMG, from my program which's stucture/class is like this
struct MyIMG{
char Header[5];
int width, height;
int Pixels[124000];
}
Then my program will create a new file of size 49613 bytes or 49 Kigabytes (which is 5 of header, +(plus) 8 of int height and width, +(plus) 4×124000 of int pixels), wether the pixels are 4, 8, 100, or whatever it will write the whole Pixels array wether empty, so why this effect cannot be same on any large softwares like MSpaint, Adobe photoshop, what do they do, which make their program to write files which's size depends on the pixels stored inn not the blank arrays...
EDIT: I have now edited my question, and clearly defined my question, pls help me, thanks in advance!!
File formats like .png and .bmp have a specific format. File formats can either specify a layout of bytes (such as 4 bytes for the width, 4 bytes for the height, 2MB of RGBA pixel data, or whatever), or the format may give you information about the size of various objects.
For example, a TIFF file will specify that there are a number tags at specific byte offsets within the file. Those tags then contain information about the size, location, and format of the image data. So you might have a fixed-sized header that says "there is a list of tags starting at byte 100, and it contains 40 tags." The tags would each be a fixed size (say 16-bytes), so you'd know to read 40 16-byte chunks starting at byte 100. The tags would then contain information such as the byte offset of the start of the image data, how many bytes are in a pixel, and how many pixels there are. From this, you can read the data without knowing ahead of time what the entire format is.
The code writing the file has to choose it's own format. For example, when writing your student structure to file, you could say something like:
size_t name_len = strlen(my_student.Name);
my_ofstream.write((const char*)&my_student, sizeof my_student - sizeof my_student.Name + name_len + 1);
This would then write the name up to and including the first 0/NUL character to the binary file. When reading the file back, the program could read a block of data from the ifstream then - knowing that a student is stored at some offset, use strlen() on the .Name part of the incoming data to recover the length, partly so it can only copy the necessary data to a student object, and also know where to start parsing the next data item from the input stream:
char buffer[32768];
student my_student;
if (my_ifstream.read(buffer, sizeof buffer) && my_ifstream.gcount() > 5)
{
// check for NUL without risking reading buffer[.gcount()]
size_t pre_name_len = std::offsetof(student, name);
const char* p_name = buffer + pre_name_len;
const char* p_nul = strnchr(p_name,
std::min(10, my_ifstream.gcount() - pre_name_len),
'\0');
if (p_nul == nullptr || *p_nul != '\0')
throw std::runtime_error("file didn't contain complete student record");
memcpy(my_student, buffer, p_nul - buffer + 1);
// keep parsing input from p_nul + 1, not going past .gcount()
}
As you can see - it's a bit of a pain to scan for the single NUL while keeping track of the amount of data read from the file so you don't crash if you get a corrupt input file....
For a beginner, it's probably easiest and far more robust to learn about the boost serialisation library which abstracts much of the low level - some would say C-style - I/O, casting and offset calculations to provide a cleaner logical interface for you.
Hello I am trying to simulate two programs that send and receive files in C++ from the network, something like client and server. To begin with I have to split a file to pages of 4096 bytes and send it to the other program in order to create the file. The way I send and receive files through the network is by write and read. So in the client programm I must create a function tha receives the packages and puts them into a file. I cannot figure a way to put the packages in to the file. For example I a file has 2 pages I must create another file using these 2 pages. Also i cannot know if they come in order so I must create the file and put them in the right position.
/*consider the connections are ok and the file's name is at char* name*/
int file=open(name,"O_CREAT | O_WRONLY,0666);
char buffer[4096];
int pagenumber;
for(int i=0;i<page_number;i++){
read(socket,&pagenumber,sizeof(int));
read(socket,buffer,sizeof(int));
write(file(pagenumber*4096),buffer,4096);
}
This code works for pagenumber=0 but for pagenumber=1 nothing happens! Can you help me? Thanks in advance!
To write at a certain position in the file you must use lseek
off_t lseek(int fd, off_t offset, int whence);
It takes the descriptor, the offset and the final parameter is a constant in these:
SEEK_SET The offset is set to offset bytes.
SEEK_CUR The offset is set to its current location plus offset bytes.
SEEK_END The offset is set to the size of the file plus offset bytes.
If you know how big is the file going to be, you can use ftruncate for it.
int ftruncate(int fd, off_t length);
Anyway even if you create a file that is huge, since most filesystems on Linux support sparse files, the actual file on disk will be the sum of the blocks that have been written.
The first argument to write() is a filedescriptor, which you optained with open(). So it should be
int file = open(...);
...
write(file,buffer,4096);
not
write(file(pagenumber*4096),buffer,4096);
Regarding the question as to how to write at a specific position. You can prepare the file beforehand with write, and then use seek() to position the file where you want to write at. For a description of seek you can look here.
Mario, first of all, lets no rely on garbage in 'pagenumber' to continue the loop (which is happening when loop boundary condition is checked here for the first time). Now, if you are writing page number '0' and then page following it, pagenumber will be initialized to 0 and your loop will come out. Also, please check bytes written and read in write and read system calls respectively.
try pwrite
int file=open(name,"O_CREAT | O_WRONLY,0666);
char buffer[4096];
int pagenumber;
for(int i=0;i<page_number;i++){
read(socket,&pagenumber,sizeof(int));
read(socket,buffer,sizeof(int));
pwrite(file,buffer,4096,4096*i);
}