Trouble reading data from Parquet File - c++

Question

I am attempting to read parquet data from a binary stream (via API posts). For example: I have a rather large parquet file on the other side of an REST API and need to fetch parts of the file. I have been attempting to follow file spec here: https://github.com/apache/parquet-format however, the pattern seems to be failing (or I am misunderstanding part).
For my test, I have moved a parquet file onto my local system and am reading in binary data from the file using ifstream. My steps are as follows:
Read in magic number from header
Read in magic number from footer
Read in FileMetaData length
Read in FileMetaData (from bottom of file)
Convert stream to FileMetaData Type using:
std::shared_ptr<parquet::FileMetaData> _metadata = parquet::FileMetaData::Make(metadataBuffer.data(), &metadataLength);
Read in RowGroup(0) and RowGroup(1) file_offset and total_byte_size from the FileMetaData like this:
_metadata->RowGroup(x)->file_offset();
_metadata->RowGroup(x)->total_byte_size();
After storing this data, I proceed to read in each RowGroup from the file using ifstream again. My start position is the file_offset from the beginning of the file.
Once my RowGroup data is read in to a vector of objects, I attempt to convert the buffered data into RowGroupMetaData
std::shared_ptr<parquet::RowGroupMetaData> _rowGroupMetaData = parquet::RowGroupMetaData::Make(rowGroupData[x].rowGroupBuffer.data(), rowGroupData[x].schema);
This is where I get stuck. When I try to access parts of the _rowGroupMetaData, I am getting junk back. It seems I must be skipping a step or overlooking part of the file spec.
I noticed that there is data between the magic number PAR1 at the top of the file an the file offset of RowGroup(0). the magic number is 4 characters long but the RowGroup(0) file_offset = 113. I am not sure what the data between 4-113 is and I cannot find information on it in the spec.
My parquet file is rather simple. 2 RowGroups with 2 columns. Total of 5 rows across both RowGroups.
Code:
ifstream inFile("parquet-arrow-example.parquet", std::ofstream::binary | std::ios::ate);
std::streamsize fileSize = inFile.tellg();
inFile.seekg(0, std::ios::beg);
std::vector<char> headBuffer;
std::vector<char> tailBuffer;
std::vector<uint8_t> metadataBuffer;
headBuffer.resize(4);
tailBuffer.resize(4);
struct RowGroupData {
int groupId;
int64_t byteLength;
int64_t offset;
const parquet::SchemaDescriptor* schema;
vector<uint8_t> rowGroupBuffer;
};
uint32_t metadataLength = 0;
string header;
string footer;
//Header
inFile.read((char*)&headBuffer[0], headBuffer.size()); //PAR1
header = string(headBuffer.begin(), headBuffer.end());
cout << header << endl;
//Footer
inFile.seekg(-4, std::ios::end);
inFile.read((char*)&tailBuffer[0], tailBuffer.size()); //PAR1
footer = string(tailBuffer.begin(), tailBuffer.end());
cout << footer << endl;
//Metadata Size
inFile.seekg(-8, std::ios::end);
inFile.read((char*)&metadataLength, 4);
cout << "Metadata Length: " << metadataLength << endl;
int len = -8 - metadataLength;
//Get MetaData
inFile.seekg(len, std::ios::end);
metadataBuffer.resize(metadataLength);
inFile.read((char*)&metadataBuffer[0], metadataBuffer.size());
cout << string(metadataBuffer.begin(), metadataBuffer.end()) << endl;
std::shared_ptr<parquet::FileMetaData> _metadata = parquet::FileMetaData::Make(metadataBuffer.data(), &metadataLength);
cout << "Num Rows: " << _metadata->num_rows() << endl;
cout << "Num Columns: " << _metadata->num_columns() << endl;
cout << "Num RowGroups: " << _metadata->num_row_groups() << endl;
vector<RowGroupData> rowGroupData;
//int seeqPos = 4;
for (int x = 0; x < _metadata->num_row_groups(); x++) {
cout << "RowGroup " << x << " Byte Size: " << _metadata->RowGroup(x)->total_byte_size() << endl;
cout << "RowGroup " << x << " File Offset: " << _metadata->RowGroup(x)->file_offset() << endl;
cout << "RowGroup " << x << " Column 0 File Offset: " << _metadata->RowGroup(x)->ColumnChunk(0)->file_offset() << endl;
cout << "RowGroup " << x << " Column 0 Byte Size: " << _metadata->RowGroup(x)->ColumnChunk(0)->total_compressed_size() << endl;
cout << "RowGroup " << x << " Column 1 File Offset: " << _metadata->RowGroup(x)->ColumnChunk(1)->file_offset() << endl;
cout << "RowGroup " << x << " Column 1 Byte Size: " << _metadata->RowGroup(x)->ColumnChunk(1)->total_compressed_size() << endl;
RowGroupData rgData;
rgData.groupId = x;
rgData.byteLength = _metadata->RowGroup(x)->total_byte_size();
rgData.offset = _metadata->RowGroup(x)->file_offset();
rgData.schema = _metadata->RowGroup(x)->schema();
rgData.rowGroupBuffer.resize(rgData.byteLength);
//Store rowGroup Length
//Store rowGroup Data
inFile.seekg(rgData.offset, std::ios::beg);
inFile.read((char*)&rgData.rowGroupBuffer[0], rgData.rowGroupBuffer.size());
rowGroupData.push_back(rgData);
//seeqPos = seeqPos + rgData.byteLength;
}
cout << endl;
for (int x = 0; x < rowGroupData.size(); x++) {
vector<uint8_t> rgBuffer;
//rgBuffer = rowGroupData[x].rowGroupBuffer;
cout << "RowGroupId: " << rowGroupData[x].groupId << endl;
cout << "RowGroupData: " << string(rowGroupData[x].rowGroupBuffer.begin(), rowGroupData[x].rowGroupBuffer.end()) << endl;
std::shared_ptr<parquet::RowGroupMetaData> _rowGroupMetaData = parquet::RowGroupMetaData::Make(rowGroupData[x].rowGroupBuffer.data(), rowGroupData[x].schema);
cout << "RowGroup Rows: " << _rowGroupMetaData->num_rows() << endl;
cout << "Byte Size: " << _rowGroupMetaData->total_byte_size() << endl;
}

Answer 1

The data between the file header and the file_offset is the column_chunk metadata for the first column.
The parquet spec is a little confusing because there are two different file offsets. The one on the RowGroup is an offset to the first page of data in the row group. And the column chunk file_offset which points to the column chunks metadata.
To my knowledge the first offset is mostly used for splitting files, I think most other readers use the latter offset for parsing columns.
Also note that in C++ at least file_offset was being written out incorrectly prior to the release of Arrow 6.0 (it pointed to the same byte offset as that the column offset chunk did).
Last, parquet is a non-trivial format and it is easy to have subtle bugs, I'd strongly recommend trying to use a standard implementation which has been battle tested rather then creating your own. If something is missing from the API it might be simpler to contribute it to an existing implementation instead of trying to build everything from scratch.