SHCreateStreamOnFileEx on files larger than 2**32 bytes - c++

Question

I'm getting an IStream for a file using SHCreateStreamOnFileEx, but its Read() method appears to misbehave on extremely large files when the new position of the seek pointer is 2 ** 32 bytes or further into the file.
ISequentialStream::Read's documentation says:
This method adjusts the seek pointer by the actual number of bytes read.
This is the same behaviour as read(2) and fread(3) on all platforms I'm aware of.
But with these streams, this isn't the actual behaviour I see in some cases:
Seek(2 ** 32 - 2, SEEK_SET, &pos), Read(buf, 1, &bytesRead), Seek(0, MOVE_CUR, &pos) → bytesRead == 1 and pos == 2 ** 32 - 1, as expected.
Seek(2 ** 32 - 1, SEEK_SET, &pos), Read(buf, 1, &bytesRead), Seek(0, MOVE_CUR, &pos) → bytesRead == 1, but pos == (2 ** 32 - 1) + 4096, which is incorrect. This means that any subsequent reads (without another Seek to fix the cursor position) read the wrong data, and my application doesn't work!
Am I “holding it wrong”? Is there some flag I need to set to make this class behave properly? Or is this a bug in Shlwapi.dll?
The code below reproduces this problem for me. (Set OFFSET = WORKS to see the successful case.)
#include "stdafx.h"
static const int64_t TWO_THIRTY_TWO = 4294967296LL;
static const int64_t WORKS = TWO_THIRTY_TWO - 2LL;
static const int64_t FAILS = TWO_THIRTY_TWO - 1LL;
static const int64_t OFFSET = FAILS;
static void checkPosition(CComPtr< IStream > fileStream, ULONGLONG expectedPosition)
{
LARGE_INTEGER move;
ULARGE_INTEGER newPosition;
move.QuadPart = 0;
HRESULT hr = fileStream->Seek(move, SEEK_CUR, &newPosition);
ASSERT(SUCCEEDED(hr));
ULONGLONG error = newPosition.QuadPart - expectedPosition;
ASSERT(error == 0);
}
int main()
{
const wchar_t *path = /* path to a file larger than 2**32 bytes */ L"C:\\users\\wjt\\Desktop\\eos-eos3.1-amd64-amd64.170216-122002.base.img";
CComPtr< IStream > fileStream;
HRESULT hr;
hr = SHCreateStreamOnFileEx(path, STGM_READ, FILE_ATTRIBUTE_NORMAL, FALSE, NULL, &fileStream);
ASSERT(SUCCEEDED(hr));
LARGE_INTEGER move;
ULARGE_INTEGER newPosition;
// Advance
move.QuadPart = OFFSET;
hr = fileStream->Seek(move, SEEK_SET, &newPosition);
ASSERT(SUCCEEDED(hr));
ASSERT(newPosition.QuadPart == OFFSET);
// Check position
checkPosition(fileStream, OFFSET);
// Read
char buf[1];
ULONG bytesRead = 0;
hr = fileStream->Read(buf, 1, &bytesRead);
ASSERT(SUCCEEDED(hr));
ASSERT(bytesRead == 1);
// Check position: this assertion fails if the Read() call moves the cursor
// across the 2**32 byte boundary
checkPosition(fileStream, OFFSET + 1);
return 0;
}

Answer 1

this is really windows bug. tested on several windows version including latest SHCore.DLL version 10.0.14393.0 x64. simple way for reproduce:
void BugDemo(PCWSTR path)
{
// FILE_FLAG_DELETE_ON_CLOSE !
HANDLE hFile = CreateFile(path, FILE_GENERIC_WRITE, FILE_SHARE_READ|FILE_SHARE_DELETE, 0,
CREATE_NEW, FILE_ATTRIBUTE_TEMPORARY|FILE_FLAG_DELETE_ON_CLOSE, 0);
if (hFile != INVALID_HANDLE_VALUE)
{
ULONG dwBytesRet;
// i not want really take disk space
if (DeviceIoControl(hFile, FSCTL_SET_SPARSE, NULL, 0, NULL, 0, &dwBytesRet, NULL))
{
static FILE_END_OF_FILE_INFO eof = { 0, 2 };// 8GB
if (SetFileInformationByHandle(hFile, FileEndOfFileInfo, &eof, sizeof(eof)))
{
IStream* pstm;
if (!SHCreateStreamOnFileEx(path, STGM_READ|STGM_SHARE_DENY_NONE, 0,FALSE, NULL, &pstm))
{
LARGE_INTEGER pos = { 0xffffffff };
ULARGE_INTEGER newpos;
if (!pstm->Seek(pos, STREAM_SEEK_SET, &newpos) && !pstm->Read(&newpos, 1, &dwBytesRet))
{
pos.QuadPart = 0;
if (!pstm->Seek(pos, STREAM_SEEK_CUR, &newpos))
{
DbgPrint("newpos={%I64x}\n", newpos.QuadPart);//newpos={100000fff}
}
}
pstm->Release();
}
}
}
// close and delete
CloseHandle(hFile);
}
}
void BugDemo()
{
WCHAR path[MAX_PATH];
if (ULONG len = GetTempPath(RTL_NUMBER_OF(path), path))
{
if (len + 16 < MAX_PATH)
{
FILETIME ft;
GetSystemTimeAsFileTime(&ft);
swprintf(path + len, L"%08x%08x", ~ft.dwLowDateTime, ft.dwHighDateTime);
BugDemo(path);
}
}
}
I trace virtual long CFileStream::Seek(LARGE_INTEGER, ULONG, ULARGE_INTEGER* ); under debugger and can confirm that this function not design to work with files more than 4GB size
if be more exactly, why is 100000FFF offset - CFileStream use internal buffer for read 1000 byte size. when you ask read 1 byte from FFFFFFFF offset - it actually read 1000 bytes to the buffer and file offset become 100000FFF. when you then call Seek(0, STREAM_SEEK_CUR, &newpos) - CFileStream call SetFilePointer(hFile, 1-1000, 0/*lpDistanceToMoveHigh*/, FILE_CURRENT)
(1 this is internal position in buffer, because we read 1 byte minus buffer size 1000) . if not take to account overflow can be (100000FFF + (1 - 1000)) == 100000000 but
read about SetFilePointer
If lpDistanceToMoveHigh is NULL and the new file position does not fit
in a 32-bit value, the function fails and returns
INVALID_SET_FILE_POINTER.
as result SetFilePointer fail (return INVALID_SET_FILE_POINTER) but CFileStream even not check for this. and then it call SetFilePointerEx(hFile, 0, &newpos, FILE_CURRENT) and return to you newpos which still 100000FFF