I'm creating and writing a file with CreateFile2 and WriteFile, then later using readfile with the to read 16 bytes at a time into an __m128i and then performing simd operations on it. Works fine in debug mode, but throws the access denied (0xc0000005) error code in release mode. In my experience, that happens when I'm trying to shove non 16-byte-aligned stuff into 16-byte-aligned stuff. However, I'm unsure where the lack of 16-byte-alignment is first rearing its ugly head.
#define simd __m128i
Is it in the CreateFile2() call?
_CREATEFILE2_EXTENDED_PARAMETERS extend = { 0 };
extend.dwSize = sizeof(CREATEFILE2_EXTENDED_PARAMETERS);
extend.dwFileAttributes = FILE_ATTRIBUTE_NORMAL;
extend.dwFileFlags = /*FILE_FLAG_NO_BUFFERING |*/ FILE_FLAG_OVERLAPPED;
extend.dwSecurityQosFlags = SECURITY_ANONYMOUS;
extend.lpSecurityAttributes = nullptr;
extend.hTemplateFile = nullptr;
hMappedFile = CreateFile2(
testFileName.c_str(),
GENERIC_READ | GENERIC_WRITE,
0,
OPEN_ALWAYS,
&extend);
...in the WriteFile() call?
_OVERLAPPED positionalData;
positionalData.Offset = 0;
positionalData.OffsetHigh = 0;
positionalData.hEvent = 0;
bool writeCheck = WriteFile(
hMappedFile,
&buffer[0],
vSize,
NULL,
&positionalData);
...in the later ReadFile() call?
const simd* FileNodePointer(
_In_ const uint32_t index) const throw()
{
std::vector<simd> Node(8);
_OVERLAPPED positionalData;
positionalData.Offset = index;
positionalData.OffsetHigh = 0;
positionalData.hEvent = 0;
ReadFile(
hMappedFile,
(LPVOID)&Node[0],
128,
NULL,
&positionalData);
return reinterpret_cast<const simd*>(&Node[0]);
}
How can I enforce 16-byte-alignment here?
Thanks!
TL;DR You have a classic "use after free" error.
None of these functions require 16 byte alignment. If buffering is enabled, they don't care about alignment at all, and if direct I/O is enabled, they require page alignment which is much more restrictive than 16 bytes.
If your data buffer is unaligned, it's because you created it that way. The file I/O is not moving your buffer in memory.
But your access violation is not caused by alignment problems at all, it is the dangling pointer you return from FileNodePointer:
return reinterpret_cast<const simd*>(&Node[0]);
That's a pointer into content of a vector with automatic lifetime, the vector destructor runs during the function return process and frees the memory containing the data you just read from the file.
Related
I need to read a large file in parts to a limited buffer. My code works, but always reads from the beginning. I think I need to use dwFileOffsetHigh somehow and dwFileOffsetLow, but I can't figure out how. Mapper_Winapi_Uptr is a unique_ptr with a custom deleter, if necessary I can lay out its code.
System: 64bit Win10.
const std::vector<BYTE>& ReadFile(size_t pos) {
memory = Mapper_Winapi_Uptr{ static_cast<BYTE*>(MapViewOfFile(mapping, FILE_MAP_READ, 0, 0, bufferSize)) };
std::memcpy(&data[0], memory.get(), bufferSize);
return data;
}
You are mapping the view at file offset 0, ignoring your pos parameter which is presumably the desired file offset.
MapViewOfFile() takes a 64bit offset as input, split into 32bit low and high values. A size_t may be a 32bit or 64bit type, depending on compiler and platform. You can put your desired offset into a ULARGE_INTEGER first, that will give you the low and high values you can then give to MapViewOfFile().
Note that the file offset you give to MapViewOfFile() must be a multiple of the system allocation granularity. See Creating a View Within a File on MSDN for details on how to handle that.
Try something like this:
SYSTEM_INFO SysInfo;
GetSystemInfo(&SysInfo);
DWORD SysGran = SysInfo.dwAllocationGranularity;
...
const std::vector<BYTE>& ReadFile(size_t pos)
{
size_t MapViewStart = (pos / SysGran) * SysGran;
DWORD MapViewSize = (pos % SysGran) + bufferSize;
DWORD ViewDelta = pos - MapViewStart;
ULARGE_INTEGER ulOffset;
ulOffset.QuadPart = MapViewStart;
memory = Mapper_Winapi_Uptr{ static_cast<BYTE*>(MapViewOfFile(mapping, FILE_MAP_READ, ulOffset.HighPart, ulOffset.LowPart, bufferSize)) };
if (!memory.get()) {
// error handling ...
}
std::memcpy(&data[0], &(memory.get())[ViewDelta], bufferSize);
return data;
}
__m128i* pData = reinterpret_cast<__m128i*>(
_aligned_malloc(
128,
16));
std::vector<byte> sectorBytes = dataSet.GetSectorBytes(index);
index &= 0xfff;
memcpy(pData, §orBytes[index], 128);
as you can see, I'm trying to copy 128 bytes from sectorBytes (which has no guarantees of 16-byte-alignment) into pData, which does. Unfortunately, after setting a breakpoint to check, I've found that while sectorBytes has exactly what I'd anticipated it to have, pData[0] through pData[7] contain only zeroes... so nothing is actually getting copied.
I don't understand why nothing's being copied... why is this happening?
Of course, the larger task I'm trying to accomplish is taking 128 bytes from a file at a specified offset, and then performing _mm_xor_si128() on them without an access violation popping up from trying to do simd operations on data that isn't 16-byte-aligned. In the interest of facilitating that discussion, here's GetSectorBytes():
std::vector<byte> GetSectorBytes(
_In_ const uint32_t index) const throw()
{
std::vector<byte> buffer(0x1000);
int indexOuter = index & 0xfffff000;
_OVERLAPPED positionalData;
positionalData.Offset = indexOuter;
positionalData.OffsetHigh = 0;
positionalData.hEvent = 0;
ReadFile(
hMappedFile,
&buffer[0],
0x1000,
NULL,
&positionalData);
return buffer;
}
hMappedFile was created with CreateFile2 with the FILE_FLAG_NO_BUFFERING flag set.
I wonder why this code doesn't work:
#include <iostream>
using namespace std;
int main()
{
int *pointer = (int*)0x02F70BCC;
cout<<*pointer;
return 0;
}
In my opinion it should write on the screen value of 0x02F70BCC,
instead of this my programm crashes.
I know that memory with adress 0x02F70BCC stores value of 20.
But like I said no matter what it just doesn't want to show correct number.
Please help me guys, detailed explanation would be very nice of you.
It doesn't work, because you won't get access to every location in memory you want. Not every location in memory is valid, you may want to read about Virtual Address Space.
Some addresses are reserved for device drivers and kernel mode operations. Another range of addresses (for example 0xCCCCCCCC and higher) may be reserved for uninitialized pointers.
Even if some location is valid, operating system may still deny access to write to/read from certain location, if that would cause undefined behaviour or violate system safety.
EDIT
I think you might be interested in creating some kind of "GameHack", that allows you to modify amount of resources, number of units, experience level, attributes or anything.
Memory access is not a simple topic. Different OSes use different strategies to prevent security violations. But many thing can be done here, after all there is a lot software for doing such things.
First of all, do you really need to write your own tool? If you just want some cheating, use ArtMoney - it is a great memory editor, that I have been using for years.
But if you really have to write it manually, you need to do some research first.
On Windows, for example, I would start from these:
ReadProcessMemory
WriteProcessMemory
Also, I am quite certain, that one of possible techniques is to pretend, that you are a debugger:
DebugActiveProcess.
EDIT 2
I have done some research and it looks, that on Windows (I assume this is your platform, since you mentioned gaming; can't imagine playing anything on crappy Linux), steps required to write another process' memory are:
1. Enumerate processes: (EnumProcesses)
const size_t MAX_PROC_NUM = 512;
DWORD procIDs[MAX_PROC_NUM] = { 0 };
DWORD idsNum = 0;
if(!EnumProcesses(procIDs, sizeof(DWORD) * MAX_PROC_NUM, &idsNum))
//handle error here
idsNum /= sizeof(DWORD); //After EnumProcesses(), idsNum contains number of BYTES!
2. Open required process. (OpenProcess,GetModuleFileNameEx)
const char* game_exe_path = "E:\\Games\\Spellforce\\Spellforce.exe"; //Example
HANDLE game_proc_handle = nullptr;
DWORD proc_access = PROCESS_QUERY_INFORMATION | PROCESS_VM_READ | PROCESS_VM_WRITE; //read & write memory, query info needed to get .exe name
const DWORD MAX_EXE_PATH_LEN = 1024;
for(DWORD n = 0 ; n < idsNum ; ++idsNum)
{
DWORD current_id = procIDs[n];
HANDLE current_handle = OpenProcess(proc_access, false, current_id);
if(!current_handle)
{
//handle error here
continue;
}
char current_path[MAX_EXE_PATH_LEN];
DWORD length = GetModuleFileNameEx(current_handle, nullptr, current_path, MAX_EXE_PATH_LEN);
if(length > 0)
{
if(strcmp(current_path, game_exe_path) == 0) //that's our game!
{
game_proc_handle = current_handle;
break;
}
}
CloseHandle(current_handle); //don't forget this!
}
if(!game_proc_handle)
//sorry, game not found
3. Write memory (WriteProcessMemory)
void* pointer = reinterpret_cast<void*>(0x02F70BCC);
int new_value = 5000; //value to be written
BOOL success = WriteProcessMemory(game_proc_handle, pointer, &new_value, sizeof(int), nullptr);
if(success)
//data successfully written!
else
//well, that's... em...
This code is written just 'as is', but I see no errors, so you can use it as your starting point. I also provided links for all functions I used, so with some additional research (if necessary), you can achieve what you are trying to.
Cheers.
When you use,
cout<<*pointer;
the program tries to dereference the value of the pointer and writes the value at the address.
If you want to print just the pointer, use:
cout << pointer;
Example:
int main()
{
int i = 20;
int* p = &i;
std::cout << *p << std::endl; // print the value stored at the address
// pointed to by p. In this case, it will
// print the value of i, which is 20
std::cout << p << std::endl; // print the address that p points to
// It will print the address of i.
}
I have a very large file and I need to read it in small pieces and then process each piece. I'm using MapViewOfFile function to map a piece in memory, but after reading first part I can't read the second. It throws when I'm trying to map it.
char *tmp_buffer = new char[bufferSize];
LPCWSTR input = L"input";
OFSTRUCT tOfStr;
tOfStr.cBytes = sizeof tOfStr;
HANDLE inputFile = (HANDLE)OpenFile(inputFileName, &tOfStr, OF_READ);
HANDLE fileMap = CreateFileMapping(inputFile, NULL, PAGE_READONLY, 0, 0, input);
while (offset < fileSize)
{
long k = 0;
bool cutted = false;
offset -= tempBufferSize;
if (fileSize - offset <= bufferSize)
{
bufferSize = fileSize - offset;
}
char *buffer = new char[bufferSize + tempBufferSize];
for(int i = 0; i < tempBufferSize; i++)
{
buffer[i] = tempBuffer[i];
}
char *tmp_buffer = new char[bufferSize];
LPCWSTR input = L"input";
HANDLE inputFile;
OFSTRUCT tOfStr;
tOfStr.cBytes = sizeof tOfStr;
long long offsetHigh = ((offset >> 32) & 0xFFFFFFFF);
long long offsetLow = (offset & 0xFFFFFFFF);
tmp_buffer = (char *)MapViewOfFile(fileMap, FILE_MAP_READ, (int)offsetHigh, (int)offsetLow, bufferSize);
memcpy(&buffer[tempBufferSize], &tmp_buffer[0], bufferSize);
UnmapViewOfFile(tmp_buffer);
offset += bufferSize;
offsetHigh = ((offset >> 32) & 0xFFFFFFFF);
offsetLow = (offset & 0xFFFFFFFF);
if (offset < fileSize)
{
char *next;
next = (char *)MapViewOfFile(fileMap, FILE_MAP_READ, (int)offsetHigh, (int)offsetLow, 1);
if (next[0] >= '0' && next[0] <= '9')
{
cutted = true;
}
UnmapViewOfFile(next);
}
ostringstream path_stream;
path_stream << tempPath << splitNum;
ProcessChunk(buffer, path_stream.str(), cutted, bufferSize);
delete buffer;
cout << (splitNum + 1) << " file(s) sorted" << endl;
splitNum++;
}
One possibility is that you're not using an offset that's a multiple of the allocation granularity. From MSDN:
The combination of the high and low offsets must specify an offset within the file mapping. They must also match the memory allocation granularity of the system. That is, the offset must be a multiple of the allocation granularity. To obtain the memory allocation granularity of the system, use the GetSystemInfo function, which fills in the members of a SYSTEM_INFO structure.
If you try to map at something other than a multiple of the allocation granularity, the mapping will fail and GetLastError will return ERROR_MAPPED_ALIGNMENT.
Other than that, there are many problems in the code sample that make it very difficult to see what you're trying to do and where it's going wrong. At a minimum, you need to solve the memory leaks. You seem to be allocating and then leaking completely unnecessary buffers. Giving them better names can make it clear what they are actually used for.
Then I suggest putting a breakpoint on the calls to MapViewOfFile, and then checking all of the parameter values you're passing in to make sure they look right. As a start, on the second call, you'd expect offsetHigh to be 0 and offsetLow to be bufferSize.
A few suspicious things off the bat:
HANDLE inputFile = (HANDLE)OpenFile(inputFileName, &tOfStr, OF_READ);
Every cast should make you suspicious. Sometimes they are necessary, but make sure you understand why. At this point you should ask yourself why every other file API you're using requires a HANDLE and this function returns an HFILE. If you check OpenFile documentation, you'll see, "This function has limited capabilities and is not recommended. For new application development, use the CreateFile function." I know that sounds confusing because you want to open an existing file, but CreateFile can do exactly that, and it returns the right type.
long long offsetHigh = ((offset >> 32) & 0xFFFFFFFF);
What type is offset? You probably want to make sure it's an unsigned long long or equivalent. When bitshifting, especially to the right, you almost always want an unsigned type to avoid sign-extension. You also have to make sure that it's a type that has more bits than the amount you're shifting by--shifting a 32-bit value by 32 (or more) bits is actually undefined in C and C++, which allows the compilers to do certain types of optimizations.
long long offsetLow = (offset & 0xFFFFFFFF);
In both of these statements, you have to be careful about the 0xFFFFFFFF value. Since you didn't cast it or give it a suffix, it can be hard to predict whether the compiler will treat it as an int or unsigned int. In this case,
it'll be an unsigned int, but that won't be obvious to many people. In fact,
I got this wrong when I first wrote this answer. [This paragraph corrected 16-MAY-2017] With bitwise operations, you almost always want to make sure you're using unsigned values.
tmp_buffer = (char *)MapViewOfFile(fileMap, FILE_MAP_READ, (int)offsetHigh, (int)offsetLow, bufferSize);
You're casting offsetHigh and offsetLow to ints, which are signed values. The API actually wants DWORDs, which are unsigned values. Rather than casting in the call, I would declare offsetHigh and offsetLow as DWORDs and do the casting in the initialization, like this:
DWORD offsetHigh = static_cast<DWORD>((offset >> 32) & 0xFFFFFFFFul);
DWORD offsetLow = static_cast<DWORD>( offset & 0xFFFFFFFFul);
tmp_buffer = reinterpret_cast<const char *>(MapViewOfFile(fileMap, FILE_MAP_READ, offsetHigh, offsetLow, bufferSize));
Those fixes may or may not resolve your problem. It's hard to tell what's going on from the incomplete code sample.
Here's a working sample you can compare to:
// Calls ProcessChunk with each chunk of the file.
void ReadInChunks(const WCHAR *pszFileName) {
// Offsets must be a multiple of the system's allocation granularity. We
// guarantee this by making our view size equal to the allocation granularity.
SYSTEM_INFO sysinfo = {0};
::GetSystemInfo(&sysinfo);
DWORD cbView = sysinfo.dwAllocationGranularity;
HANDLE hfile = ::CreateFileW(pszFileName, GENERIC_READ, FILE_SHARE_READ,
NULL, OPEN_EXISTING, 0, NULL);
if (hfile != INVALID_HANDLE_VALUE) {
LARGE_INTEGER file_size = {0};
::GetFileSizeEx(hfile, &file_size);
const unsigned long long cbFile =
static_cast<unsigned long long>(file_size.QuadPart);
HANDLE hmap = ::CreateFileMappingW(hfile, NULL, PAGE_READONLY, 0, 0, NULL);
if (hmap != NULL) {
for (unsigned long long offset = 0; offset < cbFile; offset += cbView) {
DWORD high = static_cast<DWORD>((offset >> 32) & 0xFFFFFFFFul);
DWORD low = static_cast<DWORD>( offset & 0xFFFFFFFFul);
// The last view may be shorter.
if (offset + cbView > cbFile) {
cbView = static_cast<int>(cbFile - offset);
}
const char *pView = static_cast<const char *>(
::MapViewOfFile(hmap, FILE_MAP_READ, high, low, cbView));
if (pView != NULL) {
ProcessChunk(pView, cbView);
}
}
::CloseHandle(hmap);
}
::CloseHandle(hfile);
}
}
You have a memory leak in your code:
char *tmp_buffer = new char[bufferSize];
[ ... ]
while (offset < fileSize)
{
[ ... ]
char *tmp_buffer = new char[bufferSize];
[ ... ]
tmp_buffer = (char *)MapViewOfFile(fileMap, FILE_MAP_READ, (int)offsetHigh, (int)offsetLow, bufferSize);
[ ... ]
}
You're never delete what you allocate via new char[] during every iteration there. If your file is large enough / you do enough iterations of this loop, the memory allocation will eventually fail - that's then you'll see a throw() done by the allocator.
Win32 API calls like MapViewOfFile() are not C++ and never throw, they return error codes (the latter NULL on failure). Therefore, if you see exceptions, something's wrong in you C++ code. Likely the above.
I also had some troubles with memory mapped files.
Basically I just wanted to share memory (1Mo) between 2 apps on the same Pc.
- Both apps where written in Delphi
- Using Windows8 Pro
At first one application (the first one launched) could read and write the memoryMappedFile, but the second one could only read it (error 5 : AccessDenied)
Finally after a lot of testing It suddenly worked when both application where using CreateFileMapping. I even tried to create my on security descriptor, nothing helped.
Just before my applications where first calling OpenFileMapping and then CreateFileMapping if the first one failed
Another thing that misleaded me is that the handles , although visibly referencing the same MemoryMappedFile where different in both applications.
One last thing, after this correction my application seemed to work all right, but after a while I had error_NotEnough_Memory. when calling MapViewOfFile.
It was just a beginner's mistake of my part, I was not always calling UnmapViewOfFile.
Please look at this code below.
#include <windows.h>
void Write(char *pBuffer)
{
// pBuffer -= 4*sizeof(int);
for(int i = 0; i<20; i++)
*(pBuffer + sizeof(int)*i) = i+1;
}
void main()
{
HANDLE hFile = ::CreateFile("file", GENERIC_READ|GENERIC_WRITE, 0, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);
if(INVALID_HANDLE_VALUE == hFile)
{
::MessageBox(NULL, "", "Error", 0);
return;
}
HANDLE hMMF = ::CreateFileMapping(hFile, NULL, PAGE_READWRITE, 0, 32, NULL);
char *pBuffer = (char*)::MapViewOfFile(hMMF, FILE_MAP_WRITE, 0, 0, 0);
Write(pBuffer);
::FlushViewOfFile(pBuffer, 100);
::UnmapViewOfFile(pBuffer);
}
I have allocated only 32 bytes yet when I attempt to write past the allocated size, I don't get any error at all. Is this by design or is this a bug in Windows code? However, if you include the commented part, it gives error, as expected.
I ask this because I am thinking of using this "feature" to my advantage. Can I? FYI, I have Win XP ver 2002 SP 3 but I suspect this to be "fixed" in newer Windows' which might fail my code, IDK. Any useful link explaining some internals of this would really help.
Thanks
This isn't any different then writing past the end of a buffer that's allocated on the heap. The operating system can only slap your fingers if you write to virtual memory that isn't mapped. Mapping is page based, one page is 4096 bytes. You'll have to write past this page to get the kaboom. Change your for-loop to end at (4096+4)/4 to repro it.
The virtual memory manager has to map memory by the page, so the extent will in effect be rounded up to the nearest 4kB (or whatever your system page size is).
I don't think it's documented whether writes into the same page as mapped data, but beyond the end of the mapping, will be committed back to the file. So don't rely on that behavior, it could easily change between Windows versions.