How to use OVERLAPPED to write data at specific offset (win32 programming) - c++

(I'm not native English speaker so sorry for potential mistake :p)
I want to write a Windows tool that can accelerate big files copying just like FastCopy. But get stuck at the very first step: Write data to specific offset of a file via OVERLAPPED (with WriteFile).
Here is my code:
#define _CRT_SECURE_NO_WARNINGS
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <Windows.h>
#include <fileapi.h>
LPOVERLAPPED SetupOvl(DWORD OffsetHigh, DWORD Offset) {
printf("Ovl??: Got Offset %d\n", Offset);
LPOVERLAPPED Ovl = (LPOVERLAPPED)malloc(sizeof(OVERLAPPED));
Ovl->Internal = 0;
Ovl->InternalHigh = 0;
Ovl->OffsetHigh = OffsetHigh;
Ovl->Offset = Offset;
Ovl->Pointer = NULL;
Ovl->hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);
return Ovl;
}
int main()
{
HANDLE hf,hf2;
void* buffer2 = (void*)malloc(2 << 16);
void* buffer = (void*)malloc(2 << 16);
int readed_number = 0, writed_number = 0;
DWORD written_number = 0;
DWORD wn1 = 0, wn2 = 0;
// OpenThread
DeleteFileW(L"C:\\Users\\Lev\\Downloads\\dst.txt");
hf = CreateFileW(
L"C:\\Users\\Lev\\Downloads\\src.txt",
GENERIC_READ,
FILE_SHARE_READ,
NULL,
OPEN_EXISTING,
FILE_ATTRIBUTE_NORMAL,
0);
hf2 = CreateFileW(
L"C:\\Users\\Lev\\Downloads\\dst.txt",
GENERIC_READ | GENERIC_WRITE,
0,
NULL,
CREATE_NEW,
FILE_ATTRIBUTE_NORMAL | FILE_FLAG_OVERLAPPED,
NULL);
// ReadThread
BOOL ret = ReadFile(
hf,
buffer,
2 << 15,
(LPDWORD)&readed_number,
NULL
);
SetFilePointer(hf2, 16, 0, FILE_BEGIN);
SetEndOfFile(hf2);
LPOVERLAPPED Ovl1 = SetupOvl(0, 2);
WriteFile(hf2, buffer, 6, NULL, Ovl1);
LPOVERLAPPED Ovl2 = SetupOvl(0, 12);
buffer = (void*)((char*)buffer + (unsigned int)(10));
WriteFile(hf2, buffer, 4, NULL, Ovl2);
ret = GetOverlappedResult(hf2, Ovl1, &wn1, TRUE);
ret = GetOverlappedResult(hf2, Ovl2, &wn2, TRUE);
printf("Write_A: Write %d Bytes!\n", wn1);
printf("Write_B: Write %d Bytes!\n", wn2);
free(Ovl1);
free(Ovl2);
CloseHandle(hf);
CloseHandle(hf2);
return 0;
}
Output from console:
Ovl??: Got Offset 2
Ovl??: Got Offset 8
Write_A: Write 4 Bytes!
Write_B: Write 5 Bytes!
Content of src.txt: 1234567890abcdefghijklmn
Content of dst.txt: abcd56 (6 charactors and lots of 0x00)
In the code I try to write "123456" at 3rd byte and "abcd" at 13th byte. But it didn't work. I have look up MSDN and some books but found not code examples I need.
By the way, which is the best way to make a multi-thread acceleration? Multi-thread reading or multi-thread writing? Is it possible to use OVERLAPPED I/O to implement multiple threads writing to the same file in parallel? Or does it just write sequentially?
Any helpful suggestion is appreciated!

Related

C++ wininet fetching data in 513 byte chunks

I am trying to fetch my server html text data using wininet, but it seems to be reading it and breaking it into 513 byte size chunks. However I would prefer the data to be fetched as a whole. Is there a way I can go around this?
Here is a full code for references:
#include <windows.h>
#include <wininet.h>
#include <stdio.h>
#include <iostream>
using namespace std;
#pragma comment (lib, "Wininet.lib")
int main(int argc, char** argv) {
HINTERNET hSession = InternetOpen(L"Mozilla/5.0", INTERNET_OPEN_TYPE_PRECONFIG, NULL, NULL, 0);
HINTERNET hConnect = InternetConnect(hSession, L"www.google.com", 0, L"", L"", INTERNET_SERVICE_HTTP, 0, 0);
HINTERNET hHttpFile = HttpOpenRequest(hConnect, L"GET", L"/", NULL, NULL, NULL, 0, 0);
while (!HttpSendRequest(hHttpFile, NULL, 0, 0, 0)) {
printf("Server Down.. (%lu)\n", GetLastError());
InternetErrorDlg(GetDesktopWindow(), hHttpFile, ERROR_INTERNET_CLIENT_AUTH_CERT_NEEDED, FLAGS_ERROR_UI_FILTER_FOR_ERRORS | FLAGS_ERROR_UI_FLAGS_GENERATE_DATA | FLAGS_ERROR_UI_FLAGS_CHANGE_OPTIONS, NULL);
}
DWORD dwFileSize;
dwFileSize = BUFSIZ;
char* buffer;
buffer = new char[dwFileSize + 1];
while (true) {
DWORD dwBytesRead;
BOOL bRead;
bRead = InternetReadFile(hHttpFile, buffer, dwFileSize + 1, &dwBytesRead);
if (dwBytesRead == 0) break;
if (!bRead) {
printf("InternetReadFile error : <%lu>\n", GetLastError());
}
else {
buffer[dwBytesRead] = 0;
std::string newbuff = buffer;
std::wcout << "\n\nSize: " << newbuff.size() << std::endl;
std::cout << newbuff;
}
}
InternetCloseHandle(hHttpFile);
InternetCloseHandle(hConnect);
InternetCloseHandle(hSession);
}
dwFileSize = BUFSIZ;
If you look at your header files you will discover that BUFSIZ is #defined as 512.
buffer = new char[dwFileSize + 1];
// ...
Read = InternetReadFile(hHttpFile, buffer, dwFileSize + 1, &dwBytesRead);
Your program then proceeds to allocate a buffer of 513 bytes, and then read the input, 513 bytes at a time. And that's exactly the result you're observing. This is the only thing that the shown code knows how to do.
If you want to use a larger buffer size, you can easily change that (and there is no reason to new the buffer anyway, all that does is create an opportunity for memory leaks, just use std::vector, or a plain array for smaller buffers of this size).
Unless you have advance knowledge of the size of the file to retrieve you have no alternative but to read the file, in pieces, like that. Even though it's being retrieved in pieces it looks like the file is getting retrieved in a single request, and it's just that the shown code retrieve it one chunk at a time, which is perfectly fine. Switching to a large buffer size might improve the performance, but not dramatically.

Getting Directory Size by reading through the MFT in C++ (like TreeSize)

Please read before quoting “repost” - I am aware similar questions have been asked, but I am yet to find a satisfactory answer
My goal is to provide a tree-like directory structure of disk space usage allowing the user to drill down the hierarchy in order to locate sizable folders.
The program TreeSize is an excellent example of this, and I am looking to get the same response times as this program.
My current code can iterate through my 480GB of files in aprx 25 seconds using the MFT. I am looking from this point to start building directory sizes by getting the file information (MFT contains only fileName and parentId, not full file path)
To get file information from MFT journal entry my current code calls
TCHAR filePath[MAX_PATH];
HANDLE hh = OpenFileById(hDevice, &(getFileIdDescriptor(pRecord->FileReferenceNumber)), 0, 0, 0, 0);
GetFinalPathNameByHandle(hh, filePath, MAX_PATH, 0);
Unfortunately this code increases the overall execution time of the program from 25 seconds to 5 minutes.
Is there a better way to get the file information?
Many thanks if you suggest FindFirstFile and FindNextFile but for processing large directories these options are too slow
Code as below (I’m not a C programmer as you might notice!)
#include <iostream>
#include <string>
#include <fstream>
#include <windows.h>
#include <fstream>
#include <atlbase.h>
#include <windows.h>
#include <stdio.h>
using namespace std;
typedef std::basic_string<TCHAR> tstring;
FILE_ID_DESCRIPTOR getFileIdDescriptor(const DWORDLONG fileId)
{
FILE_ID_DESCRIPTOR fileDescriptor;
fileDescriptor.Type = FileIdType;
fileDescriptor.FileId.QuadPart = fileId;
fileDescriptor.dwSize = sizeof(fileDescriptor);
return fileDescriptor;
}
bool ReadMFT()
{
HANDLE hDevice = CreateFile(TEXT("\\\\.\\C:"),
GENERIC_READ | GENERIC_WRITE,
FILE_SHARE_READ | FILE_SHARE_WRITE,
0,
OPEN_EXISTING,
FILE_FLAG_OVERLAPPED,
0);
if (hDevice == INVALID_HANDLE_VALUE) // cannot open the drive
{
printf("Error %d", GetLastError());
return (FALSE);
}
USN_JOURNAL_DATA ujd = { 0 };
DWORD cb = 0;
BYTE pData[sizeof(DWORDLONG) + 0x10000] = { 0 };
if (!DeviceIoControl(hDevice, FSCTL_QUERY_USN_JOURNAL, NULL, 0, &ujd, sizeof(USN_JOURNAL_DATA), &cb, NULL))
{
printf("Error %d", GetLastError());
return (FALSE);
}
MFT_ENUM_DATA med = { 0 };
med.StartFileReferenceNumber = 0;
med.LowUsn = 0;
med.HighUsn = ujd.NextUsn;
while (TRUE)
{
if (!DeviceIoControl(hDevice, FSCTL_ENUM_USN_DATA, &med, sizeof(med), pData, sizeof(pData), &cb, NULL))
{
printf("Error %d", GetLastError());
break;
}
PUSN_RECORD pRecord = (PUSN_RECORD)&pData[sizeof(USN)];
//Inner Loop
while ((PBYTE)pRecord < (pData + cb))
{
tstring sz((LPCWSTR)
((PBYTE)pRecord + pRecord->FileNameOffset),
pRecord->FileNameLength / sizeof(WCHAR));
pRecord = (PUSN_RECORD)((PBYTE)pRecord + pRecord->RecordLength);
// *******************************************************************************
// APPROACH 1
// Adding these lines of code increases the time from 25 seconds to 340 seconds
// Although it may be possible to push this onto a queue and run these in parrallel
// I still think it's an expensive option
/*TCHAR filePath[MAX_PATH];
HANDLE hh = OpenFileById(hDevice, &(getFileIdDescriptor(pRecord->FileReferenceNumber)), 0, 0, 0, 0);
GetFinalPathNameByHandle(hh, filePath, MAX_PATH, 0);*/
}
med.StartFileReferenceNumber = *(DWORDLONG *)pData;
}
}
int main()
{
ReadMFT();
}
Many thanks
After a few trials and errors, running the
FILE_ID_DESCRIPTOR f = getFileIdDescriptor(pRecord->FileReferenceNumber);
q.Dispatch(f, [f]
{
TCHAR filePath[MAX_PATH];
HANDLE hh = OpenFileById(hDevice, (LPFILE_ID_DESCRIPTOR)&(f), 0, 0, 0, 0);
GetFinalPathNameByHandle(hh, filePath, MAX_PATH, 0);
});
in parrallel brings the time down to 1:30
Check out this guys dispatch queue implementation
https://github.com/embeddedartistry/embedded-resources/blob/master/examples/cpp/dispatch.cpp
You might want to iterate the $INDEX_ALLOCATION attribute to get a list of the child nodes.

Reading large mapped text file in C++

I am attempting to display a large amount of text(barely less than 1GB).
My code:
HANDLE hFile;
DWORD dwBytesRead = 0;
OVERLAPPED ol = {0};
HANDLE m_hMapFile;
hFile = CreateFile(_T("test.txt"),
GENERIC_WRITE | GENERIC_READ,
0,
NULL,
OPEN_EXISTING,
FILE_ATTRIBUTE_NORMAL | FILE_FLAG_OVERLAPPED,
NULL);
m_hMapFile = CreateFileMapping(hFile, NULL, PAGE_READWRITE, 0, 0, NULL);
LPVOID m_lpMapAddress = MapViewOfFile(m_hMapFile, FILE_MAP_ALL_ACCESS, 0,
0, 0);
}
Now that the text file is mapped, how do I display its contents? I have attempted the following (poor) implementation:
char *mappedData = (char*)m_lpMapAddress;
for(int k = 0; k < strlen(mappedData); k++){
cout<<mappedData [k];
}
This is obviously not the right way to display the text contents. Is there a more efficient method?
You may try doing all your output at once:
cout.write(mappedData, mappedSize);
But note that printing a gigabyte of data to console is not likely to be efficient anyhow.
Console output has purpose of being read by user (programmatic parsing is secondary thing). Do you expect user to read 1 GB of data?

WMI returns information in different formats

I want to read the hardware configuration to check if a license for my software is valid. Currently, I tried using WMI. This works fine on many machines for several weeks but sometimes, without an obvious reason, WMI returns the hardware configuration in a different format. For example, the serial number of the primary hard disc is converted from characters to a hex string, with all character hex values being swapped pair wise. I figured out that different Windows user types (admin/normal) influence the format but it also changes in other situations and in different ways, for which I am unable to figure out a pattern.
Does anybody know how to reliably check the hardware configuration using WMI? Or would it be possible to avoid the above problem using MFC?
WMI is indeed unreliable. You should avoid using it when you don't need it.
Here's one way without WMI:
#include <string>
#include <Dbt.h>
#include <winioctl.h>
#include <SetupAPI.h>
#pragma comment(lib, "SetupAPI.lib")
#include <initguid.h>
DWORD getDeviceNumber(HANDLE hDeviceHandle)
{
STORAGE_DEVICE_NUMBER sdn = { 0 };
sdn.DeviceNumber = -1;
DWORD dwBytesReturned = 0;
if (!DeviceIoControl(hDeviceHandle, IOCTL_STORAGE_GET_DEVICE_NUMBER, nullptr, 0, &sdn, sizeof(sdn), &dwBytesReturned, nullptr))
{
return -1; //Error
}
return sdn.DeviceNumber;
}
bool GetDeviceString(std::wstring &out)
{
wchar_t wDevicePath[] = L"\\\\.\\#:";
wDevicePath[4] = L'C'; //Replace this with your drive-letter & adjust code (C: / D: whatever)
HANDLE deviceHandle = CreateFileW(wDevicePath, 0, FILE_SHARE_READ | FILE_SHARE_WRITE, nullptr, OPEN_EXISTING, 0, nullptr);
if (deviceHandle == INVALID_HANDLE_VALUE)
return false;
DWORD dwVolumeDeviceNumber = getDeviceNumber(deviceHandle);
CloseHandle(deviceHandle);
HDEVINFO hDevInfo = SetupDiGetClassDevsW(&GUID_DEVINTERFACE_DISK, nullptr, nullptr, DIGCF_PRESENT | DIGCF_DEVICEINTERFACE);
if (hDevInfo == INVALID_HANDLE_VALUE)
return false; //Error
std::vector<BYTE> buf(1024);
PSP_DEVICE_INTERFACE_DETAIL_DATA_W psp = reinterpret_cast<PSP_DEVICE_INTERFACE_DETAIL_DATA_W>(buf.data());
SP_DEVICE_INTERFACE_DATA spInt;
SP_DEVINFO_DATA spDev;
spInt.cbSize = sizeof(spInt);
DWORD dwIndex = 0;
while (true)
{
if (!SetupDiEnumDeviceInterfaces(hDevInfo, nullptr, &GUID_DEVINTERFACE_DISK, dwIndex, &spInt))
break;
DWORD dwSize = 0;
SetupDiGetDeviceInterfaceDetailW(hDevInfo, &spInt, nullptr, 0, &dwSize, nullptr);
if (dwSize && dwSize <= buf.size())
{
psp->cbSize = sizeof(*psp);
memset(&spDev, sizeof(spDev), 0);
spDev.cbSize = sizeof(spDev);
long res = SetupDiGetDeviceInterfaceDetailW(hDevInfo, &spInt, psp, dwSize, &dwSize, &spDev);
if (res)
{
HANDLE hDrive = CreateFileW(psp->DevicePath, 0, FILE_SHARE_READ | FILE_SHARE_WRITE, nullptr, OPEN_EXISTING, 0, nullptr);
if (hDrive != INVALID_HANDLE_VALUE)
{
DWORD dwUsbDeviceNumber = getDeviceNumber(hDrive);
if (dwUsbDeviceNumber == dwVolumeDeviceNumber)
{
//Found
out = psp->DevicePath;
break;
}
}
CloseHandle(hDrive);
}
}
++dwIndex;
}
SetupDiDestroyDeviceInfoList(hDevInfo);
if (out.empty()) //Was not found
return false;
return true;
}
After that, you will get a large device string. You might want to read the needed information out of it.
Check the following regular expressions to retrieve these:
(Note that the string CAN change, depending on the device-type, so test it and add/adjust the regular expressions - these are from an USB-stick test)
ven_([^&#]+) //Vendor String/ID
prod_([^&#]+) //Product String/ID
rev_([^&#]+) //Revision String/ID
&[^#]*#([^&#]+) //Serial String/Number
Regular expressions ? Another example:
std::wregex (see std::basic_regex...).
std::wsmatch (see std::match_results...)
std::wstring wsDeviceString;
if (GetDeviceString(wsDeviceString))
{
std::wregex regexSerialNumber(L"&[^#]*#([^&#]+)");
std::wsmatch match;
if (std::regex_search(wsDeviceString, match, regexSerialNumber))
std::wcout << L"Serial Number of device is: " << match[1].str() << std::endl;
}
One license for your product please =)

Crash when calling ReadFile after LockFileEx

I have several processes that try to read and write the same file. I want each of them to lock the file so that only one of them accesses it at a time.
I tried this (edit: this is a complete test code this time):
#include "stdafx.h"
#include "Windows.h"
bool test()
{
const char* path = "test.txt";
HANDLE hFile = CreateFileA(path,
GENERIC_READ | GENERIC_WRITE,
FILE_SHARE_READ | FILE_SHARE_WRITE,
NULL,
OPEN_ALWAYS,
FILE_ATTRIBUTE_NORMAL,
NULL);
if (hFile == INVALID_HANDLE_VALUE)
{
printf("ERROR: Cannot open file %s\n", path);
return false;
}
// Lock the file
{
OVERLAPPED overlapped = {0};
BOOL res = LockFileEx(hFile, LOCKFILE_EXCLUSIVE_LOCK, 0, ~0, ~0, &overlapped);
if (!res)
{
printf("ERROR: Cannot lock file %s\n", path);
return false;
}
}
DWORD fileSize = GetFileSize(hFile, NULL);
if (fileSize > 0)
{
char* content = new char[fileSize+1];
// Read the file
BOOL res = ReadFile(hFile, content, fileSize, NULL, NULL);
if (!res)
{
printf("ERROR: Cannot read file %s\n", path);
}
delete[] content;
}
const char* newContent = "bla";
int newContentSize = 3;
// Write the file
BOOL res = WriteFile(hFile, newContent, newContentSize, NULL, NULL);
if (!res)
{
//int err = GetLastError();
printf("ERROR: Cannot write to file\n");
}
// Unlock the file
{
OVERLAPPED overlapped = {0};
UnlockFileEx(hFile, 0, ~0, ~0, &overlapped);
}
CloseHandle(hFile);
return true;
}
int _tmain(int argc, _TCHAR* argv[])
{
bool res = test();
return 0;
}
This works fine on my computer, which has Windows 8. But on my colleague's computer, which has Windows 7, it crashes. Specifically, the calls to ReadFile and WriteFile crash, always.
Note that it never enters the code paths with the error printfs. This code triggers no error except for a write at location 0x00000000 in ReadFile (when run on Windows 7).
We tried to also pass the overlapped struct to the ReadFile and WriteFile calls. It prevents the crash but the lock doesn't work anymore, the file is all scrambled (not with this test code, with the real code).
What am I doing wrong?
Looks like your problem is:
lpNumberOfBytesRead [out, optional] argument is null in your call.
This parameter can be NULL only when the lpOverlapped parameter is not NULL.
http://msdn.microsoft.com/en-us/library/windows/desktop/aa365467%28v=vs.85%29.aspx
Heres your problem :
You are missing a necessary struct-member and:
0 and ~0 and {0} are all bad code, constant expressions like these will always produce unepected results -- WINAPI doesnt work like libc, parameters are not always compared against constants, instead they are tested against/via macros and other preprocessor-definitions themselves so passing constant values or initializing WINAPI structs with constants will often lead to errors like these.
After years of experimenting i have found that there is only one surefire way of avoiding them, i will express it in corrected code :
OVERLAPPED overlapped;
overlapped.hEvent = CreateEvent( ........... ); // put valid parameters here!
UnlockFileEx(hFile, 0 /*"reserved"*/, ULONG_MAX, ULONG_MAX, &overlapped);
please read this carefully : http://msdn.microsoft.com/en-us/library/windows/desktop/aa365716%28v=vs.85%29.aspx