I am trying to find ways to manage system storage efficiently.
One thing I've noticed while using MS OneDrive is that whenever I click "Free up space" on its context menu, it converts the "Storage on disk" to 0 bytes while its File size remains the same. I want to mimic what is happening over there.
Then I researched to see if I can partially modify its attribute (allocation size) through SetFileInformationByHandle using FileAllocationInfo as its parameter. It succeeded but both of its file size and allocation size became 0.
Test code as follows:
WCHAR wcsDebug[2048];
// testfile is 1224 bytes
HANDLE hFile = CreateFile(_TEXT("C:\\test\\testfile.txt"), GENERIC_READ | GENERIC_WRITE, 0, NULL, OPEN_EXISTING, 0, NULL);
if (hFile != INVALID_HANDLE_VALUE)
{
FILE_ALLOCATION_INFO allocInfo;
ZeroMemory(&allocInfo, sizeof(FILE_ALLOCATION_INFO));
BOOL fResult = SetFileInformationByHandle(hFile,
FileAllocationInfo,
&allocInfo,
sizeof(FILE_ALLOCATION_INFO));
// after the execution, the size becomes 0
if (fResult)
{
swprintf_s(wcsDebug, _countof(wcsDebug), L"SetFileInfomationByHandle() Success.");
OutputDebugString(wcsDebug);
}
else
{
swprintf_s(wcsDebug, _countof(wcsDebug), L"SetFileInfomationByHandle() Failed. Err(%08X)", GetLastError());
OutputDebugString(wcsDebug);
}
}
Any more ideas on developing the thoughts?
I am trying to get a number from a HANDLE file, store it in an int, and possibly replace it in the same file. My code right now looks like this
HANDLE numFile = INVALID_HANDLE_VALUE; //Just in case file not found
numFile = CreateFile("numFile.txt",
GENERIC_READ | GENERIC_WRITE,
FILE_SHARE_READ,
NULL,
OPEN_EXISTING,
FILE_ATTRIBUTE_NORMAL,
NULL);
ReadFile(numFile, input, sizeof(char), &bytesRead, NULL);
int myNumber = input[0];
I know that there exists a WriteFile method in the API, but it looks like it will append the file as opposed to overwriting the contents. I have briefly considered deleting and recreating the file each time, but this seems unnecessarily complex for this problem. Any ideas out there?
Use the function SetFilePointer prior WriteFile for moving back to the begin
SetFilePointer(numFile, 0, NULL, FILE_BEGIN);
Given one normal file, it could be read and written. Then I change this file attribute to Read-only through
However, this file still could be written through file handler. Here are my codes
#define CREATE_FILE_OPT FILE_ATTRIBUTE_NORMAL | FILE_FLAG_WRITE_THROUGH | FILE_FLAG_RANDOM_ACCESS
std::string name("test.txt");
HANDLE hfile = ::CreateFile(name.c_str(),
GENERIC_READ | GENERIC_WRITE, NULL, NULL, OPEN_EXISTING, CREATE_FILE_OPT, NULL);
if (hfile == INVALID_HANDLE_VALUE) {
hfile = ::CreateFile(name.c_str(),
GENERIC_READ | GENERIC_WRITE, NULL, NULL, CREATE_NEW, CREATE_FILE_OPT, NULL);
if (hfile == INVALID_HANDLE_VALUE) {
printf("so sad, invalid file handler....");
return -1;
}
}
int i = 0;
char rbuf[] = "you are";
DWORD bytesWritten;
do {
Sleep(5000);
++i;
bytesWritten = 0;
BOOL bret = ::WriteFile(hfile, rbuf, strlen(rbuf), &bytesWritten, NULL);
if (bret == FALSE) {
printf("Cannot write bytes into file.....");
DWORD err = GetLastError();
printf("The error code is %d\n", err);
}
else
printf("write %d bytes to file\n", bytesWritten);
DWORD ret = GetFileAttributes(name.c_str());
printf("The file attribute is %d\n", ret);
} while (i < 10000);
The file attribute is 32 before I change it to Read-only, but it will be 33 after this file is Read-only.
I want to know why the file could be written after change it to Read-only? Is there any issue in my test codes?
PS: test it in VS2015 under windows 7.
Sorry, I get your meaning finally. I guess you are talking about creating a file first, setting it readonly without closing the program. The READONLY attribute check only happens in CreateFile routine.
In Windows kernel, every object is assigned a temporary access right list once created, unless explicitly refreshed, a thing seldom happens on actual files on fixed disks. So even if you deny all rights after CreateFile using NTFS access control, your program will behave just as when CreateFile is called.
In conclusion, it is natural your program can still write to the file, after your hot-changing it into READONLY, which only writes information onto disk, not changing kernel access table.
My company is developing a "fancy" USB Mass Storage Device running under Windows 7. The Mass Storage Client Driver that handles the reading and writing to the actual storage media on the client side is being written in C++. The problem we are having is very, very slow write speeds. About 30 times slower than expected. We are using calls to WriteFile() to write blocks of data to the storage media (specifically the physical drive 'PhysicalDrive2') as they are received from the Host device. I have read in many other forums that people have experience very slow write speeds using WriteFile() especially on Windows 7. So I am trying to figure out if I am using the best method and function calls for this particular task.
Below are two blocks of code. One for LockVolume() function that gets called one time by the program during initialization and actually just unmounts the volume. The other block of code is WriteSector() which is used to write the actual data to the physical drive when its received by the USB Client controller driver. I am hoping that someone can shed some light on what I might be doing wrong or provided suggestions on a better way to implement this.
short WriteSector
(LPCWSTR _dsk, // disk to access
char *&_buff, // buffer containing data to be stored
unsigned int _nsect, // sector number, starting with 0
ULONG Blocks
)
{
DWORD bytesWritten;
wchar_t errMsg[256];
//attempt to get a handle to the specified volume or physical drive
HANDLE hDisk = CreateFile(_dsk, GENERIC_READ | GENERIC_WRITE, FILE_SHARE_READ | FILE_SHARE_WRITE, 0, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, 0);
//make sure we have a handle to the specified volume or physical drive
if(hDisk==INVALID_HANDLE_VALUE)
{
FormatMessage(FORMAT_MESSAGE_FROM_SYSTEM, NULL, GetLastError(),
MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT), errMsg, 255, NULL);
OutputDebugString(errMsg);
printf("Error attempting to get a handle to the device! (%s)\n", errMsg);
goto exit;
}
// set pointer to the sector on the disk that we want to write to
SetFilePointer(hDisk, (_nsect * SIZE_OF_BLOCK), 0, FILE_BEGIN);
//write the data
if (!WriteFile(hDisk, _buff, (Blocks * SIZE_OF_BLOCK), &bytesWritten, NULL))
{
FormatMessage(FORMAT_MESSAGE_FROM_SYSTEM, NULL, GetLastError(),
MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT), errMsg, 255, NULL);
printf("WriteFile failed! (%s)\n", errMsg);
goto exit;
}
exit:
CloseHandle(hDisk);
writeMutex.unlock();
return 0;
}
UINT Disk_LockVolume(LPCWSTR _dsk)
{
HANDLE hVol;
LPWSTR errMsg;
DWORD status;
bool success = false;
//now try to get a handle to the specified volume so we can write to it
hVol = CreateFile(_dsk, GENERIC_READ | GENERIC_WRITE, FILE_SHARE_READ | FILE_SHARE_WRITE, 0, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, 0);
//check to see if we were able to obtain a handle to the volume
if( hVol == INVALID_HANDLE_VALUE )
{
FormatMessage(FORMAT_MESSAGE_FROM_SYSTEM, NULL, GetLastError(),
MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT), errMsg, 255, NULL);
printf("Disk_LockVolume() - CreateFile failed (%s)\n", errMsg);
goto exit;
}
// now lock volume
if (!DeviceIoControl(hVol, FSCTL_LOCK_VOLUME, NULL, 0, NULL, 0, &status, NULL))
{
FormatMessage(FORMAT_MESSAGE_FROM_SYSTEM, NULL, GetLastError(),
MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT), errMsg, 255, NULL);
printf("Disk_LockVolume() - Error attempting to lock device! (%s)\n", errMsg);
goto exit;
}
//dismount the device
if (!DeviceIoControl(hVol, FSCTL_DISMOUNT_VOLUME, NULL, 0, NULL, 0, &status, NULL))
{
FormatMessage(FORMAT_MESSAGE_FROM_SYSTEM, NULL, GetLastError(),
MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT), errMsg, 255, NULL);
printf("Disk_LockVolume() - Error attempting to dismount volume. (%s)\n", errMsg);
goto exit;
}
exit:
CloseHandle(hVol);
return 1;
}
EDIT #1 (2/10/2015)
So I incorporated the suggestions made by Ben Voigt and found that calling CreateFile and CloseHandle only once (instead of every time we want to write data to the drive) significantly improved write speed. 80% increase. Even with that increase, the write speed was still much slower than expected. Around 6 times slower. So I then incorporated his other suggested change which included eliminating the original call to SetFilePointer() and replace it with and OVERLAPPED structure that now gets passed to WriteFile. After I made that change, I now get and error that states "stack around the variable 'MyOverLappedStructure' was corrupted". Below is the updated version of my SectorWrite function along with the new Disk_GetHandle() function which gets the initial handle to the Physical Drive. Also, I am still calling Disk_LockVolume(), one time, after I call Disk_GetHandle(). However, I have modified the Disk_LockVolume() function so that the handle to the volume (in this case) does not get closed at the end of the function. Ultimately that would be closed at the end of the program, prior to closing the handle on the Physical Drive. Any thoughts on this new error would be greatly appreciated. Oh, the FILE_FLAG_NO_BUFFERING had no impact on performance that I could see.
UINT WriteSector(HANDLE hWriteDisk, PBYTE Buf, ULONG Lba, ULONG Blocks)
{
DWORD bytesWritten;
LPTSTR errMsg = "";
//setup overlapped structure to tell WriteFile function where to write the data
OVERLAPPED overlapped_structure;
memset(&overlapped_structure, 0, (Blocks * SIZE_OF_BLOCK));
overlapped_structure.Offset = (Lba * SIZE_OF_BLOCK);
overlapped_structure.hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);
//write the data
if (!WriteFile(hWriteDisk, Buf, (Blocks * SIZE_OF_BLOCK), &bytesWritten, &overlapped_structure))
{
FormatMessage(FORMAT_MESSAGE_FROM_SYSTEM, NULL, GetLastError(),
MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT), errMsg, 255, NULL);
printf("WriteSector() - WriteFile failed (%s)\n", errMsg);
}
if (bytesWritten != (Blocks * SIZE_OF_BLOCK))
{
printf("WriteSector() - Bytes written did not equal the number of bytes to be written\n");
return 0;
}
else
{
return Blocks;
}
}
HANDLE Disk_GetHandle(UINT Lun)
{
HANDLE hVol;
LPTSTR errMsg = "";
bool success = false;
//now try to get a handle to the specified volume so we can write to it
hVol = CreateFile(MassStorageDisk[Lun].PhysicalDisk, GENERIC_READ | GENERIC_WRITE, FILE_SHARE_READ | FILE_SHARE_WRITE, 0, OPEN_EXISTING, FILE_FLAG_OVERLAPPED|FILE_FLAG_NO_BUFFERING, 0);
//check to see if we were able to obtain a handle to the volume
if( hVol == INVALID_HANDLE_VALUE )
{
FormatMessage(FORMAT_MESSAGE_FROM_SYSTEM, NULL, GetLastError(),
MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT), errMsg, 255, NULL);
printf("Disk_WriteData() - CreateFile failed (%s)\n", errMsg);
}
return hVol;
}
EDIT #2 (2/10/2015)
So I eliminated the FILE_FLAG_OVERLAPPED from CreateFile() call per Ben's comment. I also modified part of the WriteSector() function to include a check to see if IO is pending after call to WriteFile(). If so, I call WaitForSingleObject() which waits indefinitely until the IO operation is complete. Lastly, I call CloseHandle() on the OVERLAPPED structure hEvent. Even with these changes I still get the error "stack around the variable 'osWrite' was corrupted", where osWrite is the OVERLAPPED structure. Below is a code snippet illustrating the changes.
OVERLAPPED osWrite;
memset(&osWrite, 0, (Blocks * SIZE_OF_BLOCK));
osWrite.Offset = (Lba * SIZE_OF_BLOCK);
osWrite.hEvent = 0;
//write the data
if (!WriteFile(hWriteDisk, Buf, (Blocks * SIZE_OF_BLOCK), &bytesWritten, &osWrite))
{
DWORD Errorcode = GetLastError();
if (Errorcode == ERROR_IO_PENDING)
{
WaitForSingleObject(osWrite.hEvent, INFINITE);
}
else
{
FormatMessage(FORMAT_MESSAGE_FROM_SYSTEM, NULL, GetLastError(),
MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT), errMsg, 255, NULL);
printf("WriteSector() - WriteFile failed (%s)\n", errMsg);
goto exit;
}
}
EDIT #3 (2/10/2015)
So the code is now working with Ben's inputs. The code above has been modified to reflect those changes. I need to mention that up until this afternoon, all of my testing was done where the storage media on the client side was a USB flash drive. I have since changed that so the client now writes to an attached SSD. With the USB flash drive setup, the speed at which I can write data to the client over the USB connection is virtually identical now to the speed at which the client SBC can transfer the same file directly from itself to the storage media (without the host connected). However, with the SSD now being used, this is not the case. The test file I am using which is 34MB takes 2.5 seconds when transferred directly from the client SBC to the SSD. It takes 2.5 MINUTES from host to client over the USB. Other than changing the volume letter and physical drive number, no other changes were made to the code
You should not call CreateFile and CloseHandle for each sector overwritten. CreateFile is a very expensive operation that has to do security checks (evaluate group membership, walk SIDs, etc).
Open the handle once, pass it to WriteFile many times, and close it once. This means changing your _dsk parameter from a volume path to a HANDLE.
You probably also want to lose the call to SetFilePointer, and use an OVERLAPPED structure instead, which lets you supply the position to write to as part of the write call. (The operation won't be overlapped unless you use FILE_FLAG_OVERLAPPED, but non-overlapped I/O respects the position information in the OVERLAPPED structure).
I'm trying to create a process on another machine that deletes itself when its done. I'm using the DELETE_ON_CLOSE flag with CreateFile. This method is a bit popular but I'm having trouble because I can't execute it while it's open(expected but that's what some solutions do). To work around this I tried opening the file with read permissions. The DELETE_ON_CLOSE flag says that it should only delete a file when all pointers to it are gone. I have a pointer to it with the read, I close the write handle and the file deletes leaving my open handle unreadable. Any other way around this would be gladly appreciated.
I also have considered the possibility that since this is a remote file system there's something funky going on with the handles.
I cannot modify the code of the execuatble I'm sending over so a self-deleting executable is the last thing I want to do.
Making my program wait around to clean up the service will cause it to hang for an unacceptably long time because of how long it can take to destroy a service on the remote box.
//Open remote file for reading and set the delete flag
HANDLE remote_fh = CreateFile(&remote_file_location[0],
GENERIC_WRITE,
FILE_SHARE_DELETE | FILE_SHARE_READ | FILE_SHARE_WRITE,
NULL,
CREATE_ALWAYS,
FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_TEMPORARY | FILE_FLAG_DELETE_ON_CLOSE,
NULL);
if(!remote_fh)
{
debug.DebugMessage(Error::GetErrorMessageW(GetLastError()));
RevertToSelf();
return dead_return;
}
//File to read from
HANDLE local_fh = CreateFile(&local_file_location[0],
GENERIC_READ,
FILE_SHARE_DELETE | FILE_SHARE_READ | FILE_SHARE_WRITE,
NULL,
OPEN_EXISTING,
FILE_ATTRIBUTE_NORMAL,
NULL);
if(!local_fh)
{
debug.DebugMessage(Error::GetErrorMessageW(GetLastError()));
RevertToSelf();
return dead_return;
}
byte buf[256];
DWORD bytesRead;
DWORD bytesWritten;
//Copy the file
while(ReadFile(local_fh, buf, 256, &bytesRead, NULL) && bytesRead > 0)
{
WriteFile(remote_fh, buf, bytesRead, &bytesWritten, NULL);
}
CloseHandle(local_fh);
//Create a file retainer to hold the pointer so the file doesn't get deleted before the service starts
HANDLE remote_retain_fh = CreateFile(&remote_file_location[0],
GENERIC_READ,
FILE_SHARE_DELETE | FILE_SHARE_READ | FILE_SHARE_WRITE,
NULL,
OPEN_EXISTING,
FILE_ATTRIBUTE_NORMAL,
NULL);
if (!remote_retain_fh)
{
debug.DebugMessage(Error::GetErrorMessageW(GetLastError()));
RevertToSelf();
return dead_return;
}
CloseHandle(remote_fh);
//if(!CopyFile(&local_file_location[0], &remote_file_location[0], false))
//{
// debug.DebugMessage(Error::GetErrorMessageW(GetLastError()));
// RevertToSelf();
// return dead_return;
//}
remote_service.Create(Service::GetServiceName().c_str());
//In the words of my daughter: "OH, OH, FILE ALL GONE!"
Pipe pipe(L"\\\\" + *hostname + L"\\pipe\\dbg");
CloseHandle(remote_fh);
FILE_FLAG_DELETE_ON_CLOSE requires FILE_SHARE_DELETE permission. The problem with that is that when a process starts up, the operating system opens a file handle to the executable and requires NOT sharing the FILE_SHARE_DELETE permission so that you can't delete the executable file while the process is open.
Consider: you open a handle to an executable file without exclusive FILE_SHARE_DELETE, but say just FILE_GENERIC_READ. Someone else (you, later, or indeed another thread or process) then launches that executable. No problem because nobody's trying to delete the executable. But if you were then to try to gain FILE_SHARE_DELETE permission, you'd fail because the executable is already running with exclusive file delete permission.