I am writing / reading raw data on a SD card. The code for writing is working up to approx. 4700 blocks and fails after this limit. Here is the code:
//Data to be written
uint8_t* sessions;
sessions = (uint8_t *) malloc(2048*sizeof(uint8_t));
unsigned int i;
for(i=0;i<(2048*sizeof(uint8_t));i++) sessions[i]=8;
DWORD dwWrite;
HANDLE hDisk=CreateFileA("\\\\.\\K:", // drive to open = SD CARD
GENERIC_WRITE, // access to the drive
FILE_SHARE_READ | // share mode
FILE_SHARE_WRITE,
NULL, // default security attributes
OPEN_EXISTING, // disposition
FILE_FLAG_NO_BUFFERING, // file attributes
NULL); // do not copy file attributes
if(hDisk==INVALID_HANDLE_VALUE)
{
CloseHandle(hDisk);
printf("ERROR opening the file !!! ");
}
DWORD dwPtr = SetFilePointer(hDisk,10000*512,0,FILE_BEGIN); //4700 OK
if (dwPtr == INVALID_SET_FILE_POINTER) // Test for failure
{
printf("CANNOT move the file pointer !!! ");
}
//Try using this structure but same results: CAN BE IGNORED
OVERLAPPED osWrite = {0,0,0};
memset(&osWrite, 0, sizeof(osWrite));
osWrite.Offset = 10000*512; //4700 OK
osWrite.hEvent = CreateEvent(FALSE, FALSE, FALSE, FALSE);
if( FALSE == WriteFile(hDisk,sessions,2048,&dwWrite,&osWrite) ){
printf("CANNOT write data to the SD card!!! %lu",dwWrite);
}else{
printf("Written %lu on SD card",dwWrite);
}
CloseHandle(hDisk);
The issue is with the function "Writefile" (windows.h). If the number of block is less than 4700. everything is fine (data are written on the SD card) but if the block number is let's say 5000 or 10000, the function fails "Written 0".
Notice that without FILE_FLAG_NO_BUFFERING, no way to open the drive (SD card). The "OVERLAPPED" is a failed attempt to make it works, not using it (WriteFile(hDisk,sessions,2048,&dwWrite,NULL) )leads to the same behaviour. "SetFilePointer" works also for blocks higher than 4700. Have tested as well 2 different SD cards. I am on Windows 10.
Any hint as to what is happening?
Thank you for your input
From the documentation for WriteFile:
A write on a volume handle will succeed if the volume does not have a mounted file system, or if one of the following conditions is true:
The sectors to be written to are boot sectors.
The sectors to be written to reside outside of file system space.
You have explicitly locked or dismounted the volume by using FSCTL_LOCK_VOLUME or FSCTL_DISMOUNT_VOLUME.
The volume has no actual file system. (In other words, it has a RAW file system mounted.)
You are able to write to the first couple of megabytes because (for historical reasons) the file system doesn't use that space. In order to write to the rest of the volume, you'll first have to lock the volume using the FSCTL_LOCK_VOLUME control code.
You should pass Null as the 3rd parameter of SetFilePointer, lpDistanceToMoveHigh, unless you are using the higher order 32 bits of a 64-bit address. Also, if you are not using the OVERLAPPED structure, make sure to pass Null to WriteFile for that parameter.
Also, be sure that you are not having any overflows for the data types you are using. And, be mindful of the addressing limitations of the system you are working on.
MSDN WriteFile
MSDN SetFilePointer
Related
Quoted from MSDN entry for TransmitFile:
The maximum number of bytes that can be transmitted using a single call to the TransmitFile function is 2,147,483,646, the maximum value for a 32-bit integer minus 1. The maximum number of bytes to send in a single call includes any data sent before or after the file data pointed to by the lpTransmitBuffers parameter plus the value specified in the nNumberOfBytesToWrite parameter for the length of file data to send. If an application needs to transmit a file larger than 2,147,483,646 bytes, then multiple calls to the TransmitFile function can be used with each call transferring no more than 2,147,483,646 bytes. Setting the nNumberOfBytesToWrite parameter to zero for a file larger than 2,147,483,646 bytes will also fail since in this case the TransmitFile function will use the size of the file as the value for the number of bytes to transmit.
Alright. Sending a file of size 2*2,147,483,646 bytes (~ 4 GiB) with TransmitFile would then have to be divided into two parts at minimum (e.g. 2 GiB + 2 GiB in two calls to TransmitFile). But how exactly would one go about doing that, while preferably also keeping the underlying TCP connection alive in between?
When the file is indeed <=2,147,483,646 bytes in size, one could just write:
HANDLE fh = CreateFile(filename, GENERIC_READ, FILE_SHARE_READ, NULL,
OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL | FILE_FLAG_SEQUENTIAL_SCAN, NULL);
TransmitFile(SOCK_STREAM_socket, fh, 0, 0, NULL, NULL, TF_DISCONNECT);
to let Windows handle all the lower-level stuff (caching, chunking the data up into pieces for efficient transmission etc. However, unlike the comparable Linux sendfile() syscall, there is no immediately obvious offset argument in the call (although the fifth argument, LPOVERLAPPED lpOverlapped probably is exactly what I'm looking for). I suppose I could hack something together, but I'm also looking for a graceful, good practice Win32 solution from someone who actually knows about this stuff.
You can use the lpOverlapped parameter to specify a 64-bit offset within the file at which to start the file data transfer by setting the Offset and OffsetHigh member of the OVERLAPPED structure. If lpOverlapped is a NULL pointer, the transmission of data always starts at the current byte offset in the file.
So, for lack of a minimal example readily available on the net, which calls are necessary to accomplish such a task?
Managed to figure it out based on the comments.
So, if LPOVERLAPPED lpOverlapped is a null pointer, the call starts transmission at the current file offset of the file (much like the Linux sendfile() syscall and its off_t *offset parameter). This offset (pointer) can be manipulated with SetFilePointerEx, so one could write:
#define TRANSMITFILE_MAX ((2<<30) - 1)
LARGE_INTEGER total_bytes;
memset(&total_bytes, 0, sizeof(total_bytes));
while (total_bytes < filesize) {
DWORD bytes = MIN(filesize-total_bytes, TRANSMITFILE_MAX);
if (!TransmitFile(SOCK_STREAM_socket, fh, bytes, 0, NULL, NULL, 0))
{ /* error handling */ }
total_bytes.HighPart += bytes;
SetFilePointerEx(fh, total_bytes, NULL, FILE_BEGIN);
}
closesocket(SOCK_STREAM_socket);
to accomplish the task.
Not very elegant imo, but it works.
I am m trying to develop a mini benchmarking system in C++ and I have trouble measuring the HDD read and write speed. More exactly, the transfer speed measured by me is huge: 400-600 MB/s for read and above 1000 MB/s for write. I have a 5400 RPM hard disk drive (not SSD), the real read/write speed (according to a benchmarking program) is roughly about 60 MB/s.
//blockSize is 4096
//my data buffer
char* mydata = (char*)malloc(1*blockSize);
//initialized with random data
srand(time(NULL));
for(int i=0;i<blockSize;i++){
mydata[i] = rand()%256;
}
double startt, endt, difft;
int times = 10*25000;
int i=0,j=0;
DWORD written;
HANDLE f, g;
DWORD read;
f=CreateFileA(
"newfolder/myfile.txt",
GENERIC_WRITE,
0,
NULL,
CREATE_ALWAYS,
FILE_ATTRIBUTE_NORMAL,
NULL
);
if(f==INVALID_HANDLE_VALUE){
std::cout<<"Error openning for write.";
return -1;
}
startt = clock();
for(i=0;i<times;i++){
WriteFile(
f,
mydata,
blockSize,
&written,
NULL
);
}
endt = clock();
difft = 1.0*(endt-startt)/(1.0*CLOCKS_PER_SEC);
std::cout<<"\nWrite time: "<<difft;
std::cout<<"\nWrite speed: "<<1.0*times*blockSize/difft/1024/1024<<" MB/s";
CloseHandle(f);
//------------------------------------------------------------------------------------------------
g=CreateFile("newfolder/myfile.txt",
GENERIC_READ,
0,
NULL,
OPEN_ALWAYS,
FILE_ATTRIBUTE_NORMAL,
NULL
);
if(g==INVALID_HANDLE_VALUE){
std::cout<<"Error opening for read.";
return -1;
}
startt = clock();
for(i=0;i<times;i++){
ReadFile(
g,
mydata,
blockSize,
&read,
NULL
);
}
endt = clock();
difft = 1.0*(endt-startt)/(1.0*CLOCKS_PER_SEC);
std::cout<<"\nRead time:"<<difft;
std::cout<<"\nRead speed: "<<1.0*times*blockSize/difft/1024/1024<<" MB/s";
CloseHandle(g);
I tried using fopen and fwrite functions too and I got similar results.
I ran my app on another computer. The write speed was about right, the read speed was still huge.
The most interesting thing is that the application actually creates a 1GB file in about 2 seconds which corresponds to a 500 MB/s write speed.
Does anybody have any idea what am I doing wrong?
Technically, you are doing nothing wrong. The problem is, that every OS uses caching for all I/O operations. The HDD itself also caches some data, so it can perform them efficiently.
This question is very platform-specific. You would need to fool caching somehow.
Perhaps, you should look at this library: Bonnie++. You may find it useful. It was written for Unix systems, but source code could reveal some useful techniques.
On Windows, based on this resource, additional flag FILE_FLAG_NO_BUFFERING passed to CreateFile function should be enough to disable buffering for this file.
Quote:
In these situations, caching can be turned off. This is done at the time the file is opened by passing FILE_FLAG_NO_BUFFERING as a value for the dwFlagsAndAttributes parameter of CreateFile. When caching is disabled, all read and write operations directly access the physical disk. However, the file metadata may still be cached. To flush the metadata to disk, use the FlushFileBuffers function.
You are measuring the performance of cache.
Try storing a lot more data than that, once the cache fills the data should be written straight to the disk.
I think I have figured it out.
Unbuffered file writing speed depends on the size of data the WriteFile function is writing. My experiments show that the bigger the data size, the higher the writing speed. For large amounts of data (>1MB) it even outperforms buffered writing, which I was able to measure by writing data larger than 2GB.
To summarize, one can measure the hard drive writing speed accurately by:
Opening the file using CreateFile and setting the FILE_FLAG_NO_BUFFERING flag.
Writing a lot of data at a time, using WriteFile.
I have a problem with win32 I/O performance:
I'm trying to achieve a decent writing speed using OpenFile/WriteFile.
Using Resource Monitor (it comes with windows) I measured the writing speed of the following piece of code and I found that it is writing at 2MB/sec...
HANDLE hFile = INVALID_HANDLE_VALUE;
hFile = CreateFile(
L"test",
(GENERIC_READ | GENERIC_WRITE),
FILE_SHARE_READ,
NULL,
OPEN_ALWAYS,
(FILE_ATTRIBUTE_NORMAL |
FILE_FLAG_WRITE_THROUGH |
FILE_FLAG_NO_BUFFERING),
NULL);
if (hFile != INVALID_HANDLE_VALUE)
{
//OK
unsigned long bytesWritten = 0;
unsigned long* Buffer = (unsigned long*)malloc(4096*sizeof(unsigned long));
ZeroMemory(Buffer, 4096); //thanks to 'bash.d'
while (true)
{
/*the infinite loop is intentional
because I wanted to see if the writing speed of 2MB/sec
was right */
WriteFile(hFile,
Buffer,
4096,
&bytesWritten,
NULL);
if (bytesWritten <= 0)
{
break;
}
}
}
I tried with the following and it's the same...
hFile = CreateFile(
L"test",
(GENERIC_READ | GENERIC_WRITE),
FILE_SHARE_READ,
NULL,
OPEN_ALWAYS,
(FILE_ATTRIBUTE_NORMAL);
What am I doing wrong(about the writing speed) ? and how can I improve the writing speed ?
Thank you and sorry for my english
Edit:
I'm writing on a local disk
This is very interesting, and similar to an issue I have, and can reproduce on 2 different servers with Windows Server 2003 SP2 64-bit (single hard drives, not RAID). Simply doing a WriteFile() of 36 bytes and then 99964 bytes in a loop produces similar behavior (I'm guessing it would be the same with a single write, and some other versions of Windows; that's just what I happened to be using). CPU usage starts off very low, and then increases gradually -- on one server, the test was around 50% CPU usage at around 175GB (about 95% of that is kernel time; 60% in my program and 40% in 'System').
You may also try async IO to get the test performance. That is opening the file with FILE_FLAG_OVERLAPPED and using the LPOVERLAPPED argument of WriteFile. You may or may not get better performance with FILE_FLAG_NO_BUFFERING. You will have to test to see.
FILE_FLAG_NO_BUFFERING will generally give you more consistent speeds and better streaming behavior, and it avoids polluting your disk cache with data that you may not need again, but it isn't necessarily faster overall.
You should also test to see what the best size is for each block of IO. In my experience There is a huge performance difference between copying a file 4k at a time and copying it 1Mb at a time.
In my past testing of this (a few years ago) I found that block sizes below about 64kB were dominated by overhead, and total throughput continued to improve with larger block sizes up to about 512KB. I wouldn't be surprised if with today's drives you needed to use block sizes larger than 1MB to get maximum throughput.
The numbers you are currently using appear to be reasonable, but may not be optimal. Also I'm fairly certain that FILE_FLAG_WRITE_THROUGH prevents the use of the on-disk cache and thus will cost you a fair bit of performance.
It would be of worth to try below thing...
1) Enabling FILE_FLAG_SEQUENTIAL_SCAN flag
2) "Enable advanced performance" in "Disk Policies" in the Device Manager
3) Varying disk chunk size from 64 KB to 4096 ...
4) Try FILE_FLAG_NO_BUFFERING
Use async IO bound to a completion port
Pre-grow the file using SetFileValidData
Open the handle with FILE_FLAG_NO_BUFFERING | FILE_FLAG_WRITE_THROUGH
A consumer grade drive (even 5400RPM) should be able to write ~130MB/sec (single spindle, no raid). No other IO should occur at the same time (no head movement).
See https://github.com/rusanu/writing-a-binary-file-in-c-very-fast for an example.
The relevant Microsoft doc is:
Blocking Direct Write Operations to Volumes and Disks
CreateFile, remarks on Physical Disks and Volumes
The executable is written in C++ and it calls CreateFile() to open an SD card that has no filesystem. The CreateFile() and consecutive ReadFile() calls are successful for GENERIC_READ without Administrator privileges.
CreateFile fails for GENERIC_WRITE even with Administrator privileges. In the explorer, I set Run as Administrator under Properties > Compatibility > Privilege Level. I also tried to run the executable from an Administrator cmd (started with Ctrl+Shift+Enter, "Administrator:" is in the window title, properly elevated). Still, I get ERROR_ACCESS_DENIED (0x5).
Do I have to pass something else to CreateFile? I have no idea what security attributes are, I just pass NULL, relevant code is here at line 92, and here at line 48.
Or is there anything else that should be set to run the process with Administrator privileges?
A related questions:
Can I get write access to raw disk sectors under Vista and Windows 7 in user mode?
Raw partition access in Windows Vista
How to obtain direct access to raw HD data in C?
Is there a clean way to obtain exclusive access to a physical partition under Windows?
While the answer of #MSalters makes sense, it is not how my code works. In fact it is so counter-intuitive, I spent several days making sure the code does in fact work.
These code snippets are in a proven, mass consumer market software product. When it needs to modify an on-disk structure, it dismounts the win32 volume so it can modify NTFS or FAT filesystem structures. Interestingly, the volume access handle is read-only:
char fn [30];
snprintf (fn, sizeof fn, "\\\\.\\%s:", vol -> GetVolName ());
vol_handle = CreateFile (fn, GENERIC_READ,
FILE_SHARE_READ | FILE_SHARE_WRITE, NULL,
OPEN_EXISTING,
FILE_FLAG_NO_BUFFERING | FILE_FLAG_RANDOM_ACCESS,
NULL);
if (vol_handle == INVALID_HANDLE_VALUE)
{
// show error message and exit
}
If unable to get write access to a volume or partition, this code forces a dismount if the user authorizes such after a stern warning:
if (!DeviceIoControl (vol_handle, FSCTL_DISMOUNT_VOLUME,
NULL, 0, NULL, 0, &status, NULL))
{
DWORD err = GetLastError ();
errormsg ("Error %d attempting to dismount volume: %s",
err, w32errtxt (err));
}
// lock volume
if (!DeviceIoControl (vol_handle, FSCTL_LOCK_VOLUME,
NULL, 0, NULL, 0, &status, NULL))
{
// error handling; not sure if retrying is useful
}
Writing is then fairly straightforward, except for positioning the file pointer by 512-byte sector:
long hipart = sect >> (32-9);
long lopart = sect << 9;
long err;
SetLastError (0); // needed before SetFilePointer post err detection
lopart = SetFilePointer (vol_handle, lopart, &hipart, FILE_BEGIN);
if (lopart == -1 && NO_ERROR != (err = GetLastError ()))
{
errormsg ("HWWrite: error %d seeking drive %x sector %ld: %s",
err, drive, sect, w32errtxt (err));
return false;
}
DWORD n;
if (!WriteFile (vol_handle, buf, num_sects*512, &n, NULL))
{
err = GetLastError ();
errormsg ("WriteFile: error %d writing drive %x sectors %lu..%lu: %s",
err, drv, sect, sect + num_sects - 1,
w32errtxt (err));
return false;
}
It's quite rare to want only GENERIC_WRITE. You most likely want GENERIC_READ|GENERIC_WRITE.
There is note in MSDN in documentation of CreateFile:
Direct access to the disk or to a volume is restricted. For more information, see "Changes to the file system and to the storage stack to restrict direct disk access and direct volume access in Windows Vista and in Windows Server 2008" in the Help and Support Knowledge Base at http://support.microsoft.com/kb/942448.
It refers to Vista/2008, but maybe apply to Win7 also.
I had a similar issue when porting from x86 to x64 code. You mention that you are passing null for your SECURITY_ATTRIBUTES parameter; I was getting access-denied errors myself using this approach until I actually began creating/passing this parameter.
I'm writing an SDL application for Linux, that works from the console (no X server). One function I have is a file copy mechanism, that copies specific files from HDD to USB Flash device, and showing progress of this copy in the UI. To do this, I'm using simple while loop and copying file by 8kB chunks to get copy progress. The problem is, that it's slow. I get to copy a 100 MB file in nearly 10 minutes, which is unacceptable.
How can I implement faster file copy? I was thinking about some asynchronous API that would read file from HDD to a buffer and store the data to USB in separate thread, but I don't know if I should implement this myself, because it doesn't look like an easy task. Maybe you know some C++ API/library that can that for me? Or maybe some other, better method?
Don't synchronously update your UI with the copy progress, that will slow things down considerably. You should run the file copy on a separate thread from the main UI thread so that the file copy can proceed as fast as possible without impeding the responsiveness of your application. Then, the UI can update itself at the natural rate (e.g. at the refresh rate of your monitor).
You should also use a larger buffer size than 8 KB. Experiment around, but I think you'll get faster results with larger buffer sizes (e.g. in the 64-128 KB range).
So, it might look something like this:
#define BUFSIZE (64*1024)
volatile off_t progress, max_progress;
void *thread_proc(void *arg)
{
// Error checking omitted for expository purposes
char buffer[BUFSIZE];
int in = open("source_file", O_RDONLY);
int out = open("destination_file", O_WRONLY | O_CREAT | O_TRUNC);
// Get the input file size
struct stat st;
fstat(in, &st);
progress = 0;
max_progress = st.st_size;
ssize_t bytes_read;
while((bytes_read = read(in, buffer, BUFSIZE)) > 0)
{
write(out, buffer, BUFSIZE);
progress += bytes_read;
}
// copy is done, or an error occurred
close(in);
close(out);
return 0;
}
void start_file_copy()
{
pthread_t t;
pthread_create(&t, NULL, &thread_proc, 0);
}
// In your UI thread's repaint handler, use the values of progress and
// max_progress
Note that if you are sending a file to a socket instead of another file, you should instead use the sendfile(2) system call, which copies the file directly in kernel space without round tripping into user space. Of course, if you do that, you can't get any progress information, so that may not always be ideal.
For Windows systems, you should use CopyFileEx, which is both efficient and provides you a progress callback routine.
Let the OS do all the work:
Map the file to memory: mmap, will drastically speed up the reading process.
Save it to a file using msync.