I had the following code which got executed every two
minutes all day long:
int sucessfully_deleted = DeleteFile(dest_filename);
if (!sucessfully_deleted)
{
// this never happens
}
rename(source_filename,dest_filename);
Once every several hours the rename() would fail with errno=13 (EACCES). The files involved were all sitting on a DropBox directory and I had a hunch that DropBox could be the cause. I figured that it might just be possible that the DeleteFile() function may return with a non-zero successfully_deleted but actually DropBox could still be busy doing some stuff in relation to the deletion that prevented rename() from succeeding. What I did next was to change rename() to my_rename() which would attempt a rename() and upon any failure would Sleep() for one second and try a second time. Sure enough that has worked perfectly ever since. What's more, I get a diagnostic message displaying first-attempt-failures every several hours. It has never failed on the second attempt.
So you could say that the problem is entirely solved... but I would like to understand what might be going on so as to better defend myself against any related DropBox issues in the future...
Really I would like to have a new super_delete() function which does not return until the file is properly deleted and finished with in all respects.
under windows request to delete file really never delete file just. it mark it FCB (File Control Block) with special flag (FCB_STATE_DELETE_ON_CLOSE). real deletion will be only when the last file handle will be closed.
The DeleteFile function marks a file for deletion on close. Therefore,
the file deletion does not occur until the last handle to the file is
closed. Subsequent calls to CreateFile to open the file fail with
ERROR_ACCESS_DENIED.
also if exist section ( memory-mapped file ) open on file - file even can not be marked for delete. api call fail with STATUS_CANNOT_DELETE. so in general impossible always delete file.
in case exist another open handles for file (but not section !) begin from windows 10 rs1 exist new functional for delete - FileDispositionInformationEx with FILE_DISPOSITION_POSIX_SEMANTICS. in this case:
Normally a file marked for deletion is not actually deleted until all
open handles for the file have been closed and the link count for the
file is zero. When marking a file for deletion using
FILE_DISPOSITION_POSIX_SEMANTICS, the link gets removed from the visible namespace as soon as the POSIX delete handle has been closed,
but the file’s data streams remain accessible by other existing
handles until the last handle has been closed.
ULONG DeletePosix(PCWSTR lpFileName)
{
HANDLE hFile = CreateFileW(lpFileName, DELETE, FILE_SHARE_VALID_FLAGS, 0, OPEN_EXISTING,
FILE_FLAG_BACKUP_SEMANTICS|FILE_FLAG_OPEN_REPARSE_POINT, 0);
if (hFile == INVALID_HANDLE_VALUE)
{
return GetLastError();
}
static FILE_DISPOSITION_INFO_EX fdi = { FILE_DISPOSITION_DELETE| FILE_DISPOSITION_POSIX_SEMANTICS };
ULONG dwError = SetFileInformationByHandle(hFile, FileDispositionInfoEx, &fdi, sizeof(fdi))
? NOERROR : GetLastError();
// win10 rs1: file removed from parent folder here
CloseHandle(hFile);
return dwError;
}
Update
Sorry i didn't get the question correctly the first time. I thought DeleteFile returned error 13.
Now I understand that DeleteFile succeeds but rename fails immediatlely after.
It could be just a sync issue with the filesystem. After calling DeleteFile the file will be deleted when the OS commits the changes to the filesystem. That may not appen immediately.
If you need to perform multiple operations to the same path, you should have a look at transactions https://learn.microsoft.com/it-it/windows/desktop/api/winbase/nf-winbase-deletefiletransacteda.
-- OLD ANSWER --
That is correct. If the another application handles to that file, DeleteFile will fail.
Citing MSDN docs https://learn.microsoft.com/en-us/windows/desktop/api/winbase/nf-winbase-deletefile :
The DeleteFile function fails if an application attempts to delete a file that has other handles open for normal I/O or as a memory-mapped file (FILE_SHARE_DELETE must have been specified when other handles were opened).
This applies to dropbox, the antivirus, or in general, any other application that may open those files.
Dropbox may open the file to compute its hash (to look for changes) at any moment. Same goes with the antivirus.
Related
I observe pretty strange behavior in my application.
I have following functions:
bool File::Exists(const std::string & Path)
{
struct stat S;
if(stat(Path.c_str(), &S) != 0)
return false;
if(!S_ISREG(S.st_mode))
return false;
return true;
}
void File::Remove(const std::string & Path)
{
if(unlink(Path.c_str()) != 0)
throw Exceptions::Exception(EXCEPTION_PARAMS, errno);
}
The code i use is:
...
const std::string Path = ...;
if(File::Exists(Path))
File::Remove(Path);
...
And at this point exception is thrown:
No such file or directory (2)
Key facts:
Happens once every 1k-10k calls
All files being removed are binary files, about 20MB
This is called in application thread, however this is the only thread accessing those files (no other threads / process access them). Even no other process / user access partition they are located on.
File being removed is located on mounted CIFS (SMB) (Network) mount point
Why does stat() report file being present, but unlink() sometimes fails?
Why does stat() report file being present, but unlink() sometimes fails?
Because the calls don't happen at the same time.
This bug is so common it has its own name and even a Wikipedia page:
In software development, time of check to time of use (TOCTTOU or
TOCTOU, pronounced "TOCK too") is a class of software bug caused by
changes in a system between the checking of a condition (such as a
security credential) and the use of the results of that check. This is
one example of a race condition.
How intensively are these calls being made? Are you trying to delete thousands of files at a time? Or one or two a second/minute? Over a network you could get all sorts of timeouts that could be interpreted as "No such file or directory", but in actuality is another problem.
And if you're about to delete them anyway, why bother checking for their existence at all? Just delete them, and check the exception to find out if it didn't exist, or whether there was a different error - or not even do that. Either they're gone, or you can't delete them anyway...
PROBLEM HISTORY:
Now I use Windows Media Player SDK 9 to play AVI files in my desktop application. It works well on Windows XP but when I try to run it on Windows 7 I caught an error - I can not remove AVI file immediately after playback. The problem is that there are opened file handles exist. On Windows XP I have 2 opened file handles during the playing file and they are closed after closing of playback window but on Windows 7 I have already 4 opened handles during the playing file and 2 of them remain after the closing of playback window. They are become free only after closing the application.
QUESTION:
How can I solve this problem? How to remove the file which has opened handles? May be exists something like "force deletion"?
The problem is that you're not the only one getting handles to your file. Other processes and services are also able to open the file. So deleting it isn't possible until they release their handles. You can rename the file while those handles are open. You can copy the file while those handles are open. Not sure if you can move the file to another container, however?
Other processes & services esp. including antivirus, indexing, etc.
Here's a function I wrote to accomplish "Immediate Delete" under Windows:
bool DeleteFileNow(const wchar_t * filename)
{
// don't do anything if the file doesn't exist!
if (!PathFileExistsW(filename))
return false;
// determine the path in which to store the temp filename
wchar_t path[MAX_PATH];
wcscpy_s(path, filename);
PathRemoveFileSpecW(path);
// generate a guaranteed to be unique temporary filename to house the pending delete
wchar_t tempname[MAX_PATH];
if (!GetTempFileNameW(path, L".xX", 0, tempname))
return false;
// move the real file to the dummy filename
if (!MoveFileExW(filename, tempname, MOVEFILE_REPLACE_EXISTING))
{
// clean up the temp file
DeleteFileW(tempname);
return false;
}
// queue the deletion (the OS will delete it when all handles (ours or other processes) close)
return DeleteFileW(tempname) != FALSE;
}
Technically you can delete a locked file by using MoveFileEx and passing in MOVEFILE_DELAY_UNTIL_REBOOT. When the lpNewFileName parameter is NULL, the Move turns into a delete and can delete a locked file. However, this is intended for installers and, among other issues, requires administrator privileges.
Have you checked which application is still using the avi file?
you can do this by using handle.exe. You can try deleting/moving the file after closing the process(es) that is/are using that file.
The alternative solution would be to use unlocker appliation (its free).
One of the above two method should fix your problem.
Have you already tried to ask WMP to release the handles instead? (IWMPCore::close seems to do that)
Right now, I am facing a new problem that I can't figure out how to fix. I have two files. One is a video file and other is a thumbnail. They have same name. I want to rename these two files using C++. I am using the rename function and it works. This is what I've written:
if(rename(videoFile) == 0)
{
if(rename(thumbnail) != 0)
{
printf("Fail rename \n");
}
}
The problem occurs when the video file is renamed successfully but for some reason the thumbnail can't be renamed. When this happens, I would like to rollback the renaming of the video file because the video file name and the thumbnail file name should be the same in my program. What I want to do is to rename after both files are okay to rename. Please guide me, any design pattern for function like rollback or third party software.
There is no absolutely foolproof way to do this.
Fundamental rule of disk I/O: The filesystem can change at any time. You can't check whether a rename would succeed; your answer is already wrong. You can't be certain that undoing the rename will succeed; somebody else might have taken the name while you briefly weren't using it.
On systems that support hard links, you can use them to get about 90% of the way there, assuming you're not moving between filesystems. Suppose you're renaming A to B and C to D. Then do these things:
Create hard link B which links to A. This is written as link("A", "B") in C, using the Unix link(2) system call. Windows users should call CreateHardLink() instead.
If (1) succeeded, create hard link D which links to C. Otherwise, return failure now.
If (2) succeeded, delete A and C and return success. Otherwise, delete B and return failure. If the deletions fail, there is no obvious means of recovery. In practice, you can probably ignore failed deletions assuming the reason for failure was "file not found" or equivalent for your platform.
This is still vulnerable to race conditions if someone deletes one of the files out from under you at the wrong time, but that is arguably not an issue since it is largely equivalent to the rename failing (or succeeding) and then the person deleting the file afterwards.
Technically, you should also be opening the containing directory (in O_RDONLY mode) and fsync(2)'ing it after each operation, at least under Unix. If moving between directories, that's both the source and the destination directories. In practice, nobody does this, particularly since it will lead to degraded performance under ext3. Linus takes the position that the filesystem ought to DTRT without this call, but it is formally required under POSIX. As for Windows, I've been unable to find any authoritative reference on this issue on MSDN or elsewhere. So far as I'm aware, Windows does not provide an API for synchronizing directory entries (you can't open() a directory, so you can't get a file descriptor suitable to pass to fsync()).
Nitpick: To some extent, this sort of thing can be done perfectly on transactional filesystems, but just about the only one in common use right now is NTFS, and Microsoft specifically tells developers not to use that feature. If/when btrfs hits stable, transactions might become genuinely useful.
On Windows platform starting from Vista, you can use code such as the following.
#include "KtmW32.h"
bool RenameFileTransact( LPCTSTR lpctszOldVideoFile, LPCTSTR lpctszNewVideoFile, LPCTSTR lpctszOldThumbnailFile, LPCTSTR lpctszNewThumbnailFile )
{
bool bReturn = false;
HANDLE hRnameTransaction = CreateTransaction(NULL, NULL, 0, 0, 0, 0, NULL);
if (MoveFileTransacted(lpctszOldVideoFile, lpctszNewVideoFile, NULL, NULL, 0, hRnameTransaction) &&
MoveFileTransacted(lpctszOldThumbnailFile, lpctszNewThumbnailFile, NULL, NULL, 0, hRnameTransaction))
{
if ( CommitTransaction(hRnameTransaction))
{
bReturn = true;
}
}
CloseHandle( hRnameTransaction );
return bReturn;
}
But as #Kevin pointed out above, Microsoft discourages the usage of this good feature.
I shouldnt be able to delete a file with an open handle, correct? So i create a file, then i straight away try to delete it, expecting this to fail. Or am i wrong and the handle doesnt have to be closed before deleting the file?
HANDLE hFile = CreateFile (TEXT(file),
GENERIC_WRITE,
0,
NULL,
OPEN_ALWAYS,
FILE_ATTRIBUTE_NORMAL,
NULL);
if (hFile == INVALID_HANDLE_VALUE)
{
//FAIL
}
if(DeleteFile(file))
{
//Should it ever get here cos i dont close the handle?
}
It depends on how the file has been opened. If the share mode has FILE_SHARE_DELETE specified, then it may be deleted by others.
Even if you memory map the file, and it has been opened with this flag (and read/write sharing), then it can still be deleted by the shell (at least I've tried this and it happens, but perhaps the file has simply been renamed and moved to the recycle bin). In such cases, subsequently accessing the memory will result in an 'InPageError' C-style exception.
Yes, it would fail.
The DeleteFile function fails if an application attempts to delete a
file that is open for normal I/O or as a memory-mapped file.
Have you tried this? MS documentation states that:
The DeleteFile function fails if an application attempts to delete a file that is open for normal I/O or as a memory-mapped file.
So if you're not getting that behaviour I'd suggest it's down to the way you've opened the file. Are you sure that your check on whether the file is open is completely comprehensive?Have you tried writing to the file first? Can you see the file outside of your own code? (i.e. from Explorer) Look here for more details.
PROBLEM HISTORY:
Now I use Windows Media Player SDK 9 to play AVI files in my desktop application. It works well on Windows XP but when I try to run it on Windows 7 I caught an error - I can not remove AVI file immediately after playback. The problem is that there are opened file handles exist. On Windows XP I have 2 opened file handles during the playing file and they are closed after closing of playback window but on Windows 7 I have already 4 opened handles during the playing file and 2 of them remain after the closing of playback window. They are become free only after closing the application.
QUESTION:
How can I solve this problem? How to remove the file which has opened handles? May be exists something like "force deletion"?
The problem is that you're not the only one getting handles to your file. Other processes and services are also able to open the file. So deleting it isn't possible until they release their handles. You can rename the file while those handles are open. You can copy the file while those handles are open. Not sure if you can move the file to another container, however?
Other processes & services esp. including antivirus, indexing, etc.
Here's a function I wrote to accomplish "Immediate Delete" under Windows:
bool DeleteFileNow(const wchar_t * filename)
{
// don't do anything if the file doesn't exist!
if (!PathFileExistsW(filename))
return false;
// determine the path in which to store the temp filename
wchar_t path[MAX_PATH];
wcscpy_s(path, filename);
PathRemoveFileSpecW(path);
// generate a guaranteed to be unique temporary filename to house the pending delete
wchar_t tempname[MAX_PATH];
if (!GetTempFileNameW(path, L".xX", 0, tempname))
return false;
// move the real file to the dummy filename
if (!MoveFileExW(filename, tempname, MOVEFILE_REPLACE_EXISTING))
{
// clean up the temp file
DeleteFileW(tempname);
return false;
}
// queue the deletion (the OS will delete it when all handles (ours or other processes) close)
return DeleteFileW(tempname) != FALSE;
}
Technically you can delete a locked file by using MoveFileEx and passing in MOVEFILE_DELAY_UNTIL_REBOOT. When the lpNewFileName parameter is NULL, the Move turns into a delete and can delete a locked file. However, this is intended for installers and, among other issues, requires administrator privileges.
Have you checked which application is still using the avi file?
you can do this by using handle.exe. You can try deleting/moving the file after closing the process(es) that is/are using that file.
The alternative solution would be to use unlocker appliation (its free).
One of the above two method should fix your problem.
Have you already tried to ask WMP to release the handles instead? (IWMPCore::close seems to do that)