I have written a function to test for the readability/writability of a folder.
For unit testing it, I need to produce the different cases:
a folder with readable and writable files
a folder with readable files (not writable)
a folder not writable and not readable.
Here is the code for the function I came to, up to this point:
void FileUtils::checkPath(std::string path, bool &readable, bool &writable)
{
namespace bf = boost::filesystem;
std::string filePath = path + "/test.txt";
// remove a possibly existing test file
remove(filePath.c_str());
// check that the path exists
if(!bf::is_directory(path))
{
readable = writable = false;
return;
}
// try to write in the location
std::ofstream outfile (filePath.c_str());
outfile << "I can write!" << std::endl;
outfile.close();
if(!outfile.fail() && !outfile.bad())
{
writable = true;
}
// look for a file to read
std::ifstream::pos_type size;
char * memblock;
for (bf::recursive_directory_iterator it(path); it != bf::recursive_directory_iterator(); ++it)
{
if (!is_directory(*it))
{
std::string sFilePath = it->path().string();
std::ifstream file(sFilePath.c_str(), std::ios::in|std::ios::binary|std::ios::ate);
if (file.is_open())
{
size = file.tellg();
if(size > 0)
{
memblock = new char [1];
file.seekg (0, std::ios::beg);
file.read (memblock, 1);
file.close();
delete[] memblock;
if(!file.fail() && !file.bad())
{
readable = true;
}
break;
}
}
else
{
// there is a non readable file in the folder
// readable = false;
break;
}
}
}
// delete the test file
remove(filePath.c_str());
}
Now with the tests (done with Google tests):
TEST_F(FileUtilsTest, shouldCheckPath)
{
// given an existing folder
namespace fs = boost::filesystem;
fs::create_directory("/tmp/to_be_deleted");
bool readable = false, writable = false;
FileUtils::checkPath("/tmp/to_be_deleted",readable, writable);
fs::boost::filesystem::remove_all("/tmp/to_be_deleted");
EXPECT_TRUE(readable && writable);
}
I will add more for the other cases when I will have gone further.
Now the game is open to propose a better solution :-)
The foolproof way to check permissions is to literally check the file mode. In the case of directory permissions, the meaning of "readable" and "writable" might be surprising:
Read - allows you to list the contents of the directory
Write - allows you to create, rename, delete files from the directory, essentially modifying the list of contents (also requires execute)
Execute - allows you to access (both read and write) and change properties of files within the directory
So if you have a directory with just the execute bit set, you can still read and write to the files inside. By turning the execute bit off, you can disable access to the files. As far as the contained files are concerned, the most you can figure out from the directory permissions is:
--x or r-x: existing files can be read and written to
-wx or rwx: existing files can be read and written to, files can be created, renamed and deleted
otherwise: you have no access to the files at all
To determine if a file is readable but not writeable (or vice versa) you need to check the permissions of the file itself. The directory can only tell you if the files can be accessed in general.
You can use stat() or access() (see BЈовић's comment) to find out the permissions for a file or directory. Since you're already using boost, you can also use boost::filesystem::status() which simply wraps stat().
To be portable and correct, the only way to test for readability/writability of a file/directory is to read/write from/to it. Permission models can be quite complex and non-portable (ACLs for example), so you can't simply check the permissions on the parent directory. Also, checking, and then subsequently trying to write is a race condition as the permissions could change between the check and the write.
If instead what you want is a high probability that a write will succeed, such as if you're letting the user choose a scratch folder for your application, just try writing a file and then delete it afterwords. This lets you know that at the time of user selection the directory was writable.
To be robust, always assume that filesystem operations are going to fail and design so that when they do, something logical will happen instead of a crash. In particular, design a system so that a user can figure out where the permission error is -- as there's a myriad of ways permissions can be set wrong, helpful error messages go a long way.
Related
I am working on a personal project, one of the first steps that my program executes is to create some folders and files when the user applies a command through the console.
For example. The user executes the command emi -start through the console, this triggers the creation of the following files and folders:
emi_base
emi_database
emi_main_db_file.csv
emi_catch_db_file.csv
emi_temp_db_file.csv
emi_version
emi_main_version_folder
emi_catch_version_folder
emi_temp_version_folder
emi_config
emi_config_file.txt
emi_ignore_file.txt
emi_repo_file.txt
For this to be possible I use a code similar to this:
// Creating folder emi_base
bool emi_directory_created = fs::create_directory(emi_default_path);
// Creating folder emi_database and its corresponding files
bool db_directory_created = fs::create_directory(db_default_path);
ofstream db_main_file_ostrm(db_main_file, std::ios::out | std::ios::binary);
ofstream db_catch_file_ostrm(db_catch_file, std::ios::out | std::ios::binary);
ofstream db_temp_file_ostrm(db_temp_file, std::ios::out | std::ios::binary);
// Creating folder emi_version and its corresponding sub-folders
bool version_directory_created = fs::create_directory(version_default_path);
bool version_main_directory_created = fs::create_directory(version_main_folder);
bool version_catch_directory_created = fs::create_directory(version_catch_folder);
bool version_temp_directory_created = fs::create_directory(version_temp_folder);
// Creating folder emi_config and its corresponding files
bool config_directory_created = fs::create_directory(config_default_path);
if (emi_directory_created &&
db_directory_created &&
db_main_file_ostrm &&
db_catch_file_ostrm &&
db_temp_file_ostrm &&
version_directory_created &&
version_main_directory_created &&
version_catch_directory_created &&
version_temp_directory_created &&
config_directory_created) {
// Code that is executed after validating that all files and folders were created correctly.
}
Well, for now everything works correctly, however, I do not know if that is the best way to validate that the folders and the respective files have been created correctly, it seems to me that putting all those Booleans in a conditional is not the right thing to do.
Is there a better way to do this? Is there a better way to validate the creation of multiple folders and files at once or in sequence?
Hope someone can give this newbie some feedback :)
If fs::create_directory is std::filesystem::create_directory, then fs::create_directory(<path>) throws an exception if the directory cannot be created, you don't need to check its return value:
The overload that does not take a std::error_code& parameter throws filesystem_error on underlying OS API errors, constructed with p as the first path argument and the OS error code as the error code argument.
But you do need to check that the files were opened successfully with ofstream::is_open call. You may like to create a helper function that opens an ofstream and checks whether that succeeded and then return ofstream, in C++11 ofstream is moveable and hence can be returned by value. E.g.:
std::ofstream open_ofstream(char const* filename) {
std::ofstream f(filename, std::ios::out | std::ios::binary);
if(!f.is_open())
throw std::system_error{errno, std::system_category(), filename}; // Hoping that std::ofstream didn't mangle errno.
// or throw std::filesystem::filesystem_error{"failed to open", filename, std::error_code{errno, std::system_category()}};
return f;
}
If all your directories and files reside in one top-level directory, you may like to create it with a different name first, create the directory/file structure in it and initialize the files, and then rename the top-level directory to its final name. So that when the top-level directory exists that means that it is correctly initialized and is ready for use.
I think better way is throw an exception in create_directory funciton if folder creation fail. So you dont need to check every create_directory call. Just one try-catch will be enough.
How to avoid TOCTOU(time-of-check, time-of-use) race condition for race condition between stat and rename for LOGFILE ?
Required to move the log file after its size value exceeds the max size.
result = stat(LOGFILE, & data);
if (result != 0) {
// stat failed
// file probably does not exist
} else if (data.st_size > MAX_LOGSIZE) {
unlink(PREV_LOGFILE);
(void) rename(LOGFILE, PREV_LOGFILE);
}
The standard way to avoid TOCTTOU on file operations is to open the file once and then do everything that you need through the file descriptor rather than the file name.
However, both renaming and unlinking a file require its path (because they need to know what link to rename or remove), so you can't use that approach here. An alternative might be to copy the file's contents elsewhere and then truncate it to zero bytes, although your scenario with log files probably requires that the operation be atomic, which may be difficult to achieve. Another approach is to require tight access controls on the directory: if an attacker cannot write to the directory, then it cannot play TOCTTOU games with your process. You can use unlinkat and renameat to restrict your paths to a specific directory's file descriptor so that you don't need to worry about the directory itself changing.
Something like this untested code might do the job, assuming a POSIX-like platform:
dirfd = open(LOGDIR, O_DIRECTORY);
// check for failure
res = fstatat(dirfd, LOGFILE, statbuf, AT_SYMLINK_NOFOLLOW);
if ((0 == res) && (S_ISREG(statbuf) && (data.st_size > MAX_LOGSIZE)) {
unlinkat(dirfd, PREV_LOGFILE, 0);
renameat(dirfd, LOGFILE, dirfd, PREV_LOGFILE);
}
close(dirfd);
Have a query. I am always using the code in C++ (below) to move a file from one location to another in same drive (call it A drive)
rename(const char* old_filename, const char* new_filename);
Recently I need to amend the code to move it to another drive (call in B-Drive). It doesn't work but I could write code to write into that particular drive (B-drive). On investigation, I found that the drive (A-drive) on which I produce the result(the old file) is in ext4 file system but the drive i am writing/moving to is in NTFS (fuseblk)
How can i amend my code to move the file to NTFS. I am using C++ in ubuntu
Regards
--------------------------------------------------------------------
New Edit after heeding call from user4581301
This is the code I have written
int main()
{
std::string dirinADrive = "/home/akaa/data/test3/test_from.txt"; // this is the parent directory
std::string dirinBDrive = "/media/akaa/Data/GIRO_repo/working/data/test5/test_to.txt"; // this is where i want to write to
std::string dirinCDrive = "/home/akaa/data/test3/test_to.txt"; // this is where i want to write to
std::string dirinDDrive = "/media/akaa/Data/GIRO_repo/working/data/test5/test_to_write.txt";
bool ok1{std::ofstream(dirinADrive).put('a')}; // create and write to file
bool ok2{std::ofstream(dirinDDrive).put('b')}; // create and write to file
if (!(ok1 && ok2))
{
std::perror("Error creating from.txt");
return 1;
}
if (std::rename(dirinADrive.c_str(), dirinCDrive.c_str())) // moving file to same drive
{
std::perror("Error renaming local");
return 1;
}
if (std::rename(dirinADrive.c_str(), dirinBDrive.c_str())) // moving file to other drive
{
std::perror("Error renaming other");
return 1;
}
std::cout << std::ifstream(dirinBDrive).rdbuf() << '\n'; // print file
}
And I have gotten an error
Error renaming other: Invalid cross-device link
So what is invalid cross-device link??
Thanks
You can’t use rename across filesystems, because the data must be copied (and having a single system call do an arbitrary amount of work is problematic even without atomicity issues). You really do have to open the source file and destination file and write the contents of one to the other. Apply whatever attributes you want to preserve (e.g., with stat and chmod), then delete the source file if you want.
In C++17, much of this has been packaged as std::filesystem::copy_file. (There is also std::filesystem::rename, but it’s no better than std::rename for this case.)
I have a folder to which files are copied. I want to watch it and process files as soon as they are copied to the directory. I can detect when a file is in the directory, whether through polling (my current implementation) or in some tests using Windows APIs from a few samples I've found online.
The problem is that I detect when the file is first created and its still being copied. This makes my program, that needs to access the file, through errors (because the file is not yet complete). How can I detect not when the copying started but when the copying ended? I'm using C++ on Windows, so the answer may be platform dependent but, if possible, I'd prefer it to be platform agnostic.
You could use either lock files or a special naming convention. The easiest is the latter and would work something like this:
Say you want to move a file named "fileA.txt" When copying it to the destination directory, instead, copy it to "fileA.txt.partial" or something like that. When the copy is complete, rename the file from "fileA.txt.partial" to "fileA.txt". So the appearance of "fileA.txt" is atomic as far as the watching program can see.
The other option as mentioned earlier is lock files. So when you copy a file named "fileA.txt", you first create a file called "fileA.txt.lock". When the copying is done, you simply delete the lock file. When the watching program see "fileA.txt", it should check if "fileA.txt.lock" exists, if it does, it can wait or revisit that file in the future as needed.
You should not be polling. Use FindFirstChangeNotification (http://msdn.microsoft.com/en-us/library/windows/desktop/aa364417%28v=vs.85%29.aspx) to watch a directory for changes.
Then use the Wait functions (http://msdn.microsoft.com/en-us/library/windows/desktop/ms687069%28v=vs.85%29.aspx) to wait on change notifications to happen.
Overview and examples here: http://msdn.microsoft.com/en-us/library/windows/desktop/aa365261%28v=vs.85%29.aspx
I'm not sure how exactly file write completion can be determined. Evan Teran's answer is a good idea.
You can use something like this, This is tested and working
bool IsFileDownloadComplete(const std::wstring& dir, const std::wstring& fileName)
{
std::wstring originalFileName = dir + fileName;
std::wstring tempFileName = dir + L"temp";
while(true)
{
int ret = rename(convertWstringToString(originalFileName).c_str(), convertWstringToString(tempFileName).c_str());
if(ret == 0)
break;
Sleep(10);
}
/** File is not open. Rename to original. */
int ret = rename(convertWstringToString(tempFileName).c_str(), convertWstringToString(originalFileName).c_str());
if(ret != 0)
throw std::exception("File rename failed");
return true;
}
I have a program that load data from a file using std::ifstream and store the data in a structure. After that, I verify if the data I want was in the file. If it is not, I ask the user to modify the file and press a key. I then reload the file. The problem is that even if the user modified the file, I always get the same data in the file because the file seems to be cache in the application. I've seen that in win32 API, it's possible to use the flag FILE_FLAG_NO_BUFFERING to avoid using a buffered copy when reading a file, but I would like to use that feature with std::ifstream. Is there any way to use the handle created through win32 api with ifstream or anyway to force it directly in std::ifstream ?
Here's a "simplified" code sample:
SomeStructure s = LoadData(fileName);
while(!DataValid(s))
s = LoadData(fileName);
SomeStructure LoadData(const std::string& fileName)
{
std::ifstream fileStream;
while(!OpenFileRead(fileName, fileStream))
{
std::cout<<"File not found, please update it";
fileStream.close();
//Wait for use input
std::string dummy;
std::getline(std::cin, dummy);
}
//... Read file, fill structure, and return
std::string line;
while(std::getline(fileStream, line) && line!="")
{
//At this point, I can see that line is wrong
StringArray namedatearray=Utils::String::Split(line, "|");
assert(namedatearray.size()==2);
//Add data to my structure ( a map)
}
fileStream.close();
//return structure
}
bool OpenFileRead(const std::string& name, std::fstream& file)
{
file.open(name.c_str(), std::ios::in);
return !file.fail();
}
Thanks.
Edit: Of course, it was a mistake because I had two time the same file in two very similar path. Looking at the handle of the file open with process explorer (and not the relative file path made me found it).
Instead of thinking that this is due to some kind of "buffering", I would look for the obvious things first.
Are you sure the user is changing the same file that you're reading?
Are you certain reloading the data is properly updating your data structure in memory?
Are you confident that DataValid() is doing what you want?
The fact that the OS uses file buffers to increase disk performance is generally not visible from the application level. As long as you're looking at the same file, the OS knows that the user updated the file, and if you reopen it, then you'll see the changed data. If the data never even had a chance to get flushed to disk, that won't affect your application.