recursive_directory_iterator's skip_permission_denied option appears to be ignored on macOS? - c++

Question

Using C++20 and std::filesystem::recursive_directory_iterator on macOS, this code:
for (auto& f : recursive_directory_iterator(getenv("HOME"), directory_options::skip_permission_denied)) {
// dummy
}
Which should, according to my understanding of the documentation, skip directories which it does not have permission to recurse into, encounters an error upon trying to recurse into ~/Library/Application Support/MobileSync/.
However:
in recursive_directory_iterator::operator++(): attempting recursion into "/Users/t/Library/Application Support/MobileSync": Operation not permitted
I assume this means that there is some permission / security feature in place that the iterator will not skip over even if skip_permission_denied is present - what might this be, and how would I cleanly make the iterator skip over directories that cause it to break regardless of permissions?
I could manually disable_recursion_pending() when encountering known directories like MobileSync or .Trash that cause this problem, but that would be a messy solution compared to being able to detect in advance when a directory will cause this issue.

Answer 1

I'm afraid there is no easy way around it, as the iterator is "closed" on error so a post-error disable_recursion_pending will not help. I opened an issue for libcxx (https://github.com/llvm/llvm-project/issues/48870) and libstdc++ (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99533) that got worked on, but the libcxx one was not fixed yet, and even then it would have to make it into macOS as on macOS the standard library is part of the system.
One admittedly ugly but possible non-hardcoded workaround would be to blacklist dynamically and retry e.g. somewhere along the lines of:
// ugly workaround to iterate the home dir on macOS with std::filesystem:
std::set<fs::path> blacklist;
while(true) {
fs::path lastPath;
try {
for(auto rdi = fs::recursive_directory_iterator(getenv("HOME"), fs::directory_options::skip_permission_denied);
rdi != fs::recursive_directory_iterator();
++rdi) {
auto& de = *rdi;
lastPath = de.path();
if(blacklist.count(de.path())) {
rdi.disable_recursion_pending();
}
else {
// collect info you need here
// ...
}
++rdi;
}
}
catch (fs::filesystem_error& fe) {
if(!blacklist.insert(lastPath).second) {
// exception on same path, double error, something went really wrong
break;
}
else {
// we blacklisted a new entry, reset your collected info here,
// we need to restart
// ...
continue;
}
}
// we are through and managed to get all info
// ...
break;
}
Of course this is a lot of code working around something that should be a single line and only needed on macOS, so if one uses this at all, it should be wrapped away.
One subtle thing to be aware of is that a range-based-for uses the fs::begin(fs::recursive_directory_iterator) function that creates an "invisible" copy, and makes it impossible to call disable_recursion_pending() on the correct instance. This is the reason why a regular for-loop is used.
Another ugly part of that workaround is that besides the standards suggestions neither path1() nor path2() of the exception deliver the offending path and as parsing the exception text is a bad idea, the paths are remembered in the otherwise useless lastPath variable.
All in all it works, but this is nothing I would actually use, as the tree needs to be scanned multiple times (on my notebooks "Application Support" it takes six rescans until it gets through and four times the runtime of an implementation that works without this hack), so I would see this more as an experiment if it is possible to generically iterate the home on macOS with std::filesystem.
So, sadly, until those issues are fixed, std::filesystem::recursive_directory_iterator is not that great on macOS, and I continue to use my drop-in filesystem replacement on that platform, that honors skip_permission_denied for EPERM and EACCES (but is utf-8 only).