Under which conditions is renaming a file an atomic operation on Linux? - c++

Question

Assumption
According to the documentation, calling rename on Linux performs an atomic replace:
If newpath already exists, it will be atomically replaced, so that there is no point at which another process attempting to access newpath will find it missing.
Contradiction
However, if I run a simple parallel test, with each thread running the following operations:
create a file foo<thread_id>
rename foo<thread_id> to cache_file (cache_file is the same for every thread)
hard link cache_file to bar<thread_id>
it will eventually fail to create the hard link with the following error:
filesystem error: cannot create hard link: No such file or directory [/app/cache_file] [/app/bar1]. So it seems that the replacement of cache_file is not atomic, as concurrently creating a hard link causes an error. (Note that cache_file is actually stored in a content addressable storage, so the overwrite shouldn't do any harm, as the content of the replaced file and the replacement file is exactly the same.)
Question
Shouldn't the hard link creation always succeed if the replacement operation is atomic, so that the created hard link refers to either the replaced file or the replacement file?
See the minimal working example on godbolt or here:
#include <thread>
#include <vector>
#include <string>
#include <algorithm>
#include <iostream>
#include <fstream>
#include <filesystem>
#include <cstdio>
#include <fcntl.h>
#include <unistd.h>
auto myrename(std::filesystem::path const& from,
std::filesystem::path const& to, int variant) -> bool {
switch (variant) {
case 0: // c++ rename
std::filesystem::rename(from, to);
return true;
case 1: // c rename
return std::rename(from.c_str(), to.c_str()) == 0;
case 2: // linux rename (same as std::rename?)
return rename(from.c_str(), to.c_str()) == 0;
case 3: // linux link and unlink (no overwrite)
return (link(from.c_str(), to.c_str()) == 0 or errno == EEXIST)
and unlink(from.c_str()) == 0;
case 4: // linux renameat2 without overwrite
return renameat2(0, from.c_str(), 0, to.c_str(), RENAME_NOREPLACE) == 0
or (errno == EEXIST and unlink(from.c_str()) == 0);
default:
return false;
}
}
auto mylink(std::filesystem::path const& from, std::filesystem::path const& to,
int variant) -> bool {
if (std::filesystem::exists(to)) std::filesystem::remove(to);
switch (variant) {
case 0: // c++ hard link
std::filesystem::create_hard_link(from, to);
return true;
case 1: // linux link
return link(from.c_str(), to.c_str()) == 0;
default:
return false;
}
}
auto create_store_stage(std::string const& id) noexcept -> bool {
try {
auto cwd = std::filesystem::current_path();
auto cache = cwd / "cache_file"; // common
auto ifile = cwd / ("foo" + id); // thread local
auto ofile = cwd / ("bar" + id); // thread local
return std::ofstream{ifile}.put('x') // 1. create input file
and myrename(ifile, cache, 0) // 2. store in cache
and mylink(cache, ofile, 0); // 3. hard link to output file
} catch (std::exception const& e) {
std::cout << "caught exception: " << e.what() << std::endl;
return false;
}
}
int main(int argc, const char *argv[]) {
bool fail{};
std::vector<std::thread> threads{};
for (int i{}; i < std::thread::hardware_concurrency(); ++i) {
threads.emplace_back([id = std::to_string(i), &fail]{
while (not fail and create_store_stage(id)) {}
if (errno) perror(("thread " + id + " failed with error").c_str());
fail = true;
});
}
std::for_each(threads.begin(), threads.end(), [](auto& t) { t.join(); });
return 0;
}
Additional Notes
tested on Debian 11 (Kernel 5.10.0) and Ubuntu 20.04 (Kernel 5.8.0)
tested with GCC 9.3/10.2 and Clang 10.0.0/11.0.0 (although I don't expect the compiler to be the issue)
myrename() variants 3 and 4 work correctly (both do not overwrite, which is fine for a content addressable storage)
as expected, neither variant 0 nor 1 of mylink() does make any difference (both use link(), according to strace)
interesting: on WSL2 with Ubuntu 20.04 (Kernel 4.4.0) the myrename() variants 0, 1, and 2 work correctly, but 3 and 4 fail with filesystem error: cannot create hard link: Invalid argument [/app/cache_file] [/app/bar3] and Invalid argument, respectively
*Update
as pointed out by the busybee, link() should be atomic as well. The Linux man pages do not mention any atomic properties, while the POSIX specification explicitly does:
The link() function shall atomically create a new link for the existing file and the link count of the file shall be incremented by one.
as mentioned by numzero, this could be an unintended side-effect. But I did some testing and this behavior dates back to at least Kernel version 2.6.32.