mmap() after deleting the file - c++

I was told, that mmap() might be in trouble, if someone deletes the original file. I was wondering if that really happens. So i created some little test-program. I am using linux.
#include <iostream>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>
int main(int, char**)
{
char const * const fileName = "/tmp/demo-file.dat";
size_t size;
{
struct stat st;
stat(fileName, &st);
size = st.st_size;
}
int fd = open(fileName, O_RDWR);
if (fd == -1)
{
std::cout << "open() failed, errno = " << errno << ":" << strerror(errno) << std::endl;
return (-1);
}
else
{
std::cout << "open() done (ok)" << std::endl;
}
for (int i = 20; i > 0; --i)
{
std::cout << "file open()'ed, wait #" << i << " seconds before mmap()" << std::endl;
sleep(1);
}
void *data = mmap((void*)0L, size, PROT_READ, MAP_SHARED, fd, (off_t)0);
if (data == (void*)-1)
{
std::cout << "mmap() failed, errno = " << errno << ":" << strerror(errno) << std::endl;
}
else
{
std::cout << "mmap() done (ok)" << std::endl;
}
for (int i = 20; i > 0; --i)
{
std::cout << "going to close() socket in #" << i << " seconds" << std::endl;
sleep (1);
}
close(fd);
for (int i = 30; i > 0; --i)
{
std::cout << "going to umap() files in #" << i << " seconds (still accessing the data)" << std::endl;
for (unsigned int x = 0; x < size; ++x)
{
char cp = *(char*) (data + x);
(void) cp;
}
sleep(1);
}
munmap(data, size);
for (int i = 30; i > 0; --i)
{
std::cout << "going to terminate #" << i << " seconds" << std::endl;
sleep(1);
}
return 0;
}
Whenever i delete the file - after the open() operation - it doesn't have negative impact to then mmap(). I can still acess the data in the test program.
When i delete the file right after close(), but before mmap(), it works. I can also still see the old file in the /proc/[pid]/fd/ area:
lrwx------ 1 frank frank 64 Mär 22 20:28 3 -> /tmp/demo-file.dat (deleted)
The rest of the program works.
Even when i delete the file after the close() it still succeeds to access the mmap() data.
However in both cases, after the close() i cannot see the
lrwx------ 1 frank frank 64 Mär 22 20:28 3 -> /tmp/demo-file.dat (deleted)
anymore. (btw: where is then noted, that this file "still exists somehow"?)
So is it the opposite, that it is guaranteed, that mmap() will still be able to operate on the data, even if the file was manually deleted (in a shell or by some other process)?

Here's what's happening.
The first thing to check is
$ls -i /tmp/demo-file.dat
65 /tmp/demo-file.dat
Note the inode number of the file is 65.
On starting the program, here's what it has in its lsof output (apart from other entries not relevant to the current discourse)
a.out 29271 zoso 3u REG 0,21 5 65 /tmp/demo-file.dat
This is a result of the open() that's been done. Note that the inode number is the same as the other file. The open() has increased the ref count on the same inode. Also, note that REG indicates a regular file.
Now if the file is deleted (using rm etc.), here's what the lsof looks like
a.out 29271 zoso 3u REG 0,21 5 65 /tmp/demo-file.dat (deleted)
This is expected since the file that was opened has been deleted but the handle that to its inode still is open in the process.
Moving on to the mmap, and here's the lsof output
a.out 29271 zoso DEL REG 0,21 65 /tmp/demo-file.dat
a.out 29271 zoso 3u REG 0,21 5 65 /tmp/demo-file.dat (deleted)
Now there's another new entry but note that this is of type DEL which indicates (lifting from lsof man page):
''DEL'' for a Linux map file that has been deleted;
Since lsof can't stat the original file anymore, it puts this mapping as DEL with no size of course, yet note that the inode number still remains the same i.e 65.
Now after close() has been called on the fd here's what lsof shows
a.out 29271 zoso DEL REG 0,21 65 /tmp/demo-file.dat
Note that the (deleted) entry is gone since that fd to a REG file has been closed and now only the mmap'd memory remains.
After munmap() this entry too is gone (no more references to /tmp/demo-file.dat and finally the program ends.

Related

Why is Linux's sendfile() incompatible with /dev/zero?

why is sendfile incompatible with /dev/zero ? why does this code
#include <iostream>
#include <sys/sendfile.h>
#include <fcntl.h>
#include <string.h>
int main(){
int f1 = fileno(tmpfile());
int devzero = open("/dev/zero", O_RDONLY);
if(f1 < 0 || devzero < 0){
throw std::runtime_error("Failed to open files");
}
ssize_t sent = sendfile(f1, devzero, 0, 1);
std::cout << "sent " << sent << " bytes. errno " << errno << " strerror " << strerror(errno) << std::endl;
return 0;
}
consistently output
sent -1 bytes. errno 22 strerror Invalid argument
?
/dev/zero is a special file. It doesn’t represent an on disk file.
From sendfile documentation:
The in_fd argument must correspond to a file which supports mmap(2)-like operations (i.e., it cannot be a socket).
...
EINVAL Descriptor is not valid or locked, or an mmap(2)-like operation is not available for in_fd, or count is negative.
EINVAL out_fd has the O_APPEND flag set. This is not currently supported by sendfile().
These section of kernel source is reporting the error:
if (unlikely(out->f_flags & O_APPEND))
return -EINVAL;
ret = rw_verify_area(WRITE, out, opos, len);
I performed some testing. Reading from a tmpfile(), a regular file, and /dev/zero works. Writing to a regular file works, but writing to /dev/zero fails one of those checks (O_APPEND permissions or rw_verify_area() ).

Minor Page Faults when writing to mmaped file buffer

I'm noticing minor page faults when writing to a mmapped file buffer where the file is backed by disk.
My understanding of mmap is that for file mappings, the page cache has the file's data, and the page table will be updated to point to the file data in the page cache. This means that on the first write to the mmapped buffer, the page table will have to be updated to point to the page cache, and we may see minor page faults. However, as my benchmark below shows, even after pre-faulting the mmapped buffer, I still see minor page faults when doing random writes.
Note that these minor page faults only show up if I write to a random buffer (buf in the benchmark below) in-between writing to the mmapped buffer. Also note that these minor page faults do not seem to happen when using a tmpfs which is not disk backed.
So my question is why do we see these minor page faults when writing to a disk-backed file?
Here is the benchmark:
#include <iostream>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <fstream>
#include <sys/mman.h>
#include <sys/resource.h>
int main(int argc, char** argv) {
// Command line parsing.
if (argc != 2) {
std::cout << "Usage: ./bench <path to file>" << std::endl;
exit(1);
}
std::string filepath = argv[1];
// Open and truncate the file to be of size `FILE_LEN`.
int fd = open(filepath.c_str(), O_CREAT | O_TRUNC | O_RDWR, 0664);
const size_t FILE_LEN = (1 << 26); // 64MiB
if (fd < 0) {
std::cout << "open failed: " << strerror(errno) << std::endl;
exit(1);
}
if (ftruncate(fd, FILE_LEN) < 0) {
std::cout << "ftruncate failed: " << strerror(errno) << std::endl;
exit(1);
}
// `mmap` the file and pre-fault it.
char* ptr = static_cast<char*>(mmap(nullptr, FILE_LEN, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_POPULATE, fd, 0));
if(ptr == MAP_FAILED) {
std::cout << "mmap failed: " << strerror(errno) << std::endl;
exit(1);
}
memset(ptr, 'A', FILE_LEN);
// Create a temporary buffer to clear the cache.
constexpr size_t BUF_LEN = (1 << 22); // 4MiB
char* buf = new char[BUF_LEN];
memset(buf, 'B', BUF_LEN);
std::cout << "Opened file " << fd << ", pre faulted " << ptr[rand() % FILE_LEN] << " " << buf[rand() % BUF_LEN]<< std::endl;
// Run the actual benchmark
rusage usage0, usage1;
getrusage(RUSAGE_THREAD, &usage0);
unsigned int x = 0;
for (size_t i = 0; i < 1000; ++i) {
char c = i % 26 + 'a';
const size_t WRITE_SIZE = 1024;
size_t start = i*WRITE_SIZE;
if (start + WRITE_SIZE >= FILE_LEN) {
start %= FILE_LEN;
}
memset(ptr + start, c, WRITE_SIZE);
x += ptr[start];
char d = (c * 142) % 26 + 'a';
for (size_t k = 0; k < BUF_LEN; ++k) {
buf[k] = d;
}
x += buf[int(d)];
}
std::cout << "Using the buffers " << ptr[rand() % FILE_LEN] << " " << buf[rand() % BUF_LEN] << " " << x << std::endl;
getrusage(RUSAGE_THREAD, &usage1);
std::cout << "========================" << std::endl;
std::cout << "Minor page faults = " << usage1.ru_minflt - usage0.ru_minflt << std::endl;
std::cout << "========================" << std::endl;
return 0;
}
Running ./bench "/dev/shm/test.txt" where /dev/shm/ uses the tmpfs filesystem, the benchmark always shows 0 minor page faults.
Running ./bench "/home/username/test.txt", the benchmark above shows ~200 minor page faults.
i.e. I see output like this with the above command:
========================
Minor page faults = 231
========================
Note that increasing the number of iterations in the benchmark correlates to an increase in number of minor page faults as well (e.g. changing number of iterations fromm 1000 to 2000 results in ~450 minor page faults).

Named pipe file descriptor

Currently I am making a C/C++ program for the Linux Operating system.
I want to use a named pipe to communicate a PID (process ID) between two programs.
The pipe has been created and is visible in the directory.
The Get PID program says that the file descriptor returns 3, while it should return 0 if it could open the pipe. What am I doing wrong?
Get PID
// Several includes
using namespace std;
int main(int argc, char *argv[]) {
pid_t pid;
int sig = 22;
int succesKill;
int iFIFO;
char sPID[5] = {0,1,2,3,'\0'};
iFIFO = open("IDpipe" , O_RDONLY);
if(iFIFO != 0)
{
cerr << "File descriptor does not return 0, but: " << iFIFO << endl;
return EXIT_FAILURE;
}
read(iFIFO, sPID, strlen(sPID));
cerr << "In sPID now is: " << sPID << endl;
close(iFIFO);
pid = atoi(sPID);
cout << "The PID I will send signals to is: " << pid << "." << endl;
while(1)
{
succesKill = kill(pid, sig);
cout << "Tried to send signal" << endl;
sleep(5);
}
return EXIT_SUCCESS;
}
Send PID
// Several includes
using namespace std;
void catch_function(int signo);
volatile sig_atomic_t iAmountSignals = 0;
int main(void) {
pid_t myPID;
int iFIFO;
char sPID[5] = {'l','e','e','g','\0'};
myPID = getpid();
sprintf(sPID, "%d",myPID);
cout << "My PID is: " << sPID << endl;
iFIFO = open("IDpipe" , O_WRONLY);
if(iFIFO == -1)
{
cerr << "Pipe can't be opened for writing, error: " << errno << endl;
return EXIT_FAILURE;
}
write(iFIFO, sPID, strlen(sPID));
close(iFIFO);
if (signal(22, catch_function) == SIG_ERR) {
cerr << "An error occurred while setting a signal handler." << endl;
return EXIT_FAILURE;
}
cout << "Raising the interactive attention signal." << endl;
if (raise(22) != 0) {
cerr << "Error raising the signal." << endl;
return EXIT_FAILURE;
}
while(1)
{
cout << "iAmountSignals is: " << iAmountSignals << endl;
sleep(1);
}
cout << "Exit." << endl;
return EXIT_SUCCESS;
}
void catch_function(int signo) {
switch(signo) {
case 22:
cout << "Caught a signal 22" << endl;
if(iAmountSignals == 9)
{iAmountSignals = 0;}
else
{++iAmountSignals;}
break;
default:
cerr << "Thats the wrong signal.." << endl;
break;
}
}
Terminal output
Output
open() returns the newly created file descriptor. It cannot return 0 for the simple reason that the new process already has a file descriptor 0. That would be standard input.
The return value of 3 is the expected result from open(), in this case, because that would be the next available file descriptor after standard input, output, and error. If open() couldn't open the file descriptor, it would return -1.
But besides that, your code also has a bunch of other bugs:
sprintf(sPID, "%d",myPID);
// ...
write(iFIFO, sPID, strlen(sPID));
If your process ID happens to be only 3 digits long (which is possible), this will write three bytes to the pipe.
If your process ID happens to be five digits long (which is even more possible), this will write 5 bytes plus the '\0' byte, for a total of six bytes written to the five byte-long sPID buffer, overrunning the array and resulting in undefined behavior.
The actual results are, of course, are undefined, but a typical C++ implementation will end up clobbering the first byte of whatever is the next variable on the stack, which is:
int iFIFO;
which is your file descriptor. So, if your luck runs out and your new process gets a five-digit process id, and this is a little-endian C++ implementation, there is no padding, then the low order byte of iFIFO gets set to 0, and if the code got compiled without any optimizations, the iFIFO file descriptor gets set to 0. Hillarity ensues.
Furthermore, on the other side of the pipe:
char sPID[5] = {0,1,2,3,'\0'};
// ...
read(iFIFO, sPID, strlen(sPID));
Because the first byte of SPID is always set to 0, this will always execute read(iFIFO, sPID, 0), and not read anything.
After that:
pid = atoi(sPID);
atoi() expects a '\0'-terminated string. read() only reads whatever it reads, it will not '\0'-terminate whatever it ends up reading. It is your responsibility to place a '\0' that terminates the read input (and, of course, making sure that the read buffer is big enough), before using atoi().
Your logic appears to be incorrect.
if(iFIFO != 0)
should be
if(iFIFO == -1)
since open returns -1 on error. Otherwise it returns a valid file descriptor.

get error if i use fread, while no error using read

I'm trying to make some experiments on disk I/O using cache and not using it. In order to perform a read directly from the disk, I open the file with the O_DIRECT flag (defining the variable DISK_DIRECT).
Now the two branches of the if beneath, should perform the same operation, with the difference that one is helped by the cache and the other not.
The files to which I try to access are stored on disk and they do not change over time.
Also the two branches access to the same files.
At some point here, when I use fread I get ferror() to be true. While when I use read everything goes fine.
I'm sure they access the same files.
Do you have any idea why this could happen?
EDIT
Ok, i'm posting here an minimal example. the code i use is:
#include <iostream>
#include <fcntl.h>
#include <sys/types.h>
#include <unistd.h>
#include <fstream>
#include <sstream>
using namespace std;
typedef float fftwf_complex [2] ;
void fetch_level(unsigned long long tid, unsigned short level, fftwf_complex* P_read, fftwf_complex* P_fread, int n_coeff_per_level, FILE** files_o_direct, fstream* & files) {
int b_read;
fseek(files_o_direct[level],(long int) (tid * sizeof(fftwf_complex)*n_coeff_per_level), SEEK_SET);
b_read = fread(reinterpret_cast<char*>(P_fread),sizeof(fftwf_complex), n_coeff_per_level,files_o_direct[level]);
if(b_read == 0){
cerr << "nothing read\n";
}
files[level].seekg((streamoff) (tid * sizeof(fftwf_complex)*n_coeff_per_level), files[level].beg);
files[level].read(reinterpret_cast<char*>(P_read),
sizeof(fftwf_complex) * n_coeff_per_level);
}
void open_files (fstream* & files){
for(int i=0; i<1;i++) {
std::ostringstream oss;
oss << "./Test_fread_read/1.txt.bin";
files[i].open(oss.str().c_str(),
std::ios::in | std::ios::out |
std::ios::binary | std::ios::ate);
if (!files[i])
{
cerr << "fstream could not open " << oss.str() << endl;
}
}
}
void open_files_o_direct (FILE** files_o_direct, int* fd){
for(unsigned int i=0;i<1; i++){
std::ostringstream oss;
oss << "./Test_fread_read/1.txt.bin";
fd[i]=open(oss.str().c_str(), O_RDONLY | O_DIRECT);
files_o_direct[i] = fdopen(fd[i], "rb");
if(!files_o_direct[i])
cerr << "Could not open " << oss.str() << endl;
}
}
int close_files(FILE** files_o_direct, int* fd, fstream* & files) {
for(unsigned int i=0; i<1; i++){
//#if defined (DISK_DIRECT)
if(files_o_direct[i])
close(fd[i]);
//#else
if(files[i].is_open())
files[i].close();
//#endif
}
return 0;
}
int main(){
FILE**files_o_direct = new FILE* [256];
fstream* files = new fstream [256];
int * fd = new int [256];
fftwf_complex * P_read = new fftwf_complex [1];
fftwf_complex * P_fread = new fftwf_complex [1];
open_files_o_direct(files_o_direct, fd);
open_files(files);
fetch_level(2, 0, P_read, P_fread, 1, files_o_direct, files);
cout << "P_read: " << P_read[0][0] << " P_fread: " << P_fread[0][0] << endl;
cout << "P_read: " << P_read[0][1] << " P_fread: " << P_fread[0][1] << endl;
fetch_level(7, 0, P_read, P_fread, 1, files_o_direct, files);
cout << "P_read: " << P_read[0][0] << " P_fread: " << P_fread[0][0] << endl;
cout << "P_read: " << P_read[0][1] << " P_fread: " << P_fread[0][1] << endl;
fetch_level(8, 0, P_read, P_fread, 1, files_o_direct, files);
cout << "P_read: " << P_read[0][0] << " P_fread: " << P_fread[0][0] << endl;
cout << "P_read: " << P_read[0][1] << " P_fread: " << P_fread[0][1] << endl;
close_files(files_o_direct, fd, files);
delete [] P_read;
delete [] P_fread;
delete [] files;
delete [] files_o_direct;
return 0;
}
and the file which is accessed is:
0.133919 0.0458176
1.67441 2.40805
0.997525 -0.279977
-2.39672 -3.076
-0.0390913 0.854464
-0.0176478 -1.3142
-0.667981 -0.486272
0.831051 0.282802
-0.638032 -0.630943
-0.669854 -1.49762
which is stored in a binary format and that can be download from here: 1.txt.bin.
The output i get is:
nothing read
P_read: 0.997525 P_fread: 0
P_read: -0.279977 P_fread: 0
nothing read
P_read: 0.831051 P_fread: 0
P_read: 0.282802 P_fread: 0
nothing read
P_read: -0.638032 P_fread: 0
P_read: -0.630943 P_fread: 0
The problem persists even if i change the type of fftwf_complex from float[2] to simple float.
If i remove the fseek line everything works correctly.
This if (b_read == 0), will be true at the end of the file, and you will enter this branch
if(ferror(this->files_o_direct[level]))
fseek(this->files_o_direct[level], 0, SEEK_END); //ftell here returns 4800000
cerr << "nothing read\n";
even if ferror returns 0, the end of the file was reached anyway
fseek(this->files_o_direct[level], 0, SEEK_END);
makes no sense, and "nothing read\n" will be output either or not ferror returns nonzero.
From the manual page
fread() does not distinguish between end-of-file and error, and callers must use feof(3) and ferror(3) to determine which occurred.
so you have to check feof and if it is false you use ferror.
For who ever may have the same problem here there is the answer:
The O_DIRECT flag may impose alignment restrictions on the length and
address of user-space buffers and the file offset of I/Os. In Linux
alignment restrictions vary by filesystem and kernel version and
might be absent entirely. However there is currently no
filesystem-independent interface for an application to discover these
restrictions for a given file or filesystem. Some filesystems
provide their own interfaces for doing so, for example the
XFS_IOC_DIOINFO operation in xfsctl(3).
Under Linux 2.4, transfer sizes, and the alignment of the user buffer
and the file offset must all be multiples of the logical block size
of the filesystem. Since Linux 2.6.0, alignment to the logical block
size of the underlying storage (typically 512 bytes) suffices. The
logical block size can be determined using the ioctl(2) BLKSSZGET
operation or from the shell using the command:
blockdev --getss
linux reference page

std::ofstream : Writing to a file using append and out flags

Below is a simple class which attempts to write an integer to a file. The mode of writing the file is to append characters at the end of the file (In this mode, file should be created if it doesn't exist)
#include <iostream>
#include <fstream>
class TestFileStream
{
private:
std::ofstream* _myFileStream;
bool isFileOpen;
public:
TestFileStream():isFileOpen(false)
{
_myFileStream = new std::ofstream("TestFile.txt", std::ios_base::out | std::ios_base::app );
isFileOpen = _myFileStream->is_open();
if( !isFileOpen )
{
std::cout << "Unable to open log file" << std::endl;
std::cout << "Good State: " << _myFileStream->good() <<std::endl;
std::cout << "Eof State: " << _myFileStream->eof() <<std::endl;
std::cout << "Fail State: " << _myFileStream->fail() <<std::endl;
std::cout << "Bad State: " << _myFileStream->bad() <<std::endl;
}
else
{
std::cout << "Opened log file" << std::endl;
}
}
~TestFileStream()
{
_myFileStream->close();
delete _myFileStream;
_myFileStream = nullptr;
}
void WriteFile( unsigned number )
{
if ( isFileOpen )
{
(*_myFileStream) << "Number: " << number << std::endl;
}
}
};
int main()
{
// Number of iterations can be multiple.
// For testing purpose, only 1 loop iteration executes
for( unsigned iter = 1; iter != 2; ++iter )
{
TestFileStream fileWriteObj;
fileWriteObj.WriteFile( 100+iter );
}
return 0;
}
When I execute the above code, I get following log output:
Unable to open log file
Good State: 0
Eof State: 0
Fail State: 1
Bad State: 0
This seems like trivial task, but I am not able to find out whats causing the failure. Note that this question is most likely related to the following question
Just to summarize the comments, there is nothing wrong about the code you posted (apart from the rather unconventional new ostream ;) )
Note however that opening files may fail for a number of reasons (Permissions, file in use, disk unavailable, the file does not exist, the file exists...). That is why you must always test it.
If you tried to run the above code in an online emulator, then chances are file IO is disabled. Which would explain why you get that the streams fail-bit is set.