I did two months search on the web for a proper file locking mechanism to be used in a C++ program.
I found a lot on "C and fnctl" which I could proof to work. But all really proper working locking mechanism, that I could proof to work in Linux are only based on file descriptors.
As this seems to be something really old fashined and in actual C++17 style of writing C++ code with file- and ip-streams not using that mechanism, I only came up with something that works with using what was presented here:
Not able to ofstream using __gnu_cxx::stdio_filebuf
My Question is, is this really the only mechanism working? To connect both worlds?
I looked in all these books to find anything about fcntl and C++, but was not successful:
[Der C++ Programmierer Cxx20]
(https://www.hanser-elibrary.com/doi/book/10.3139/9783446465510)
[The C++ Programming Language] (https://www.stroustrup.com/C++.html)
[C++ Das Umfassende Handbuch]
(https://www.rheinwerk-verlag.de/c-plusplus-das-umfassende-handbuch/)
[Modern C++ Programming Cookbook Second Edition]
(https://www.packtpub.com/product/modern-c-programming-cookbook-second-edition/9781800208988)
My question to the C++ gurus here is, if I missed something, or if the following code is, today, begin of 2021 the best we could do.
Short explanation of what the code is a proof for:
We have a C++ Code which adds usernames and its LSF-processes to a conf-file, which is read by SSH-server to allow user access to that machine. As at the same time two or more running processes of this code could lead to concurrent attempts of adding or deleting users from this file could occur, we have to proof that proper file locking is preventing that. Without using an extra "access" file, which also could be a solution.
This is some example code I tested:
#include <iostream>
#include <string>
#include <thread>
#include <chrono>
#include <fcntl.h>
#include <unistd.h>
#include <ext/stdio_filebuf.h>
using namespace std::this_thread; // for sleep_for
int main( ) {
// set unbuffered concole output
std::cout.setf(std::ios::unitbuf);
const char* filename {"testfile.txt"};
// get input from input_from_user
std::string input_from_user_string;
std::cout << "Please give input to change in the file: ";
std::cin >> input_from_user_string;
int add_1_del_2 = 0;
std::cout << "Please give 1 if you want to add to the file or 2 if you want to delete from file: ";
std::cin >> add_1_del_2;
int input_from_user_time;
std::cout << "Please give seconds to wait: ";
std::cin >> input_from_user_time;
// opening file
std::cout << "Opening File" << std::endl;
mode_t mode = S_IRUSR|S_IWUSR|S_IRGRP|S_IWGRP|S_IROTH; //664
int fd;
fd = open(filename, O_RDWR | O_CREAT, mode);
// printing out information about file descriptor
std::cout << " Dexc:" << fd << std::endl;
// generating C++-streams on filedescriptor
__gnu_cxx::stdio_filebuf<char> sourcebufin(fd, std::ios::in);
__gnu_cxx::stdio_filebuf<char> sourcebufout(fd, std::ios::out);
std::istream myfilein(&sourcebufin);
std::ostream myfileout(&sourcebufout);
// -----------
// check for file Locking or exit
// -----------
// creating structure for file locking
struct flock fl;
fl.l_type = F_RDLCK;
fl.l_whence = SEEK_SET;
fl.l_start = 0;
fl.l_len = 0;
// set file locking for read
fl.l_type = F_RDLCK;
std::cout << "Checking for Lock on file" << std::endl;
// check for file locking on file for read only once
(void) fcntl(fd, F_GETLK, &fl);
if (fl.l_type != F_UNLCK) {
std::cout << "File is locked for reading by process "
<< fl.l_pid
<< ", in status"
<< ((fl.l_type == F_WRLCK) ? 'W' : 'R')
<< ", start="
<< fl.l_start
<< ", end="
<< fl.l_len
<< std::endl;
}
else {
(void) printf("File is unlocked for reading\n");
}
// set file locking for write
fl.l_type = F_WRLCK;
// check for file locking on file for write in a loop
for (int i = 1; i < 11; i++) {
//printf("Checking for lock %d of 10 times...\n", i);
std::cout << "Checking for lock "
<< i
<< " of 10 times..."
<< std::endl;
(void) fcntl(fd, F_GETLK, &fl);
if (fl.l_type != F_UNLCK) {
//(void) printf("File is locked by process %d, in status %c, start=%8ld, end=%8ld\n", fl.l_pid,
// , fl.l_start, fl.l_len);
std::cout << "File is locked by process "
<< fl.l_pid
<< ", in status"
<< ((fl.l_type == F_WRLCK) ? 'W' : 'R')
<< ", start="
<< fl.l_start
<< ", end="
<< fl.l_len
<< std::endl;
sleep(10);
}
else {
(void) printf("File is unlocked\n");
break;
}
}
// -----------
// apply lock for write on file
// -----------
// locking file
std::cout << "Locking file for write" << std::endl;
// set file locking for write again, as checking on lock resets it
fl.l_type = F_WRLCK;
if (fcntl(fd, F_SETLKW, &fl) == -1) {
perror("fcntl");
abort();
}
// -----------
// wait some time
// -----------
std::cout << "Now waiting for " << input_from_user_time << " seconds, keeping the file locked..." << std::endl;
std::this_thread::sleep_for(std::chrono::seconds(input_from_user_time));
// -----------
// read from file
// -----------
std::cout << "Reading from file... " << std::endl;
myfilein.seekg(0, std::ios::end);
size_t size_before = myfilein.tellg();
myfilein.seekg(0);
std::string filecontent{""};
filecontent.reserve(size_before);
std::cout << "Length of file is: " << size_before << std::endl;
// read full content of file in string "filecontent"
filecontent.assign((std::istreambuf_iterator<char>(myfilein)),
std::istreambuf_iterator<char>());
// -----------
// print output about read data
// -----------
std::cout << "Length of filecontent-string: " << filecontent.size() << std::endl;
std::cout << "Content of File begin" << std::endl;
std::cout << "----------" << std::endl;
std::cout << filecontent << std::endl;
std::cout << "----------" << std::endl;
// -----------
// Apply changes on read in data depending on given input
// -----------
if (add_1_del_2 == 2) {
std::cout << "Runmode: Del" << std::endl;
std::string string_to_delete = input_from_user_string+"\n";
std::string::size_type pos_of_found_substring = filecontent.find(string_to_delete);
if (pos_of_found_substring != std::string::npos) {
filecontent.erase(pos_of_found_substring, string_to_delete.length());
}
else {
}
}
if (add_1_del_2 == 1) {
std::cout << "Runmode: Append" << std::endl;
filecontent.append(input_from_user_string);
}
std::cout << "Content of String after change" << std::endl;
std::cout << "----------" << std::endl;
std::cout << filecontent << std::endl;
std::cout << "----------" << std::endl;
// -----------
// write out to file, truncate before to length of new string
// -----------
std::cout << "Now starting the write out..." << std::endl;
myfilein.seekg(0);
ftruncate(fd,filecontent.length());
myfileout.seekp(0);
myfileout << filecontent;
myfileout.flush();
myfileout.clear();
// -----------
// read from file for a second time and printout content
// -----------
std::cout << "Reading from file again... " << std::endl;
myfilein.seekg(0, std::ios::end);
size_t size_after = myfilein.tellg();
myfilein.seekg(0);
std::string filecontent_after{""};
filecontent_after.reserve(size_after);
std::cout << "Length of file is now: " << size_after << std::endl;
// read full content of file in string "filecontent"
filecontent_after.assign((std::istreambuf_iterator<char>(myfilein)),
std::istreambuf_iterator<char>());
std::cout << "Length of filecontent_after-string: " << filecontent_after.size() << std::endl;
std::cout << "Content of File end" << std::endl;
std::cout << "----------" << std::endl;
std::cout << filecontent_after << std::endl;
std::cout << "----------" << std::endl;
// -----------
// unlocking file and close file
// -----------
printf("Unlocking...\n");
fl.l_type = F_UNLCK;
if (fcntl(fd, F_SETLK, &fl) == -1) {
perror("fcntl");
abort();
}
close(fd);
// -----------
// done
// -----------
std::cout << "done" << std::endl;
exit(0);
}
I ask for your comments on this or perhaps how to improve.
Alexander Bruns
Related
Okay so a little background this code is supposed to read through a file containing DNA and calculate the number of nucleotides A, C, T, G and print them out and also do some other slight calculations. My code runs fine for most files except for files that contain lines that start with # and + in the file. I need to skip those lines in order to get an accurate number. So my question is how to skip or ignore these lines in my calculations.
My code is
#include <iostream>
#include <stream>
#include <string>
#include <vector>
#include <map>
int main(int argc, char** argv) {
// Ignore how the above argc and argv are used here
auto arguments = std::vector<std::string>(argv, argv + argc);
// "arguments" box has what you wrote on the right side after &&
if (arguments.size() != 2) {
// ensure you wrote a file name after "./a.out"
std::cout << "Please give a file name as argument\n";
return 1;
}
auto file = std::fstream(arguments[1]);
if (!file) {
// ensure the file name you gave is from the available files
std::cout << "Cannot open " << arguments[1] << "\n";
return 1;
}
auto counts = std::map<char,int>({{'G',0.0},{'A',0.0},{'C',0.0},{'T',0.0}});
// Just a test loop to print all lines from the file
for (auto dna = std::string(); std::getline(file, dna); ) {
//std::cout << dna << "\n";
for (auto nucleotide:dna) {
counts[nucleotide]=counts[nucleotide] + 1;
}
}
double total = counts['A'] + counts['T'] + counts['G'] + counts['C'];
double GC = (counts['G'] + counts['C'])*100/total;
double AT = (counts['A'] + counts['T'])*100/total;
double ratio = AT/GC;
auto classification = "";
if ( 40.0 < GC < 60.0) {
classification = "moderate GC content";
}
if (60 <= GC) {
classification = "high GC content";
}
if (GC <= 40.0) {
classification = "low GC content";
}
std::cout << "GC-content: " << GC << "\n";
std::cout << "AT-content: " << AT << "\n";
std::cout << "G count: " << counts['G'] << "\n";
std::cout << "C count: " << counts['C'] << "\n";
std::cout << "A count: " << counts['A'] << "\n";
std::cout << "T count: " << counts['T'] << "\n";
std::cout << "Total count: " << total << "\n";
std::cout << "AT/GC Ratio: " << ratio << "\n";
std::cout << "GC Classification: " << classification << "\n";
}
The file that is giving me trouble is this which is like this
#ERR034677.1 HWI-EAS349_0046:7:1:2144:972#0 length=76
NGATGATAAACAAGAGGGTAAAAAGAAAAAAGCTACAGACATTTCTGCTAATCTATTATTTTGTTCCTTTTTTTTT
+ERR034677.1 HWI-EAS349_0046:7:1:2144:972#0 length=76
BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
If anyone can help me with this. I will be very grateful. I only need a hint or an idea of the concept I am missing so I can make my code compatible with all files. Thanks in advance
Your actual problem seems to be the standard case of "input is not always clean syntax".
The solution is always "do not expect clean syntax".
First read whole lines into a buffer.
Then check for syntax.
Skip broken syntax.
Scan clean syntax from buffer.
I am new to C++ programming and I am trying to use mkfifo command to create a named pipe that I can read from my java program. Here is my code:
int main(int, char **)
{
std::cout << "START" << std::endl;
int fd;
// FIFO file path
char *myfifo = "/sdcard/tempFifo";
// Creating the named file(FIFO)
// mkfifo(<pathname>, <permission>)
int ret = mkfifo(myfifo, 0666);
std::cout << "mkfifo ret " << ret << std::endl;
std::cout << "errno " << errno << std::endl;
std::cout << "errno str::" << std::strerror(errno) << std::endl;
char arr1[80];
int startCount = 0;
while (1)
{
// Open FIFO for Read only
std::cout << "Opening FIFO.." << std::endl;
fd = open(myfifo, O_RDONLY);
// Read from FIFO
std::cout << "Start reading.." << std::endl;
read(fd, arr1, sizeof(arr1));
if (strcmp(arr1, "start") == 0)
{
if (startCount == 1)
{
std::cout << "Start count is greater than one so exit." << std::endl;
return 0;
}
startCount++;
std::cout << "Start received" << std::endl;
}
sleep(5000);
close(fd);
}
std::cout << "STOP" << std::endl;
return 0;
}
To write "start" on the target pipe, I am using a java code as:
File file = new File("/sdcard/tempFifo");
FileOutputStream fileOutputStream = new FileOutputStream(file);
PrintWriter printWriter = new PrintWriter(fileOutputStream);
printWriter.print("start");
printWriter.flush();
printWriter.close();
I am getting Exception: /sdcard/tempFifo: open failed: ENOENT (No such file or directory) in java applicatoin and when I have executed ls -l /sdcard/tempFifo on adb shell then I am not able to see any tempFifo file on my sdcard of rooted phone.
Does anyone know what is the problem in my code?
Update on errno
mkfifo ret -1
errno 22
errno str::Invalid argument
Android doesn't allow creating pipes on /sdcard so, I have used /system/tempFifo instead of /sdcard/tempFifo to make it work.
When using a FIFO on a single process, it looks like after both ends have been opened and one is then closed, it is not possible to reuse the FIFO. Any attempt to reopen the closed end fails or the returned file descriptor is useless.
Is it possible to work around this behavior, or do we have to keep both ends of the FIFO open until we are absolutely sure we don't need it anymore?
Here is some test code that shows and attempt to reopen a closed write end of a FIFO:
#include <iostream>
#include <cstdio>
#include <errno.h>
#include <fcntl.h>
#include <limits.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>
using namespace std;
int main(int argc, const char **argv)
{
cout << "Creating an instance of a named pipe..." << endl;
mode_t prevMask = umask(0);
if (mknod("my_pipe", S_IFIFO | 0666, 0))
return -1;
umask(prevMask);
cout << "Opening Read end..." << endl;
int fdM = open("my_pipe", O_RDONLY | O_NONBLOCK);
if (fdM == -1)
return -1;
cout << "Opening Write end..." << endl;
int fdS = open("my_pipe", O_WRONLY | O_NONBLOCK);
if (fdS == -1)
return -1;
cout << "Sending data to pipe..." << endl;
const char *data = "Hello my friend!";
ssize_t NbOfBytesWritten = write(fdS, data, strlen(data));
if (NbOfBytesWritten < 0)
return -1;
cout << "Number of bytes sent: " << NbOfBytesWritten << endl;
cout << "Closing Write end..." << endl;
if (close(fdS))
return -1;
cout << "Reopening Write end..." << endl;
fdS = open("my_pipe", O_WRONLY | O_NONBLOCK);
if (fdS == -1)
{
cout << "open() - failed("<< errno << "): " << strerror(errno) << '.';
remove("my_pipe");
return -1;
}
cout << "Sending some more data to pipe..." << endl;
data = "What's up?";
NbOfBytesWritten = write(fdS, data, strlen(data));
if (NbOfBytesWritten < 0)
return -1;
cout << "Number of bytes sent: " << NbOfBytesWritten << endl;
cout << "Reading data from pipe..." << endl;
char buff[128];
ssize_t numBytesRead = read(fdM, buff, 127);
if (NbOfBytesWritten < 0)
return -1;
buff[numBytesRead] = '\0'; // null terminate the string
cout << "Number of bytes read: " << numBytesRead << endl;
cout << "Message: " << buff << endl;
cout << "Closing Write end..." << endl;
if (close(fdS))
return -1;
cout << "Closing Read end..." << endl;
if (close(fdM))
return -1;
cout << "Deleting pipe..." << endl;
if (remove("my_pipe"))
return -1;
return 0;
}
Here is the output:
Creating an instance of a named pipe...
Opening Read end...
Opening Write end...
Sending data to pipe...
Number of bytes sent: 16
Closing Write end...
Reopening Write end...
open() - failed(6): No such device or address.
I also tested similar code trying to reopen a closed read end (While the write end was kept open). In that case the open() function succeed, but the read() function using the file descriptor returned by open() fails with:
Communication error on send. (70)
EDIT:
I'm using CYGWIN.
You code works fine on Linux. I think the issue you are running into is that named pipes on CYGWIN don't work very well and fail to follow POSIX semantics. See FIFO (named pipe) is broken on Cygwin. Likely the same problem you have.
Aloha,
I'm struggling with OpenCL child kernel feature.
Kernel SRC (Minimal example):
kernel void launcher()
{
ndrange_t ndrange = ndrange_1D(1);
enqueue_kernel(get_default_queue(), CLK_ENQUEUE_FLAGS_WAIT_KERNEL, ndrange,
^{
size_t id = get_global_id(0);
}
);
}
stdafx.h:
#pragma once
#define __CL_ENABLE_EXCEPTIONS
#define CL_HPP_ENABLE_EXCEPTIONS
#define CL_HPP_TARGET_OPENCL_VERSION 200
#include "targetver.h"
#include <CL/cl2.hpp>
#include <iostream>
#include <string>
Full SRC (Minimal):
#include "stdafx.h"
std::string kernel2_source(
"kernel void launcher() ""\n"
"{ ""\n"
" ndrange_t ndrange = ndrange_1D(1);""\n"
" enqueue_kernel(get_default_queue(), CLK_ENQUEUE_FLAGS_WAIT_KERNEL, ndrange,""\n"
" ^{""\n"
" size_t id = get_global_id(0);""\n"
" }""\n"
" );""\n"
"}""\n");
//Number of Input Elements
constexpr int numTriangles = 10;
cl_int errorcode = CL_BUILD_ERROR; //Has to be set to build error, because errorcode isn't set when exception occurs
//Move variable definitions out of main for test purposes;
//Numerous definitions
cl::Program program;
std::vector<cl::Device> devices;
std::vector<cl::Platform> platforms;
cl::CommandQueue queue;
cl::Program::Sources source{ kernel2_source };
int main() {
try {
// Query for platforms
cl::Platform::get(&platforms);
std::cout << "Num Platforms: " << platforms.size() << std::endl;
// Get a list of devices on this platform
platforms[0].getDevices(CL_DEVICE_TYPE_ALL, &devices);
std::cout << "Using platform: " << platforms[0].getInfo<CL_PLATFORM_NAME>() << std::endl;
std::cout << "Num Devices: " << devices.size() << std::endl;
// Create a context for the devices
std::cout << "Using device: " << devices[0].getInfo<CL_DEVICE_NAME>() << std::endl;
//Create a context for the first device
//cl::Context context({ devices[0]});
cl::Context context({ devices[0] });
// Create a command−queue for the first device
queue = cl::CommandQueue(context, devices[0]);
cl::DeviceCommandQueue deviceQueue;
deviceQueue = cl::DeviceCommandQueue(context, devices[0]);
// Create the program from the source code
program = cl::Program(context, source);
std::cout << "Building Program" << std::endl;
// Build the program for the devices
errorcode = program.build("-cl-std=CL2.0 -g");
std::cout << "Success!" << std::endl;
cl::Kernel kernel = cl::Kernel(program, "launcher");
cl::NDRange global = numTriangles;
cl::NDRange local = 1;
queue.enqueueNDRangeKernel(kernel, cl::NullRange, global, local);
std::cout << "finished" << std::endl;
std::cin.get();
}
catch (cl::Error error)
{
std::cout << "Error!" << std::endl;
std::cout << error.what() << "(" << error.err() << ")" << std::endl;
std::cout << "Errorcode: " << errorcode << std::endl;
if (errorcode != CL_SUCCESS) { //...
std::cout << "Build Status: " << program.getBuildInfo<CL_PROGRAM_BUILD_STATUS>(devices[0]) << std::endl;
//std::cout << "Build Status: " << program.getBuildInfo<CL_PROGRAM_BUILD_STATUS>(devices[1]) << std::endl;
std::cout << "Build Options:" << program.getBuildInfo<CL_PROGRAM_BUILD_OPTIONS>(devices[0]) << std::endl;
//std::cout << "Build Options:" << program.getBuildInfo<CL_PROGRAM_BUILD_OPTIONS>(devices[1]) << std::endl;
std::cout << "Build Log:" << program.getBuildInfo<CL_PROGRAM_BUILD_LOG>(devices[0]) << std::endl;
//std::cout << "Build Log:" << program.getBuildInfo<CL_PROGRAM_BUILD_LOG>(devices[1]) << std::endl;
}
}
std::cin.get();
return 0;
}
Output:
Num Platforms: 1
Using platform: AMD Accelerated Parallel Processing
Num Devices: 2
Using device: Hawaii
Building Program
=> Exception.
There appears an uncaught exception which is strange, because all build error should be caught.
The ndrange_1D(1) is just for testing purposes (and to produce an acceptable amount of dummy output).
The device (AMD R9 390X) is OpenCL 2.0 capable.
Any ideas how to fix this?
EDIT:
Even not using exceptions and using errorcodes throws this an exception!
so i have this code for checking crc file named map.spak and compare the result with my specified crc result which stored in variable "compare"
int main(int iArg, char *sArg[])
{
char sSourceFile[MAX_PATH];
memset(sSourceFile, 0, sizeof(sSourceFile));
CCRC32 crc32;
crc32.Initialize(); //Only have to do this once.
unsigned int iCRC = 0;
strcpy(sSourceFile, "map.spak");
int compare = 399857339;
ifstream checkfile(sSourceFile);
if (checkfile){
cout << "Checking file " << sSourceFile << "..." << endl;
crc32.FileCRC(sSourceFile, &iCRC);
if(iCRC == compare){
cout << "File " << sSourceFile << " complete!\nCRC Result: " << iCRC << endl;
}else{
cout << "File " << sSourceFile << " incomplete!\nCRC Result: " << iCRC << endl;
}
}else{
cout << "File not found!" << endl;
}
system("pause");
return 0;
}
and now i want to make this code for multiple file
let's say the file name list stored in filelist.txt
the filelist.txt structure:
id|filename|specified crc
1|map.spak|399857339
2|monster.spak|274394072
how to make the crc check, loop for each file name
i'm not really good at c++ i only know some algorithm because i know PHP
c++ is too complicated
this is the full source included CRC source Source Code
or pastebin
TestApp.cpp link
I made several changes to your code. I removed guard headers since we use it only in header files. Old-fasioned memset has been replaced by operation on strings. I suspect that you need to pass char* to CCRC32 object hence sSourceFile is still const char*. I compiled code except parts with CCRC32.
#include <iostream>
#include <fstream>
#include <string>
#include <vector>
#include "../CCRC32.H"
int main(int iArg, char *sArg[])
{
std::vector<std::string> filenames;
// TODO - populate filesnames (paths?)
CCRC32 crc32;
crc32.Initialize(); //Only have to do this once.
for (unsigned int i = 0; i < filenames.size(); i++) {
const char* sSourceFile = filenames[i].c_str();
unsigned int iCRC = 0;
int compare = 399857339; // TODO - you need to change this since you are checking several files
std::ifstream checkfile(sSourceFile);
if (checkfile) {
std::cout << "Checking file " << sSourceFile << "..." << std::endl;
crc32.FileCRC(sSourceFile, &iCRC);
if(iCRC == compare){
std::cout << "File " << sSourceFile << " complete!\nCRC Result: " << iCRC << std::endl;
} else {
std::cout << "File " << sSourceFile << " incomplete!\nCRC Result: " << iCRC << std::endl;
}
} else {
std::cout << "File tidak ditemukan!" << std::endl;
}
}
return 0;
}