I'm writing to the /dev interface of a hardware device on linux. The /dev interface is presented as a linux file, to talk to the device you simply read and write the file. I am using std c++ file wrappers std::fwrite and std::fread because i need access to the file underlying file descriptor for ioctl calls, which is not exposed with the prefered std::ofstream but i digress.
The issue is simple, a write followed by a read fails when using the std:: * calls. It appears to be an issue with fseek but I am unsure. With the fseek code as shown below, successive writes return as if they are a success but no data is written, without fseek code the std::fread call returns an error value. Curiously the linux file functions (write and read) work perfectly, without any fseek mess or anything at all. My question is WHY!?
Linux function version (works perfectly):
bool Write(const std::vector<T> &data)
{
if(write(GetFileDescriptor(),&data[0],sizeof(T) * data.size()) ==
sizeof(T) * data.size())
return true;
return false;
}
std::vector<T> Read(int CountOfT)
{
std::vector<T> buf(CountOfT);
if(read(GetFileDescriptor(), &buf[0], sizeof(T) * CountOfT) !=
sizeof(T) * CountOfT)
throw "stuff"; //i actually use std::optional
return buf;
}
STD Version (fails)
bool Write(const std::vector<T> &data)
{
if(std::fwrite(data.data(), sizeof(T), data.size(), m_fd.get()) <
data.size())
return false;
return true;
}
std::vector<T> Read(int CountOfT)
{
long fileoffset = std::ftell(m_fd.get()); //get current offset
std::fseek(m_fd.get(),0,SEEK_SET); //place offset at file start
std::vector<T> buf(CountOfT);
if(std::fread(&buf[0],sizeof(T),buf.size(),m_fd.get()) < CountOfT)
throw "stuff";
std::fseek(m_fd.get(),fileoffset,SEEK_SET); //reset to where it was
return buf;
}
Related
I have one question about socket programming in C++. Most of the tutorials I found on web assume that
(binding etc. is omitted)
there is a string at client process
it is saved to a file
then the file is sent to server by first reading the file into stream
server receives the stream and write it into another file.
Then, my question is that what if we can use stringstrem at step 2 instead of saving as a file? File I/O (in C++ ifstream and ofstream are typically used) is generally slow. Is it more efficient if I use stringstream directory?
Your Original Question:
"What if we can use stringstrem at step 2 instead of saving as a file?"
My Initial Response:
stringstream has nothing to do with server sockets and IO files.
You are lacking the fundamental idea of IO operations which is the concept of files for IO devices. There is no way around. You save nothing in a logical file stream. Your file bytes are buffered temporarily on your memory and flushed.
stringstream is a nice C++ library utility that let's you treat strings as file streams. Just like you read from an input file stream bytes after bytes until EOF/some other errors, or write into an output file stream bytes after bytes, using stringstream you can treat your string like the way you do to file streams. This is really helpful when you want to divide your string in small logical units. For example, suppose
you read a string line and want to read each word from that line by treating the string line as a stream of words.
Further Instructions To Guide You To The Right Direction:
Nothing is "saved" in a logical file stream. Every I/O operation is performed through "logical" files in any computer system. Socket connection has two file descriptors on both ends: one is a client file descriptor and another one is a server file descriptor (connected file descriptor). Server listens for connection requests through a listening file descriptor which actually stays around as long as the lifetime of the server, and when it accepts a connection request, it returns another file descriptor through accept function called connected file descriptor that stays around as long as the client-server connection/transaction is ongoing.
int accept(int listenfd, struct sockaddr *addr, int *addrlen);
If you want to read from or write into a file stream and also wish to buffer your file bytes, you exactly need to do that- buffer your bytes. This is also very important in the context of servers and short counts because your connection might time out or it might get interrupted by signals. There are several options and techniques that you might implement. However, such discussions are not possible in this small thread. What I'm going to do based on your question is give you an example of how you can buffer your file stream, avoid short count, and handle signal interruptions through following steps:
For example, following is a function that reads n bytes and doesn't buffer
ssize_t rio_readn(int fd, void *usrbuf, size_t n)
{
size_t nleft = n;
ssize_t nread;
char *bufp = usrbuf;
while (nleft > 0) {
if ((nread = read(fd, bufp, nleft)) < 0) {
if (errno == EINTR) /* Interrupted by sig handler return */
nread = 0;/* and call read() again */
else
return -1;/* errno set by read() */
}
else if (nread == 0)
break;/* EOF */
nleft -= nread;
bufp += nread;
}
return (n - nleft);/* Return >= 0 */
}
We can implement the following steps to do buffered and robust IO operations (note RIO means robust IO):
Step 1: Set up empty read buffer and associate an open file descriptor so that we can implement our robust IO operations
#define RIO_BUFSIZE 8192
typedef struct {
int rio_fd;/* Descriptor for this internal buf */
int rio_cnt;/* Unread bytes in internal buf */
char *rio_bufptr;/* Next unread byte in internal buf */
char rio_buf[RIO_BUFSIZE]; /* Internal buffer */
} rio_t;
//Initialize robust IO buffer
void rio_readinitb(rio_t *rp, int fd)
{
rp->rio_fd = fd;
rp->rio_cnt = 0;
rp->rio_bufptr = rp->rio_buf;
}
Step 2: A robust read utility function to handle short count
static ssize_t rio_read(rio_t *rp, char *usrbuf, size_t n)
{
int cnt;
while (rp->rio_cnt <= 0) {/* Refill if buf is empty */
rp->rio_cnt = read(rp->rio_fd, rp->rio_buf,
sizeof(rp->rio_buf));
if (rp->rio_cnt < 0) {
if (errno != EINTR) /* Interrupted by sig handler return */
return -1;
}
else if (rp->rio_cnt == 0)/* EOF */
return 0;
else
rp->rio_bufptr = rp->rio_buf; /* Reset buffer ptr */
}
/* Copy min(n, rp->rio_cnt) bytes from internal buf to user buf */
cnt = n;
if (rp->rio_cnt < n)
cnt = rp->rio_cnt;
memcpy(usrbuf, rp->rio_bufptr, cnt);
rp->rio_bufptr += cnt;
rp->rio_cnt -= cnt;
return cnt;
}
Step 3: A robust IO function for buffered reading
ssize_t rio_readnb(rio_t *rp, void *usrbuf, size_t n)
{
size_t nleft = n;
ssize_t nread;
char *bufp = usrbuf;
while (nleft > 0) {
if ((nread = rio_read(rp, bufp, nleft)) < 0) {
if (errno == EINTR) /* Interrupted by sig handler return */
nread = 0;/* Call read() again */
else
return -1;/* errno set by read() */
}
else if (nread == 0)
break;/* EOF */
nleft -= nread;
bufp += nread;
}
return (n - nleft);/* Return >= 0 */
}
I'm currently implementing a ping/pong buffering scheme to safely write a file to disk. I'm using C++/Boost on a Linux/CentOS machine. Now I'm facing the problem to force the actual write of the file to disk. Is it possible to do so irrespective of all the caching policies of the filesystem (ext3/ext4) / SO custom rules / RAID controller / harddisk controller ?
Is it best to use plain fread()/fwrite(), c++ ostream or boost filesystem?
I've heard that simply flushing out the file (fflush()) doesn't guarantee the actual write
fflush (for FILE*), std::flush (for IOStream) to force your program to send to the OS.
POSIX has
sync(2) to ask to schedule writing its buffers, but can return before the writing is done (Linux is waiting that the data is send to the hardware before returning).
fsync(2) which is guaranteed to wait for the data to be send to the hardware, but needs a file descriptor (you can get one from a FILE* with fileno(3), I know of no standard way to get one from an IOStream).
O_SYNC as a flag to open(2).
In all cases, the hardware may have it's own buffers (but if it has control on it, a good implementation will try to flush them also and ISTR that some disks are using capacitors so that they are able to flush whatever happens to the power) and network file systems have their own caveat.
You can use fsync()/fdatasync() to force(Note 1) the data onto the storage.
Those requres a file descriptor, as given by e.g. open().
The linux manpage have more linux specific info, particularly on the difference of fsync and fdatasync.
If you don't use file desciptors directly, many abstractions will contain internal buffers residing in your process.
e.g. if you use a FILE*, you first have to flush the data out of your application.
//... open and write data to a FILE *myfile
fflush(myfile);
fsync(fileno(myfile));
Note 1: These calls force the OS to ensure that any data in any OS cache is written to the drive, and the drive acknowledges that fact. Many hard-drives lie to the OS about this, and might stuff the data in cache memory on the drive.
Not in standard C++. You'll have to use some sort of system specific
IO, like open with the O_SYNC flag under Unix, and then write.
Note that this is partially implicit by the fact that ostream (and in
C, FILE*) are buffered. If you don't know exactly when something is
written to disk, then it doesn't make much sense to insist on the
transactional integrity of the write. (It wouldn't be too hard to
design a streambuf which only writes when you do an explicit flush,
however.)
EDIT:
As a simple example:
class SynchronizedStreambuf : public std::streambuf
{
int myFd;
std::vector<char> myBuffer;
protected:
virtual int overflow( int ch );
virtual int sync();
public:
SynchronizedStreambuf( std::string const& filename );
~SynchronizedStreambuf();
};
int SynchronizedStreambuf::overflow( int ch )
{
if ( myFd == -1 ) {
return traits_type::eof();
} else if ( ch == traits_type::eof() ) {
return sync() == -1 ? traits_type::eof() : 0;
} else {
myBuffer.push_back( ch );
size_t nextPos = myBuffer.size();
myBuffer.resize( 1000 );
setp( &myBuffer[0] + nextPos, &myBuffer[0] + myBuffer.size() );
return ch;
}
}
int SynchronizedStreambuf::sync()
{
size_t toWrite = pptr() - &myBuffer[0];
int result = (toWrite == 0 || write( myFd, &myBuffer[0], toWrite ) == toWrite ? 0 : -1);
if ( result == -1 ) {
close( myFd );
setp( NULL, NULL );
myFd = -1;
} else {
setp( &myBuffer[0], &myBuffer[0] + myBuffer.size() );
}
return result;
}
SynchronizedStreambuf::SynchronizedStreambuf( std::string const& filename )
: myFd( open( filename.c_str(), O_WRONLY | O_CREAT | O_SYNC, 0664 ) )
{
}
SynchronizedStreambuf::~SynchronizedStreambuf()
{
sync();
close( myFd );
}
(This has only been superficially tested, but the basic idea is there.)
I have a relatively simple web server I have written in C++. It works fine for serving text/html pages, but the way it is written it seems unable to send binary data and I really need to be able to send images.
I have been searching and searching but can't find an answer specific to this question which is written in real C++ (fstream as opposed to using file pointers etc.) and whilst this kind of thing is necessarily low level and may well require handling bytes in a C style array I would like the the code to be as C++ as possible.
I have tried a few methods, this is what I currently have:
int sendFile(const Server* serv, const ssocks::Response& response, int fd)
{
// some other stuff to do with headers etc. ........ then:
// open file
std::ifstream fileHandle;
fileHandle.open(serv->mBase + WWW_D + resource.c_str(), std::ios::binary);
if(!fileHandle.is_open())
{
// error handling code
return -1;
}
// send file
ssize_t buffer_size = 2048;
char buffer[buffer_size];
while(!fileHandle.eof())
{
fileHandle.read(buffer, buffer_size);
status = serv->mSock.doSend(buffer, fd);
if (status == -1)
{
std::cerr << "Error: socket error, sending file\n";
return -1;
}
}
return 0
}
And then elsewhere:
int TcpSocket::doSend(const char* message, int fd) const
{
if (fd == 0)
{
fd = mFiledes;
}
ssize_t bytesSent = send(fd, message, strlen(message), 0);
if (bytesSent < 1)
{
return -1;
}
return 0;
}
As I say, the problem is that when the client requests an image it won't work. I get in std::cerr "Error: socket error sending file"
EDIT : I got it working using the advice in the answer I accepted. For completeness and to help those finding this post I am also posting the final working code.
For sending I decided to use a std::vector rather than a char array. Primarily because I feel it is a more C++ approach and it makes it clear that the data is not a string. This is probably not necessary but a matter of taste. I then counted the bytes read for the stream and passed that over to the send function like this:
// send file
std::vector<char> buffer(SEND_BUFFER);
while(!fileHandle.eof())
{
fileHandle.read(&buffer[0], SEND_BUFFER);
status = serv->mSock.doSend(&buffer[0], fd, fileHandle.gcount());
if (status == -1)
{
std::cerr << "Error: socket error, sending file\n";
return -1;
}
}
Then the actual send function was adapted like this:
int TcpSocket::doSend(const char* message, int fd, size_t size) const
{
if (fd == 0)
{
fd = mFiledes;
}
ssize_t bytesSent = send(fd, message, size, 0);
if (bytesSent < 1)
{
return -1;
}
return 0;
}
The first thing you should change is the while (!fileHandle.eof()) loop, because that will not work as you expect it to, in fact it will iterate once too many because the eof flag isn't set until after you try to read from beyond the end of the file. Instead do e.g. while (fileHandle.read(...)).
The second thing you should do is to check how many bytes was actually read from the file, and only send that amount of bytes.
Lastly, you read binary data, not text, so you can't use strlen on the data you read from the file.
A little explanations of the binary file problem: As you should hopefully know, C-style strings (the ones you use strlen to get the length of) are terminated by a zero character '\0' (in short, a zero byte). Random binary data can contain lots of zero bytes anywhere inside it, and it's a valid byte and doesn't have any special meaning.
When you use strlen to get the length of binary data there are two possible problems:
There's a zero byte in the middle of the data. This will cause strlen to terminate early and return the wrong length.
There's no zero byte in the data. That will cause strlen to go beyond the end of the buffer to look for the zero byte, leading to undefined behavior.
Suppose I have an ifstream which represents a large file containing lots of sub-files aggregated together. I want to be able to create a "sub" istream from the larger ifstream (given a size and offest) representing a part of the file so other code can read from that substream as if it was an independent istream.
Any ideas on how I might accomplish this?
EDIT
- I would prefer to avoid boost.
This is an example of a streambuf "filter" that reads from a contained streambuf starting at a specified location and reading up to a specified size. You create substreambuf, passing your original streambuf in and substreambuf then translates access so that everything is read from the desired location in the underlying streambuf.
Most of the overhead involved in calling sgetc and snextc from underflow and uflow should optimize away. Many extraction operators work byte by byte, so there should not be additional overhead beyond maintaining the read position within the subsection and checking for the end of the subsection. Of course, reading large chunks of data will be less efficient with this class (although that could be fixed).
This still needs improvements like testing that the requested location is within the underlying streambuf.
class substreambuf : public std::streambuf
{
public:
substreambuf(std::streambuf *sbuf, std::size_t start, std::size_t len) : m_sbuf(sbuf), m_start(start), m_len(len), m_pos(0)
{
std::streampos p = m_sbuf->pubseekpos(start);
assert(p != std::streampos(-1));
setbuf(NULL, 0);
}
protected:
int underflow()
{
if (m_pos + std::streamsize(1) >= m_len)
return traits_type::eof();
return m_sbuf->sgetc();
}
int uflow()
{
if (m_pos + std::streamsize(1) > m_len)
return traits_type::eof();
m_pos += std::streamsize(1);
return m_sbuf->sbumpc();
}
std::streampos seekoff(std::streamoff off, std::ios_base::seekdir way, std::ios_base::openmode which = std::ios_base::in | std::ios_base::out)
{
std::streampos cursor;
if (way == std::ios_base::beg)
cursor = off;
else if (way == std::ios_base::cur)
cursor = m_pos + off;
else if (way == std::ios_base::end)
cursor = m_len - off;
if (cursor < 0 || cursor >= m_len)
return std::streampos(-1);
m_pos = cursor;
if (m_sbuf->pubseekpos(m_start + m_pos, std::ios_base::beg) == std::streampos(-1))
return std::streampos(-1);
return m_pos;
}
std::streampos seekpos(std::streampos sp, std::ios_base::openmode which = std::ios_base::in | std::ios_base::out)
{
if (sp < 0 || sp >= m_len)
return std::streampos(-1);
m_pos = sp;
if (m_sbuf->pubseekpos(m_start + m_pos, std::ios_base::beg) == std::streampos(-1))
return std::streampos(-1);
return m_pos;
}
private:
std::streambuf *m_sbuf;
std::streampos m_start;
std::streamsize m_len;
std::streampos m_pos;
};
It can be used like this
using namespace std;
void somefunc(ifstream &bigifs)
{
substreambuf sbuf(bigifs.rdbuf(),100,100);
//new istream with the substreambuf as its streambuf
istream isub(&sbuf);
//use isub normally
}
This was inspired by Filtering Streambufs
I've done something like this using the Boost.Iostreams library. Look under Tutorial|Writing Devices. The idea is to create a "device" class which implements the low-level interface (read/write/seek) and then instantiate an istream/ostream derived class using your device class to do the actual I/O.
All iostreams put most of their custom logic in their streambuf specializations. fstream (or basic_fstream) initializes istream with an instance of file_buf. Same for stringstream (stringbuf). If you want to roll your own substream stream, you can do it by implementing your own streambuf in terms of a parent stream.
Just a little idea : If you have control over the client side of the code (i.e. the part that uses the input stream), I suggest you modify it to accept two additional parameters, like illustrated below :
// Old code
void ClassUsingInput::SetInput(std::streambuf & inputbuf)
{
// Implementation ...
}
Can become :
// New code
void ClassUsingInput::SetInput(std::streambuf & inputbuf, std::streampos position, std::streamsize size)
{
inputbuf.pubseekpos(position) ;
// internally use size to detect end-of-substream
}
ifstream::tellg() is returning -13 for a certain file.
Basically, I wrote a utility that analyzes some source code; I open all files alphabetically, I start with "Apple.cpp" and it works perfectly.. But when it gets to "Conversion.cpp", always on the same file, after reading one line successfully tellg() returns -13.
The code in question is:
for (int i = 0; i < files.size(); ++i) { /* For each .cpp and .h file */
TextIFile f(files[i]);
while (!f.AtEof()) // When it gets to conversion.cpp (not on the others)
// first is always successful, second always fails
lines.push_back(f.ReadLine());
The code for AtEof is:
bool AtEof() {
if (mFile.tellg() < 0)
FATAL(format("DEBUG - tellg(): %d") % mFile.tellg());
if (mFile.tellg() >= GetSize())
return true;
return false;
}
After it reads successfully the first line of Conversion.cpp, it always crashes with DEBUG - tellg(): -13.
This is the whole TextIFile class (wrote by me, the error may be there):
class TextIFile
{
public:
TextIFile(const string& path) : mPath(path), mSize(0) {
mFile.open(path.c_str(), std::ios::in);
if (!mFile.is_open())
FATAL(format("Cannot open %s: %s") % path.c_str() % strerror(errno));
}
string GetPath() const { return mPath; }
size_t GetSize() { if (mSize) return mSize; const size_t current_position = mFile.tellg(); mFile.seekg(0, std::ios::end); mSize = mFile.tellg(); mFile.seekg(current_position); return mSize; }
bool AtEof() {
if (mFile.tellg() < 0)
FATAL(format("DEBUG - tellg(): %d") % mFile.tellg());
if (mFile.tellg() >= GetSize())
return true;
return false;
}
string ReadLine() {
string ret;
getline(mFile, ret);
CheckErrors();
return ret;
}
string ReadWhole() {
string ret((std::istreambuf_iterator<char>(mFile)), std::istreambuf_iterator<char>());
CheckErrors();
return ret;
}
private:
void CheckErrors() {
if (!mFile.good())
FATAL(format("An error has occured while performing an I/O operation on %s") % mPath);
}
const string mPath;
ifstream mFile;
size_t mSize;
};
Platform is Visual Studio, 32 bit, Windows.
Edit: Works on Linux.
Edit: I found the cause: line endings. Both Conversion and Guid and others had \n instead of \r\n. I saved them with \r\n instead and it worked. Still, this is not supposed to happen is it?
It's difficult to guess without knowing exactly what's in Conversion.cpp. However, using < with stream positions is not defined by the standard. You might want to consider an explicit cast to the correct integer type before formatting it; I don't know what formatting FATAL and format() expect to perform or how the % operator is overloaded. Stream positions don't have to map in a predicatable way to integers, certainly not if the file isn't opened in binary mode.
You might want to consider an alternative implementation for AtEof(). Say something like:
bool AtEof()
{
return mFile.peek() == ifstream::traits_type::eof();
}