The task here is :This is basic C program which must contains two processes.First one writes in a file 60 random numbers.The seconds that needs to be written has to read the file and write only the even number into another new file.So I have this code and the parent process that needs to write the even numbers to a file its not working properly.Also my question is about the structure of the code.Is this the best way to describe the processes.
#include <stdio.h>
#include <fcntl.h>
int main() {
int pid, fd, buf, i, fd2, buf2;
pid = fork();
switch (pid) {
case 0: {
printf("\n Child process is writing numbers. \n");
fd = open("file.dat", O_RDWR | O_CREAT, 0644);
for (i = 0; i < 60; i++) {
buf = rand() % 20 - 10;
printf("\n Writing number %d %d ", i + 1, buf);
write(fd, &buf, sizeof(buf));
}
close(fd);
break;
}
case -1: {
printf("\n Fork error! \n");
}
default: {
wait(pid);
printf("\n Parent process is Copying numbers. \n");
fd = open("file.dat", O_RDONLY);
fd2 = open("file_output.dat", O_RDWR | O_CREAT, 0644);
while (read(fd, &buf, sizeof(buf)) == sizeof(buf)) {
if (buf & ~1)
(buf2 = buf);
write(fd2, &buf2, sizeof(buf2));
printf("Writing number in file_output.dat %d \n", buf2);
}
close(fd2);
close(fd);
}
}
return 0;
}
Your condition is wrong:
if (buf &~ 1) (buf2=buf);
write(fd2, &buf2, sizeof(buf2));
You write the number out regardless, but if it's odd (excepting that you need to fix the even/odd test) then you don't update buf2. If the first number is odd, buf2 is uninitialised.
You need this:
if (buf % 2 != 0) continue;
write(fd2, &buf, sizeof(buf));
printf("Writing number in file_output.dat %d \n", buf);
Note that this might not correctly test negative even numbers. You could test abs(buf) % 2 != 0. The only number that might not be handled would be INT_MIN.
Your test for even numbers is wrong. You exclude bit 1, so any number greater than one will be selected. You effectively test for
if (buf & 0xfffffffe) ...
but, you should test instead for
if (!(buf & 1)) ...
or even
if ((buf % 2) == 0) ...
well, there are a couple of bugs in the code.
it looks like you want to achieve this: the child writes the random numbers and the parent waits for the child to terminate, reads the random numbers and only writes the even ones out to the second file.
however, you have used wait(), which is the wrong function to use. instead use waitpid() like this:
waitpid(pid, NULL, 0);
the other bug is in your code that writes only even numbers. I would write it like this:
if (!(buf & 1))
write(fd2, &buf1, sizeof(buf1));
that way, you dont need buf2 at all. in fact, in your current code, if the number is not even, it will still write a number to the file! It will just write the last even number that it read again.
Why are you using non-portable open/read/write when you could be using portable fopen/fread/fwrite? That makes no sense to me.
Let us suppose your sizeof int == 4. Suppose your first process writes 4 bytes, which your second process reads as two chunks of 2 bytes for whatever reason, thus read returns 2 here in your loop: while (read(fd, &buf, sizeof(buf)) == sizeof(buf)) Is 2 == sizeof (buf)? No. Perhaps your loop is terminating on the wrong condition...
fread solves this problem with two parameters: One for the size of the item and one for the number of items you want. for (size_t x = fread(&buf, sizeof buf, 1, file_ptr); x > 0; x = fread(&buf, sizeof buf, 1, file_ptr)) { /* ... */ }... "Upon successful completion, fread() shall return the number of elements successfully read which is less than nitems only if a read error or end-of-file is encountered." This guarantee doesn't exist for read. You might need to use x to determine how many items to write, since that tells you how many items were read.
Related
I'm currently trying to write to a LTO tape device via C++. My problem is, that subsequent write requests will result in "Invalid argument" errors, while the first call seemingly works just fine.
I have set up the tape drive to use a fixed block size of 64k (Which means, that it can only be written to in chunks of 64k or 64k multiples at a time).
When now trying to write via fwrite() to it, the first call will be successful (or at least return that the requested amount of entries where written), but subsequent calls (with the same parameters) will result in no data being written (fwrite returns 0) and an error of "Invalid argument".
Following is a small sample application thrown together to replicate the issue.
#include <iostream>
#include <unistd.h>
#define BLOCK_SIZE (64 * 1024)
#define DATA_BLOCKS 32
#define byte unsigned char
int main(int argc, char *argv[]) {
if (argc <= 1) {
std::cout << "Missing first parameter: Target file\n";
return 1;
}
// Create file handle to access tape device in RW mode
FILE* handle;
const char* targetFile = argv[1];
if (access(targetFile, R_OK | W_OK) == 0) {
handle = fopen(targetFile, "r+");
printf("Create handle OK: %s\n", targetFile);
} else {
printf("Could not access %s: Missing rad or write permission\n", targetFile);
return 1;
}
// Create an byte array with data for DATA_BLOCKS blocks
byte *data = new byte[BLOCK_SIZE * DATA_BLOCKS];
// Initialize with some data
for (int i = 0; i < BLOCK_SIZE * DATA_BLOCKS; i++) {
data[i] = '5';
}
// Write data in BLOCK_SIZE chunks, blocksToWrite times per fwrite() call, in numberOfWriteCalls fwrite() calls
size_t blocksToWrite = 4;
int numberOfWriteCalls = 5;
size_t written;
for (int i = 0; i < numberOfWriteCalls; i++) {
written = fwrite(data, BLOCK_SIZE, blocksToWrite, handle);
printf("Round %d: Wrote %d entries of expected %d entries\n", i+1, (int) written, (int) blocksToWrite);
// Check if there was an error and if so, print error info to stderr
if (ferror(handle)) {
printf("Error while writing:\n");
perror("");
clearerr(handle);
}
/* Start modification: Added flushing, as pointed out by user7860670 */
fflush(handle);
// Check if there was an error and if so, print error info to stderr
if (ferror(handle)) {
printf("Error while flushing:\n");
perror("");
clearerr(handle);
}
/* End modification: Added flushing, as pointed out by user7860670 */
}
delete[] data;
// Close file handle
fclose(handle);
return 0;
}
This code calls the fwrite function 5 times, each time trying to write 4 blocks of 64k data.
This works fine on normal files, but returns the following output on all of my tape devices:
Create handle OK: /dev/nst0
Round 1: Wrote 4 entries of expected 4 entries
Round 2: Wrote 0 entries of expected 4 entries
Invalid argument
Round 3: Wrote 0 entries of expected 4 entries
Invalid argument
Round 4: Wrote 0 entries of expected 4 entries
Invalid argument
Round 5: Wrote 0 entries of expected 4 entries
Invalid argument
As can be seen, the first call reacts as expected: fwrite return that 4 blocks of data have been written. But all subsequent calls, even while using the same parameters, will return 0 written blocks and an error of "Invalid argument".
Is there any "special sauce" when trying to write to tape files that I'm not aware of, or am I maybe using the fwrite functionality in a wrong way?
I made a simple c++ program for armv7 architecture (compiled with linaro gnueabihf using raspi rootfs) that takes in arguments with baud rate, data, serial port etc and sends it to the selected serial port and receives the response. At least that's the goal of it.
I'm currently using it to send a command to disable/enable backlight on an industrial screen through an UART port. The screen takes a simple text command ended with crlf and returns a response. The specification of the screen says it uses 9600 baud, no parity, 8 data bits and 1 stop bit for communication, so pretty much standard.
While the sending works flawlessly - I cannot seem to find a way to properly receive the response. I tried configuring the termios port structure in multiple different ways (disabling hardware control, using cfmakeraw, configuring the VMIN and VTIME values) but without luck.
First thing is that, I'm receiving all the input byte by byte (so each read() call returns exactly 1 byte..), but that wouldn't be a problem.
When using nonblocking mode without select() I'm receiving all bytes, but I don't know when to stop receiving (and I want it to be universal, so I send a command, expect a simple response and if there is no more data then just exit). I made a time counter since the last message, so if nothing was received in last ~500ms then I assume nothing more will come. But this sometimes loses some bytes of the response and I don't know why.
When using blocking mode, I receive correct bytes (still byte by byte though), but I don't know when to stop and the last call to read() leaves the program hanging, because nothing else comes in the input.
When adding select() to the blocking call, to see if input is readable, I get very frequent data loss (sometimes just receiving a few bytes), and sometimes select returns 1, but read() blocks, and I'm left hanging.
When I just send data without doing any reading, and look at the input using cat -v < /dev/ttyS3 I can actually see correct input on the serial port all the time, however when I run both cat and my program as receivers, only one of them gets the data (or cat receives a few bytes and my program a few), this suggests me that something is "stealing" my bytes the same way when I try to read it, but what could it be, and why is it like that?
My current code (using the nonblocking read + 500ms timeout), that still loses some bytes from time to time:
#include <stdio.h>
#include <fcntl.h>
#include <termios.h>
#include <unistd.h>
#include <string.h>
#include <stdlib.h>
#include <errno.h>
#include <time.h>
int open_port(char* portname)
{
int fd; // file description for the serial port
fd = open(portname, O_RDWR | O_NOCTTY | O_NDELAY);
if(fd == -1) // if open is unsucessful
{
printf("Error: open_port: Unable to open %s. \n", portname);
}
else
{
//fcntl(fd, F_SETFL, 0);
fcntl(fd, F_SETFL, FNDELAY);
}
return(fd);
}
int configure_port(int fd, int baud_rate)
{
struct termios port_settings;
tcgetattr(fd, &port_settings);
cfsetispeed(&port_settings, baud_rate); // set baud rates
cfsetospeed(&port_settings, baud_rate);
cfmakeraw(&port_settings);
port_settings.c_cflag &= ~PARENB; // no parity
port_settings.c_cflag &= ~CSTOPB; // 1 stop bit
port_settings.c_cflag &= ~CSIZE;
port_settings.c_cflag |= CS8; // 8 data bits
tcsetattr(fd, TCSANOW, &port_settings); // apply the settings to the port
return(fd);
}
/**
* Convert int baud rate to actual baud rate from termios
*/
int get_baud(int baud)
{
switch (baud) {
case 9600:
return B9600;
case 19200:
return B19200;
case 38400:
return B38400;
case 57600:
return B57600;
case 115200:
return B115200;
case 230400:
return B230400;
case 460800:
return B460800;
case 500000:
return B500000;
case 576000:
return B576000;
case 921600:
return B921600;
case 1000000:
return B1000000;
case 1152000:
return B1152000;
case 1500000:
return B1500000;
case 2000000:
return B2000000;
case 2500000:
return B2500000;
case 3000000:
return B3000000;
case 3500000:
return B3500000;
case 4000000:
return B4000000;
default:
return -1;
}
}
unsigned char* datahex(char* string) {
if(string == NULL)
return NULL;
size_t slength = strlen(string);
if((slength % 2) != 0) // must be even
return NULL;
size_t dlength = slength / 2;
unsigned char* data = (unsigned char*)malloc(dlength);
memset(data, 0, dlength);
size_t index = 0;
while (index < slength) {
char c = string[index];
int value = 0;
if(c >= '0' && c <= '9')
value = (c - '0');
else if (c >= 'A' && c <= 'F')
value = (10 + (c - 'A'));
else if (c >= 'a' && c <= 'f')
value = (10 + (c - 'a'));
else {
free(data);
return NULL;
}
data[(index/2)] += value << (((index + 1) % 2) * 4);
index++;
}
return data;
}
int main(int argc, char **argv) {
int baud_rate = B9600;
baud_rate = get_baud(atoi(argv[1]));
if(baud_rate == -1) {
printf("Error: Cannot convert baud rate %s, using 9600\n", argv[1]);
baud_rate = B9600;
}
bool convertHex = false;
char portName[24] = "/dev/ttyS0";
bool debug = false;
bool noreply = false;
for(int i = 3; i < argc; i++) {
if(!strcmp(argv[i], "hex"))
convertHex = true;
else if(strstr(argv[i], "/dev/") != NULL)
strncpy(portName, argv[i], sizeof(portName));
else if(!strcmp(argv[i], "debug"))
debug = true;
else if(!strcmp(argv[i], "no-reply"))
noreply = true;
}
unsigned char* data = nullptr;
size_t len = 0;
if(convertHex) {
data = datahex(argv[2]);
if((int)data == (int)NULL) {
convertHex = false;
printf("Error: Couldn't convert hex value! Needs to be even length (2 chars per byte)\n");
}
else
len = strlen(argv[2])/2;
}
if(!convertHex) {
data = (unsigned char*)argv[2];
len = strlen(argv[2]);
}
int fd = open_port(portName);
if(fd == -1) {
printf("Error: Couldn't open port %s\n", portName);
if(convertHex)
free(data);
return 0;
}
configure_port(fd, baud_rate);
if(debug) {
printf("Sending data (raw): ");
for(int i =0; i< len; i++) {
printf("%02X", data[i]);
}
printf("\n");
}
size_t writelen = write(fd, data, len);
if(debug)
printf("Sent %d/%d bytes\n", writelen, len);
if(writelen != len)
printf("Error: not all bytes were sent (%d/%d)\n", writelen, len);
else if(noreply)
printf("WRITE OK");
if(!noreply) {
unsigned char ibuff[512] = {0};
int curlen = 0; // full length
clock_t begin_time = clock();
while(( float(clock() - begin_time) / CLOCKS_PER_SEC) < 0.5 && curlen < sizeof(ibuff)) {
int ret = read(fd, ibuff+curlen, sizeof(ibuff)-curlen-1);
if(ret < 0) {
ret = 1;
continue;
}
if(ret > 0) {
curlen += ret;
begin_time = clock();
}
}
if(curlen > 0) {
ibuff[curlen] = 0; // null terminator
printf("RESPONSE: %s", ibuff);
}
}
if(fd)
close(fd);
if(convertHex)
free(data);
return 0;
}
I launch the program like ./rs232 9600 [hex string] hex debug
The scren should return a response like #BLIGHT_ON!OK, but sometimes I receive for example #BLI_ON!O
What can be the cause of this? I made some serial communcation earlier with QtSerial <-> STM32 controller and had no such issues that would cause data loss.
First thing is that, I'm receiving all the input byte by byte (so each
read() call returns exactly 1 byte..) [...]
That's not surprising. The response is coming back at 9600 baud, which is likely much slower per byte than one iteration of the loop requires. It would also arise directly from some configurations of the serial driver. It should be possible to tune this by manipulating VMIN and VTIME, but do note that that requires disabling canonical mode (which you probably want to do anyway; see below).
When using nonblocking mode without select() I'm receiving all bytes,
but I don't know when to stop receiving (and I want it to be
universal, so I send a command, expect a simple response and if there
is no more data then just exit). I made a time counter since the last
message, so if nothing was received in last ~500ms then I assume
nothing more will come. But this sometimes loses some bytes of the
response and I don't know why.
It's all in the details, which you have not presented for that case. We cannot therefore speak to your particular data losses.
Generally speaking, if you're working without flow control, then you have to be sure to read each byte before the next one arrives, on average, else pretty soon, new bytes will overwrite previously-received ones. VMIN and VTIME can help with that, or one can try other methods for tune read timing, but note well that a 9600 baud response will deliver bytes at a rate exceeding one per millisecond, so a 500 ms delay between read attempts is much too long. Supposing that the particular responses you are trying to read are relatively short, however, this will not explain the data losses.
When using blocking mode, I receive correct bytes (still byte by byte
though), but I don't know when to stop and the last call to read()
leaves the program hanging, because nothing else comes in the input.
So the command is required to be CRLF-terminated, but the response cannot be relied upon to be likewise terminated? What a rude device you're working with. If it terminated its responses the same way it required terminated commands, then you could probably work in canonical mode, and you could definitely watch for the terminator to recognize end-of-transmission.
When adding select() to the blocking call, to see if input is
readable, I get very frequent data loss (sometimes just receiving a
few bytes), and sometimes select returns 1, but read() blocks, and I'm
left hanging.
I cannot suggest what the problem may be in that case without any relevant code to analyze, but you really shouldn't need select() for this.
When I just send data without doing any reading, and look at the input
using cat -v < /dev/ttyS3 I can actually see correct input on the
serial port all the time,
That's a good test.
however when I run both cat and my program
as receivers, only one of them gets the data (or cat receives a few
bytes and my program a few),
That's exactly as I would expect. Once a program reads a byte from the port, it is no longer available for any other program to read. Thus, if multiple programs try to read from the same port at the same time then the data available will be partitioned among them in some unspecified and not necessarily consistent fashion.
this suggests me that something is
"stealing" my bytes the same way when I try to read it, but what could
it be, and why is it like that?
That seems unlikely, considering that cat is not affected the same way when you run it alone, nor (you report) are some versions of your own program.
In the first place, if the device supports flow control then I would enable it. Hardware flow control in preference to software flow control if both are viable. This is mainly a fail-safe, however -- I don't see any reason to think that flow control is likely to actually trigger if your program is well written.
Mainly, then, in addition to setting the serial line parameters (8/n/1), you should
Disable canonical mode. This is necessary because you (apparently) cannot rely on the response to be terminated by a line terminator, among other reasons.
Disable echo.
Avoid enabling non-blocking mode on the file.
(Optional) read the first response byte with VMIN == 1 and VTIME == 0; this allows for an arbitrary delay before the device starts sending the response. Alternatively, if you have a reliable upper bound on the time you're willing to wait for the device to start sending the response then you can probably skip this step by using a suitable VTIME in the next one. Or perhaps use a a larger VTIME for this first byte to accommodate a delay before start of transmission, yet not hang if the device fails to respond.
Do read the remaining response bytes with VTIME == 1 (or larger) and VMIN == 0. This probably gets you the whole remainder of the response in one call, but do repeat the read() until it returns 0 (or negative). The 0 return indicates that all available bytes have been transferred and no new ones were received for VTIME tenths of a second -- much longer than the inter-character time in a 9600-baud transmission even for VTIME == 1. Do note that the larger you make VTIME, the longer will be the delay between the device sending the last byte of its response and the program detecting end-of-transmission.
Do not implement any artificial delay between successive read attempts.
You should not need non-blocking mode at the fcntl level, and you should not need select(). There may be other termios settings you could apply to better tune your program for the particular device at the other end of the serial link, but the above should be enough for single-command / single-response pairs with ASCII-only data and no control characters other than carriage returns and newlines.
Hie, everyone! I have a simple TCP server and client on winsock2 lib c++. The server simply send string messages. The client simply receives them. Everything is fine here. But when I use the zlib library to compress the string, the data is corrupting and I can't properly receive them on the client to unzip. Can someone help me?
Server:
{
std::lock_guard<std::mutex> lock(mtx);
std::cout << "Client connected\n";
int k = rand() % strings.size();
msg = strings[k];
msg_size = msg.size();
msgl_size = msg_size + msg_size*0.1 + 12;
msgl = new unsigned char[msgl_size + 1]{0};
if (Z_OK != compress((Bytef*)msgl,
&msgl_size,
reinterpret_cast<const unsigned char*>(msg.c_str()),
msg.size()))
{
std::cout << "Compression error! " << std::endl;
exit(2);
}
}
std::thread * thread = new std::thread([&newConnection, msgl, msgl_size, msg_size, msg]() {
std::lock_guard<std::mutex> lock(mtx);
send(newConnection, (char*)&msgl_size, sizeof(unsigned long), NULL);
send(newConnection, (char*)&msg_size, sizeof(unsigned long), NULL);
int res;
do {
res = send(newConnection, (char*)(msgl), sizeof(msgl_size), NULL);
}
while (msgl_size != res);
});
Client:
std::lock_guard<std::mutex> lock(mtxx);
unsigned long msgl_size, msg_size;
recv(Connection, (char*)&msg_size, sizeof(unsigned long), NULL);
recv(Connection, (char*)&msgl_size, sizeof(unsigned long), NULL);
unsigned char * msgl = new unsigned char[msgl_size + 1]{0};
int res;
do {
res = recv(Connection, reinterpret_cast<char*>(msgl), msgl_size, NULL);
}
while (msgl_size != res);
char * msg = new char[msg_size + 1];
if (Z_OK == uncompress(reinterpret_cast<unsigned char*>(msg),
&msg_size,
reinterpret_cast<unsigned char*>(msgl),
msgl_size))
{
msg[msg_size] = '\0';
std::cout << msg << std::endl;
std::cout << "Compress ratio: " << msgl_size / (float)msg_size << std::endl;
}
delete[] msgl;
Client side:
recv only returns whatever data is immediately available or blocks until data becomes available, this is unlikely to happen with a large file or a slow network. Quite likely recv will block until the first network packet arrives and depending on the underlying network that could be anywhere from a few hundred bytes to tens of thousands. Maybe the message fits in that and maybe not.
Setting recv's flags parameter to MSG_WAITALL is useful for shorter messages because you will either get exactly the number of bytes what you asked for or an error. Because of the possibility for error you always have to test the return value.
To repeat: Always check the return value.
recv's return value is either negative for socket failure, 0 for socket shutdown, or the number of bytes read. For more, consult the winsock documentation for recv.
So...
recv(Connection, (char*)&msg_size, sizeof(unsigned long), NULL);
and
recv(Connection, (char*)&msgl_size, sizeof(unsigned long), NULL);
do not check the return value. The socket could have failed or the call to recv could have returned less than what was requested and the remainder of the program will be operating on garbage.
These are a decent place to use MSG_WAITALL, but it's possible that the socket is fine and you were interrupted by a signal. Not sure if this can happen on Windows, but it can on Linux. Beware.
if (recv(Connection, (char*)&msg_size, sizeof(unsigned long), MSG_WAITALL) != sizeof(unsigned long) &&
recv(Connection, (char*)&msgl_size, sizeof(unsigned long), NULL) != sizeof(unsigned long)(
{
// log error
// exit function, loop, or whatever.
}
Next,
do {
res = recv(Connection, reinterpret_cast<char*>(msgl), msgl_size, NULL);
} while (msgl_size != res);
will loop until one recv returns exactly the right amount in a single call. Unlikely, but if it does, it must happen on the first read because the code writes over the previous read every time.
Say only 1/2 of the message is read from the socket on the first try. Since this isn't the full message, the loop enters and tries to read again, overwriting the first half of the message with the second half and perhaps enough bytes from the subsequent message to satisfy the requested number of bytes. This amalgam of two messages will not decrypt.
For a payload of potentially great size, loop until the program has it all.
char * bufp = reinterpret_cast<char*>(msgl);
int msg_remaining = msgl_size;
while (msg_remaining )
{
res = recv(Connection, bufp, msg_remaining, NULL);
if (res <= 0)
{
// log error
// exit function, loop, or whatever.
}
msg_remaining -= res; // reduce message remaining
bufp += res; // move next insert point in msgl
}
There may be problems with the decompression. I don't know enough about that to be able to answer. I suggest removing it and sending easily-debuggable plaintext until you have all of the network issues worked out.
Server side:
Like recv, send sends what it can. You may have to loop sending to make sure you didn't overfill the socket with a message too large for the socket to eat in one shot. And again like recv, ssend can fail. Always check the return value to see what really happened. Check the documentation for send for more information.
It looks to me like you have the right basic idea: send the size of data to expect, followed by the data itself. On the receiving side, read the size first, then read the specified amount of data.
Unfortunately, you've made a mistake or two when it came to the details of implementing that intent. The first big one is when you send the data:
do {
res = send(newConnection, (char*)(msgl), sizeof(msgl_size), NULL);
}
while (msgl_size != res);
This has a couple of problems. First of all, it uses sizeof(msg1_size), so it's only trying to send the size of an unsigned long (at least I'm guessing that msg1_size is an unsigned long).
What I'm pretty sure you intended here was to send the entire buffer instead:
unsigned long sent = 0;
unsigned long remaining = msg1_size;
do {
res = send(newConnection, (char*)(msgl + sent), remaining, NULL);
sent += res;
remaining -= res;
} while (msgl_size != sent);
With this, we start sending from the beginning of the buffer. If send returns after sending only part of that (as it's allowed to), we record how much was sent. Then on the next iteration, we re-start sending from the point where it left off. Meanwhile, we keep track of how much remains to be sent, and only attempt to send that much on each subsequent iteration.
At least at first glance, it looks like your receive loop probably needs roughly the same kind of repair, keeping track of the total received rather than trying to wait for a single transfer of the entire amount.
Oh, and of course for real code you also want to check for res being 0 or negative. As it stands right now, this doesn't even attempt to detect or properly react to most network errors.
I am writing a wrapper around generic file operations and do not know how to handle the case when write returns a smaller size written then provided.
The man page for write says:
The number of bytes written may be less than count if, for example, there is insufficient space on the underlying physical medium, or the RLIMIT_FSIZE resource limit is encountered (see setrlimit(2)), or the call was interrupted by a signal handler after having written less than count bytes. (See also pipe(7).)
From my understanding of the above, it's a mixture of errors (medium full) and incitation to come back (interrupted call). If my file descriptors are all non-blocking, I should not get the interrupt case and then the only reason would be an error. Am I right ?
Code example:
int size_written = write(fd, str, count);
if (size_written == -1) {
if (errno == EAGAIN) {
// poll on fd and come back later
} else {
// throw an error
}
} else if (size_written < count) {
// ***************
// what should I do here ?
// throw an error ?
// ***************
}
You need to use the raw I/O functions in a loop:
ssize_t todo = count;
for (ssize_t n; todo > 0; )
{
n = write(fd, str, todo);
if (n == -1 && errno != EINTR)
{
// error
break;
}
str += n;
todo -= n;
}
if (todo != 0) { /* error */ }
The special condition concerning EINTR allows the write call to be interrupted by a signal without causing the entire operation to fail. Otherwise, we expect to be able to write all data eventually.
If you can't finish writing all data because your file descriptor is non-blocking and cannot accept any data at the moment, you have to store the remaining data and try again later when the file descriptor has signalled that it's ready for writing again.
here's a problem I've solved from a programming problem website(codechef.com in case anyone doesn't want to see this solution before trying themselves). This solved the problem in about 5.43 seconds with the test data, others have solved this same problem with the same test data in 0.14 seconds but with much more complex code. Can anyone point out specific areas of my code where I am losing performance? I'm still learning C++ so I know there are a million ways I could solve this problem, but I'd like to know if I can improve my own solution with some subtle changes rather than rewrite the whole thing. Or if there are any relatively simple solutions which are comparable in length but would perform better than mine I'd be interested to see them also.
Please keep in mind I'm learning C++ so my goal here is to improve the code I understand, not just to be given a perfect solution.
Thanks
Problem:
The purpose of this problem is to verify whether the method you are using to read input data is sufficiently fast to handle problems branded with the enormous Input/Output warning. You are expected to be able to process at least 2.5MB of input data per second at runtime. Time limit to process the test data is 8 seconds.
The input begins with two positive integers n k (n, k<=10^7). The next n lines of input contain one positive integer ti, not greater than 10^9, each.
Output
Write a single integer to output, denoting how many integers ti are divisible by k.
Example
Input:
7 3
1
51
966369
7
9
999996
11
Output:
4
Solution:
#include <iostream>
#include <stdio.h>
using namespace std;
int main(){
//n is number of integers to perform calculation on
//k is the divisor
//inputnum is the number to be divided by k
//total is the total number of inputnums divisible by k
int n,k,inputnum,total;
//initialize total to zero
total=0;
//read in n and k from stdin
scanf("%i%i",&n,&k);
//loop n times and if k divides into n, increment total
for (n; n>0; n--)
{
scanf("%i",&inputnum);
if(inputnum % k==0) total += 1;
}
//output value of total
printf("%i",total);
return 0;
}
The speed is not being determined by the computation—most of the time the program takes to run is consumed by i/o.
Add setvbuf calls before the first scanf for a significant improvement:
setvbuf(stdin, NULL, _IOFBF, 32768);
setvbuf(stdout, NULL, _IOFBF, 32768);
-- edit --
The alleged magic numbers are the new buffer size. By default, FILE uses a buffer of 512 bytes. Increasing this size decreases the number of times that the C++ runtime library has to issue a read or write call to the operating system, which is by far the most expensive operation in your algorithm.
By keeping the buffer size a multiple of 512, that eliminates buffer fragmentation. Whether the size should be 1024*10 or 1024*1024 depends on the system it is intended to run on. For 16 bit systems, a buffer size larger than 32K or 64K generally causes difficulty in allocating the buffer, and maybe managing it. For any larger system, make it as large as useful—depending on available memory and what else it will be competing against.
Lacking any known memory contention, choose sizes for the buffers at about the size of the associated files. That is, if the input file is 250K, use that as the buffer size. There is definitely a diminishing return as the buffer size increases. For the 250K example, a 100K buffer would require three reads, while a default 512 byte buffer requires 500 reads. Further increasing the buffer size so only one read is needed is unlikely to make a significant performance improvement over three reads.
I tested the following on 28311552 lines of input. It's 10 times faster than your code. What it does is read a large block at once, then finishes up to the next newline. The goal here is to reduce I/O costs, since scanf() is reading a character at a time. Even with stdio, the buffer is likely too small.
Once the block is ready, I parse the numbers directly in memory.
This isn't the most elegant of codes, and I might have some edge cases a bit off, but it's enough to get you going with a faster approach.
Here are the timings (without the optimizer my solution is only about 6-7 times faster than your original reference)
[xavier:~/tmp] dalke% g++ -O3 my_solution.cpp
[xavier:~/tmp] dalke% time ./a.out < c.dat
15728647
0.284u 0.057s 0:00.39 84.6% 0+0k 0+1io 0pf+0w
[xavier:~/tmp] dalke% g++ -O3 your_solution.cpp
[xavier:~/tmp] dalke% time ./a.out < c.dat
15728647
3.585u 0.087s 0:03.72 98.3% 0+0k 0+0io 0pf+0w
Here's the code.
#include <iostream>
#include <stdio.h>
using namespace std;
const int BUFFER_SIZE=400000;
const int EXTRA=30; // well over the size of an integer
void read_to_newline(char *buffer) {
int c;
while (1) {
c = getc_unlocked(stdin);
if (c == '\n' || c == EOF) {
*buffer = '\0';
return;
}
*buffer++ = c;
}
}
int main() {
char buffer[BUFFER_SIZE+EXTRA];
char *end_buffer;
char *startptr, *endptr;
//n is number of integers to perform calculation on
//k is the divisor
//inputnum is the number to be divided by k
//total is the total number of inputnums divisible by k
int n,k,inputnum,total,nbytes;
//initialize total to zero
total=0;
//read in n and k from stdin
read_to_newline(buffer);
sscanf(buffer, "%i%i",&n,&k);
while (1) {
// Read a large block of values
// There should be one integer per line, with nothing else.
// This might truncate an integer!
nbytes = fread(buffer, 1, BUFFER_SIZE, stdin);
if (nbytes == 0) {
cerr << "Reached end of file too early" << endl;
break;
}
// Make sure I read to the next newline.
read_to_newline(buffer+nbytes);
startptr = buffer;
while (n>0) {
inputnum = 0;
// I had used strtol but that was too slow
// inputnum = strtol(startptr, &endptr, 10);
// Instead, parse the integers myself.
endptr = startptr;
while (*endptr >= '0') {
inputnum = inputnum * 10 + *endptr - '0';
endptr++;
}
// *endptr might be a '\n' or '\0'
// Might occur with the last field
if (startptr == endptr) {
break;
}
// skip the newline; go to the
// first digit of the next number.
if (*endptr == '\n') {
endptr++;
}
// Test if this is a factor
if (inputnum % k==0) total += 1;
// Advance to the next number
startptr = endptr;
// Reduce the count by one
n--;
}
// Either we are done, or we need new data
if (n==0) {
break;
}
}
// output value of total
printf("%i\n",total);
return 0;
}
Oh, and it very much assumes the input data is in the right format.
try to replace if statement with count += ((n%k)==0);. that might help little bit.
but i think you really need to buffer your input into temporary array. reading one integer from input at a time is expensive. if you can separate data acquisition and data processing, compiler may be able to generate optimized code for mathematical operations.
The I/O operations are bottleneck. Try to limit them whenever you can, for instance load all data to a buffer or array with buffered stream in one step.
Although your example is so simple that I hardly see what you can eliminate - assuming it's a part of the question to do subsequent reading from stdin.
A few comments to the code: Your example doesn't make use of any streams - no need to include iostream header. You already load C library elements to global namespace by including stdio.h instead of C++ version of the header cstdio, so using namespace std not necessary.
You can read each line with gets(), and parse the strings yourself without scanf(). (Normally I wouldn't recommend gets(), but in this case, the input is well-specified.)
A sample C program to solve this problem:
#include <stdio.h>
int main() {
int n,k,in,tot=0,i;
char s[1024];
gets(s);
sscanf(s,"%d %d",&n,&k);
while(n--) {
gets(s);
in=s[0]-'0';
for(i=1; s[i]!=0; i++) {
in=in*10 + s[i]-'0'; /* For each digit read, multiply the previous
value of in with 10 and add the current digit */
}
tot += in%k==0; /* returns 1 if in%k is 0, 0 otherwise */
}
printf("%d\n",tot);
return 0;
}
This program is approximately 2.6 times faster than the solution you gave above (on my machine).
You could try to read input line by line and use atoi() for each input row. This should be a little bit faster than scanf, because you remove the "scan" overhead of the format string.
I think the code is fine. I ran it on my computer in less than 0.3s
I even ran it on much larger inputs in less than a second.
How are you timing it?
One small thing you could do is remove the if statement.
start with total=n and then inside the loop:
total -= int( (input % k) / k + 1) //0 if divisible, 1 if not
Though I doubt CodeChef will accept it, one possibility is to use multiple threads, one to handle the I/O, and another to process the data. This is especially effective on a multi-core processor, but can help even with a single core. For example, on Windows you code use code like this (no real attempt at conforming with CodeChef requirements -- I doubt they'll accept it with the timing data in the output):
#include <windows.h>
#include <process.h>
#include <iostream>
#include <time.h>
#include "queue.hpp"
namespace jvc = JVC_thread_queue;
struct buffer {
static const int initial_size = 1024 * 1024;
char buf[initial_size];
size_t size;
buffer() : size(initial_size) {}
};
jvc::queue<buffer *> outputs;
void read(HANDLE file) {
// read data from specified file, put into buffers for processing.
//
char temp[32];
int temp_len = 0;
int i;
buffer *b;
DWORD read;
do {
b = new buffer;
// If we have a partial line from the previous buffer, copy it into this one.
if (temp_len != 0)
memcpy(b->buf, temp, temp_len);
// Then fill the buffer with data.
ReadFile(file, b->buf+temp_len, b->size-temp_len, &read, NULL);
// Look for partial line at end of buffer.
for (i=read; b->buf[i] != '\n'; --i)
;
// copy partial line to holding area.
memcpy(temp, b->buf+i, temp_len=read-i);
// adjust size.
b->size = i;
// put buffer into queue for processing thread.
// transfers ownership.
outputs.add(b);
} while (read != 0);
}
// A simplified istrstream that can only read int's.
class num_reader {
buffer &b;
char *pos;
char *end;
public:
num_reader(buffer *buf) : b(*buf), pos(b.buf), end(pos+b.size) {}
num_reader &operator>>(int &value){
int v = 0;
// skip leading "stuff" up to the first digit.
while ((pos < end) && !isdigit(*pos))
++pos;
// read digits, create value from them.
while ((pos < end) && isdigit(*pos)) {
v = 10 * v + *pos-'0';
++pos;
}
value = v;
return *this;
}
// return stream status -- only whether we're at end
operator bool() { return pos < end; }
};
int result;
unsigned __stdcall processing_thread(void *) {
int value;
int n, k;
int count = 0;
// Read first buffer: n & k followed by values.
buffer *b = outputs.pop();
num_reader input(b);
input >> n;
input >> k;
while (input >> value && ++count < n)
result += ((value %k ) == 0);
// Ownership was transferred -- delete buffer when finished.
delete b;
// Then read subsequent buffers:
while ((b=outputs.pop()) && (b->size != 0)) {
num_reader input(b);
while (input >> value && ++count < n)
result += ((value %k) == 0);
// Ownership was transferred -- delete buffer when finished.
delete b;
}
return 0;
}
int main() {
HANDLE standard_input = GetStdHandle(STD_INPUT_HANDLE);
HANDLE processor = (HANDLE)_beginthreadex(NULL, 0, processing_thread, NULL, 0, NULL);
clock_t start = clock();
read(standard_input);
WaitForSingleObject(processor, INFINITE);
clock_t finish = clock();
std::cout << (float)(finish-start)/CLOCKS_PER_SEC << " Seconds.\n";
std::cout << result;
return 0;
}
This uses a thread-safe queue class I wrote years ago:
#ifndef QUEUE_H_INCLUDED
#define QUEUE_H_INCLUDED
namespace JVC_thread_queue {
template<class T, unsigned max = 256>
class queue {
HANDLE space_avail; // at least one slot empty
HANDLE data_avail; // at least one slot full
CRITICAL_SECTION mutex; // protect buffer, in_pos, out_pos
T buffer[max];
long in_pos, out_pos;
public:
queue() : in_pos(0), out_pos(0) {
space_avail = CreateSemaphore(NULL, max, max, NULL);
data_avail = CreateSemaphore(NULL, 0, max, NULL);
InitializeCriticalSection(&mutex);
}
void add(T data) {
WaitForSingleObject(space_avail, INFINITE);
EnterCriticalSection(&mutex);
buffer[in_pos] = data;
in_pos = (in_pos + 1) % max;
LeaveCriticalSection(&mutex);
ReleaseSemaphore(data_avail, 1, NULL);
}
T pop() {
WaitForSingleObject(data_avail,INFINITE);
EnterCriticalSection(&mutex);
T retval = buffer[out_pos];
out_pos = (out_pos + 1) % max;
LeaveCriticalSection(&mutex);
ReleaseSemaphore(space_avail, 1, NULL);
return retval;
}
~queue() {
DeleteCriticalSection(&mutex);
CloseHandle(data_avail);
CloseHandle(space_avail);
}
};
}
#endif
Exactly how much you gain from this depends on the amount of time spent reading versus the amount of time spent on other processing. In this case, the other processing is sufficiently trivial that it probably doesn't gain much. If more time was spent on processing the data, multi-threading would probably gain more.
2.5mb/sec is 400ns/byte.
There are two big per-byte processes, file input and parsing.
For the file input, I would just load it into a big memory buffer. fread should be able to read that in at roughly full disc bandwidth.
For the parsing, sscanf is built for generality, not speed. atoi should be pretty fast. My habit, for better or worse, is to do it myself, as in:
#define DIGIT(c)((c)>='0' && (c) <= '9')
bool parsInt(char* &p, int& num){
while(*p && *p <= ' ') p++; // scan over whitespace
if (!DIGIT(*p)) return false;
num = 0;
while(DIGIT(*p)){
num = num * 10 + (*p++ - '0');
}
return true;
}
The loops, first over leading whitespace, then over the digits, should be nearly as fast as the machine can go, certainly a lot less than 400ns/byte.
Dividing two large numbers is hard. Perhaps an improvement would be to first characterize k a little by looking at some of the smaller primes. Let's say 2, 3, and 5 for now. If k is divisible by any of these, than inputnum also needs to be or inputnum is not divisible by k. Of course there are more tricks to play (you could use bitwise and of inputnum to 1 to determine whether you are divisible by 2), but I think just removing the low prime possibilities will give a reasonable speed improvement (worth a shot anyway).