I am working on a network programming and I have this code
void WorkHandler::workLoop(){
.
.
.
while(1){
if(remainLength >= MAX_LENGTH)
currentSentLength = send(client->getFd(), sBuffer, MAX_LENGTH, MSG_NOSIGNAL);
else
currentSentLength = send(client->getFd(), sBuffer, remainLength,MSG_NOSIGNAL);
if(currentSentLength == -1){
log("WorkHandler::workLoop, connection has been lost \n");
break;
}
sBuffer += currentSentLength;
remainLength -= currentSentLength;
if(remainLength == 0)
break;
}
}
Also, I am creating a child thread like this
bool WorkHandler::initThreads(){
for(int i=0; i < m_maxThreads; i++){
pthread_t *thread(new pthread_t);
m_workThreadList.push_back(thread);
if(pthread_create(thread, NULL, runWorkThread, reinterpret_cast<void *>(this))!=0){
log("WorkHandler::initThreads, pthread_create error \n");
return false;
}
pthread_detach(*thread);
}
return true;
}
void* WorkHandler::runWorkThread(void *delegate){
printf("WorkHandler::runWorkThread, called\n");
WorkHandler *ptr = reinterpret_cast<WorkHandler*>(delegate);
ptr->workLoop();
return NULL;
}
I am running this code on gdb and it doesn't blow up but it gets stuck at the second send function in the if then else loop. I put log statements every single line and it prints a log right above the second send function and stopped.
currentSentLength = send(client->getFd(), sBuffer, remainLength, MSG_NOSIGNAL);
What might cause this problem and how do I fix this issue?
Thanks in advance..
With blocking IO send will block if the kernel buffer is full and will block untill the clients have read the data. Do you send large chunks? If so, check your client.
If you don't trust clients (they can abuse this to do denial of service attacks) there are a couple of ways to do this properly: poll (with timeout) on the sockets for writeability, send with timeout, use nonblocking I/O, ...
I guess you're calling send() with a negative size...
Your test to exit the while should be
remainLength <= 0
and not
remainLength == 0
Related
I need the simplest most reliable IPC method from one C++ app running on the RPi to another app.
All I'm trying to do is send a string message of 40 characters from one app to another
The first app is running as a service on boot, the other app is started at a later time and is frequently exited and restarted for debugging
The frequent debugging for the second app is whats causing problems with the IPCs I've tried so far
I've tried about 3 different methods and here is where they failed:
File FIFO, the problem is one program hangs while the other program is writing to the file
Shared memory: cannot initialize on one thread and read from another thread. Also frequent exiting while debugging causing GDB crashes with the following GDB command is taking too long to complete -stack-list-frames --thread 1
UDP socket with localhost - same issue as above, plus improper exits block the socket, forcing me to reboot device
Non blocking pipe - not getting any messages on the receiving process
What else can I try? I dont want to get the DBus library, seems too complex for this application.
Any simple server and client code or a link to it would be helpful
Here is my non-blockign pipe code, that doesnt work for me,
I assume its because I dont have a reference to the pipe from one app to the other
Code sourced from here: https://www.geeksforgeeks.org/non-blocking-io-with-pipes-in-c/
char* msg1 = "hello";
char* msg2 = "bye !!";
int p[2], i;
bool InitClient()
{
// error checking for pipe
if(pipe(p) < 0)
exit(1);
// error checking for fcntl
if(fcntl(p[0], F_SETFL, O_NONBLOCK) < 0)
exit(2);
//Read
int nread;
char buf[MSGSIZE];
// write link
close(p[1]);
while (1) {
// read call if return -1 then pipe is
// empty because of fcntl
nread = read(p[0], buf, MSGSIZE);
switch (nread) {
case -1:
// case -1 means pipe is empty and errono
// set EAGAIN
if(errno == EAGAIN) {
printf("(pipe empty)\n");
sleep(1);
break;
}
default:
// text read
// by default return no. of bytes
// which read call read at that time
printf("MSG = % s\n", buf);
}
}
return true;
}
bool InitServer()
{
// error checking for pipe
if(pipe(p) < 0)
exit(1);
// error checking for fcntl
if(fcntl(p[0], F_SETFL, O_NONBLOCK) < 0)
exit(2);
//Write
// read link
close(p[0]);
// write 3 times "hello" in 3 second interval
for(i = 0 ; i < 3000000000 ; i++) {
write(p[0], msg1, MSGSIZE);
sleep(3);
}
// write "bye" one times
write(p[0], msg2, MSGSIZE);
return true;
}
Please consider ZeroMQ
https://zeromq.org/
It is lightweight and has wrapper for all major programming languages.
I'm writing IOCP server for video streaming from desktop client to browser.
Both sides uses WebSocket protocol to unify server's achitecture (and because there is no other way for browsers to perform a full-duplex exchange).
The working thread starts like this:
unsigned int __stdcall WorkerThread(void * param){
int ThreadId = (int)param;
OVERLAPPED *overlapped = nullptr;
IO_Context *ctx = nullptr;
Client *client = nullptr;
DWORD transfered = 0;
BOOL QCS = 0;
while(WAIT_OBJECT_0 != WaitForSingleObject(EventShutdown, 0)){
QCS = GetQueuedCompletionStatus(hIOCP, &transfered, (PULONG_PTR)&client, &overlapped, INFINITE);
if(!client){
if( Debug ) printf("No client\n");
break;
}
ctx = (IO_Context *)overlapped;
if(!QCS || (QCS && !transfered)){
printf("Error %d\n", WSAGetLastError());
DeleteClient(client);
continue;
}
switch(auto opcode = client->ProcessCurrentEvent(ctx, transfered)){
// Client owed to receive some data
case OPCODE_RECV_DEBT:{
if((SOCKET_ERROR == client->Recv()) && (WSA_IO_PENDING != WSAGetLastError())) DeleteClient(client);
break;
}
// Client received all data or the beginning of new message
case OPCODE_RECV_DONE:{
std::string message;
client->GetInput(message);
// Analizing the first byte of WebSocket frame
switch( opcode = message[0] & 0xFF ){
// HTTP_HANDSHAKE is 'G' - from GET HTTP...
case HTTP_HANDSHAKE:{
message = websocket::handshake(message);
while(!client->SetSend(message)) Sleep(1); // Set outgoing data
if((SOCKET_ERROR == client->Send()) && (WSA_IO_PENDING != WSAGetLastError())) DeleteClient(client);
break;
}
// Browser sent a closing frame (0x88) - performing clean WebSocket closure
case FIN_CLOSE:{
websocket::frame frame;
frame.parse(message);
frame.masked = false;
if( frame.pl_len == 0 ){
unsigned short reason = 1000;
frame.payload.resize(sizeof(reason));
frame.payload[0] = (reason >> 8) & 0xFF;
frame.payload[1] = reason & 0xFF;
}
frame.pack(message);
while(!client->SetSend(message)) Sleep(1);
if((SOCKET_ERROR == client->Send()) && (WSA_IO_PENDING != WSAGetLastError())) DeleteClient(client);
shutdown(client->Socket(), SD_SEND);
break;
}
IO context struct:
struct IO_Context{
OVERLAPPED overlapped;
WSABUF data;
char buffer[IO_BUFFER_LENGTH];
unsigned char opcode;
unsigned long long debt;
std::string message;
IO_Context(){
debt = 0;
opcode = 0;
data.buf = buffer;
data.len = IO_BUFFER_LENGTH;
overlapped.Offset = overlapped.OffsetHigh = 0;
overlapped.Internal = overlapped.InternalHigh = 0;
overlapped.Pointer = nullptr;
overlapped.hEvent = nullptr;
}
~IO_Context(){ while(!HasOverlappedIoCompleted(&overlapped)) Sleep(1); }
};
Client Send function:
int Client::Send(){
int var_buf = O.message.size();
// "O" is IO_Context for Output
O.data.len = (var_buf>IO_BUFFER_LENGTH)?IO_BUFFER_LENGTH:var_buf;
var_buf = O.data.len;
while(var_buf > 0) O.data.buf[var_buf] = O.message[--var_buf];
O.message.erase(0, O.data.len);
return WSASend(connection, &O.data, 1, nullptr, 0, &O.overlapped, nullptr);
}
When the desktop client disconnects (it uses just closesocket() to do it, no shutdown()) the GetQueuedCompletionStatus returns TRUE and sets transfered to 0 - in this case WSAGetLastError() returns 64 (The specified network name is no longer available), and it has sense - client disconnected (line with if(!QCS || (QCS && !transfered))). But when the browser disconnects, the error codes confuse me... It can be 0, 997 (pending operation), 87 (invalid parameter)... and no codes related to end of connection.
Why do IOCP select this events? How can it select a pending operation? Why the error is 0 when 0 bytes transferred? Also it leads to endless trying to delete an object associated with the overlapped structure, because the destructor calls ~IO_Context(){ while(!HasOverlappedIoCompleted(&overlapped)) Sleep(1); } for secure deleting. In DeleteClient call the socket is closing with closesocket(), but, as you can see, I'm posting a shutdown(client->Socket(), SD_SEND); call before it (in FIN_CLOSE section).
I understand that there are two sides of a connection and closing it on a server side does not mean that an other side will close it too. But I need to create a stabile server, immune to bad and half opened connections. For example, the user of web application can rapidly press F5 to reload page few times (yeah, some dudes do so :) ) - the connection will reopen few times, and the server must not lag or crash due to this actions.
How to handle this "bad" events in IOCP?
you have many wrong code here.
while(WAIT_OBJECT_0 != WaitForSingleObject(EventShutdown, 0)){
QCS = GetQueuedCompletionStatus(hIOCP, &transfered, (PULONG_PTR)&client, &overlapped, INFINITE);
this is not efficient and wrong code for stop WorkerThread. at first you do excess call WaitForSingleObject, use excess EventShutdown and main this anyway fail todo shutdown. if your code wait for packet inside GetQueuedCompletionStatus that you say EventShutdown - not break GetQueuedCompletionStatus call - you continue infinite wait here. correct way for shutdown - PostQueuedCompletionStatus(hIOCP, 0, 0, 0) instead call SetEvent(EventShutdown) and if worked thread view client == 0 - he break loop. and usually you need have multiple WorkerThread (not single). and multiple calls PostQueuedCompletionStatus(hIOCP, 0, 0, 0) - exactly count of working threads. also you need synchronize this calls with io - do this only after all io already complete and no new io packets will be queued to iocp. so "null packets" must be the last queued to port
if(!QCS || (QCS && !transfered)){
printf("Error %d\n", WSAGetLastError());
DeleteClient(client);
continue;
}
if !QCS - the value in client not initialized, you simply can not use it and call DeleteClient(client); is wrong under this condition
when object (client) used from several thread - who must delete it ? what be if one thread delete object, when another still use it ? correct solution will be if you use reference counting on such object (client). and based on your code - you have single client per hIOCP ? because you retriever pointer for client as completion key for hIOCP which is single for all I/O operation on sockets bind to the hIOCP. all this is wrong design.
you need store pointer to client in IO_Context. and add reference to client in IO_Context and release client in IO_Context destructor.
class IO_Context : public OVERLAPPED {
Client *client;
ULONG opcode;
// ...
public:
IO_Context(Client *client, ULONG opcode) : client(client), opcode(opcode) {
client->AddRef();
}
~IO_Context() {
client->Release();
}
void OnIoComplete(ULONG transfered) {
OnIoComplete(RtlNtStatusToDosError(Internal), transfered);
}
void OnIoComplete(ULONG error, ULONG transfered) {
client->OnIoComplete(opcode, error, transfered);
delete this;
}
void CheckIoError(ULONG error) {
switch(error) {
case NOERROR:
case ERROR_IO_PENDING:
break;
default:
OnIoComplete(error, 0);
}
}
};
then are you have single IO_Context ? if yes, this is fatal error. the IO_Context must be unique for every I/O operation.
if (IO_Context* ctx = new IO_Context(client, op))
{
ctx->CheckIoError(WSAxxx(ctx) == 0 ? NOERROR : WSAGetLastError());
}
and from worked threads
ULONG WINAPI WorkerThread(void * param)
{
ULONG_PTR key;
OVERLAPPED *overlapped;
ULONG transfered;
while(GetQueuedCompletionStatus(hIOCP, &transfered, &key, &overlapped, INFINITE)) {
switch (key){
case '_io_':
static_cast<IO_Context*>(overlapped)->OnIoComplete(transfered);
continue;
case 'stop':
// ...
return 0;
default: __debugbreak();
}
}
__debugbreak();
return GetLastError();
}
the code like while(!HasOverlappedIoCompleted(&overlapped)) Sleep(1); is always wrong. absolute and always. never write such code.
ctx = (IO_Context *)overlapped; despite in your concrete case this give correct result, not nice and can be break if you change definition of IO_Context. you can use CONTAINING_RECORD(overlapped, IO_Context, overlapped) if you use struct IO_Context{
OVERLAPPED overlapped; } but better use class IO_Context : public OVERLAPPED and static_cast<IO_Context*>(overlapped)
now about Why do IOCP select this events? How to handle this "bad" events in IOCP?
the IOCP nothing select. he simply signaling when I/O complete. all. which specific wsa errors you got on different network operation absolute independent from use IOCP or any other completion mechanism.
on graceful disconnect is normal when error code is 0 and 0 bytes transferred in recv operation. you need permanent have recv request active after connection done, and if recv complete with 0 bytes transferred this mean that disconnect happens
I'm creating a simple IRC bot in C++ but I'm coming across some issues I can't seem to resolve. My code is based on Beej's Networking Guide.
Also have I used following code to try and fix my problem.
main.cpp snippet
// as long we don't lose connection, keep receiving data
while(true) {
num_receives++;
if(num_receives == 3) {
snd("NICK q1w2\r\n");
snd("USER q1w2 0 * :q\r\n");
}
if(num_receives == 4) {
snd("JOIN q1w2\r\n");
}
num_bytes = recv(s, buf, MAXDATASIZE-1, 0);
buf[num_bytes]='\0';
std::cout << num_receives << "\t" << buf;
if(num_bytes == -1) {
perror("ERROR receiving");
break;
}
if(num_bytes == 0) {
perror("DISCONNECTED");
break;
}
}
and the output is (first line is command to run the bot)
~$ ./bot irc.snt.utwente.nl 6667
1 :irc.snt.utwente.nl 020 * :Please wait while we process your connection.
2 ERROR :Closing Link: [unknown#91.177.66.186] (Ping timeout)
DISCONNECTED: Network is unreachable
So, the problem is that it only receives data once. Even though it's placed in a while loop. I don't really see what's blocking it from looping or receiving. If I place a cout inside the while loop you can see it only loops once and a second time in the end when it disconnects.
Edit: I have it working. The commands send to the severs should be put outside the while loop. Because inside the loop it's waiting to be initialized. But as w don't receive anything we just lose connection because we didn't send anything either.
[correct] main.cpp snippet
snd("NICK q1w2\r\n");
snd("USER q1w2 0 * :q\r\n");
snd("JOIN q1w2\r\n");
// as long we don't lose connection, keep receiving data
while(true) {
num_receives++;
num_bytes = recv(s, buf, MAXDATASIZE-1, 0);
buf[num_bytes]='\0';
std::cout << num_receives << "\t" << buf;
if(num_bytes == -1) {
perror("ERROR receiving");
break;
}
if(num_bytes == 0) {
perror("DISCONNECTED");
break;
}
}
You forgot to implement the IRC protocol.
Your code assumes that each call to the TCP connection's recv function will return exactly one IRC message. This assumption is totally and completely invalid. The TCP implementation has no idea what an IRC message is. You have to implement the IRC protocol and the IRC protocol's definition of a message if you want to count how many messages you've received.
I am responsible for a server that exports data over a TCP connection. With each data record that the server transmits, it requires the client to send a short "\n" acknowledgement message back. I have a customer who claims that the acknowledgement that he sends is not read from the web server. The following is code that I am using for I/O on the socket:
bool can_send = true;
char tx_buff[1024];
char rx_buff[1024];
struct pollfd poll_descriptor;
int rcd;
poll_descriptor.fd = socket_handle;
poll_descriptor.events = POLLIN | POLLOUT;
poll_descriptor.revents = 0;
while(!should_quit && is_connected)
{
// if we know that data can be written, we need to do this before we poll the OS for
// events. This will prevent the 100 msec latency that would otherwise occur
fill_write_buffer(write_buffer);
while(can_send && !should_quit && !write_buffer.empty())
{
uint4 tx_len = write_buffer.copy(tx_buff, sizeof(tx_buff));
rcd = ::send(
socket_handle,
tx_buff,
tx_len,
0);
if(rcd == -1 && errno != EINTR)
throw SocketException("socket write failure");
write_buffer.pop(rcd);
if(rcd > 0)
on_low_level_write(tx_buff, rcd);
if(rcd < tx_len)
can_send = false;
}
// we will use poll for up to 100 msec to determine whether the socket can be read or
// written
if(!can_send)
poll_descriptor.events = POLLIN | POLLOUT;
else
poll_descriptor.events = POLLIN;
poll(&poll_descriptor, 1, 100);
// check to see if an error has occurred
if((poll_descriptor.revents & POLLERR) != 0 ||
(poll_descriptor.revents & POLLHUP) != 0 ||
(poll_descriptor.revents & POLLNVAL) != 0)
throw SocketException("socket hung up or socket error");
// check to see if anything can be written
if((poll_descriptor.revents & POLLOUT) != 0)
can_send = true;
// check to see if anything can be read
if((poll_descriptor.revents & POLLIN) != 0)
{
ssize_t bytes_read;
ssize_t total_bytes_read = 0;
int bytes_remaining = 0;
do
{
bytes_read = ::recv(
socket_handle,
rx_buff,
sizeof(rx_buff),
0);
if(bytes_read > 0)
{
total_bytes_read += bytes_read;
on_low_level_read(rx_buff,bytes_read);
}
else if(bytes_read == -1)
throw SocketException("read failure");
ioctl(
socket_handle,
FIONREAD,
&bytes_remaining);
}
while(bytes_remaining != 0);
// recv() will return 0 if the socket has been closed
if(total_bytes_read > 0)
read_event::cpost(this);
else
{
is_connected = false;
closed_event::cpost(this);
}
}
}
I have written this code based upon the assumption that poll() is a level triggered function and will unblock immediately as long as there is data to be read from the socket. Everything that I have read seems to back up this assumption. Is there a reason that I may have missed that would cause the above code to miss a read event?
It is not edge triggered. It is always level triggered. I will have to read your code to answer your actual question though. But that answers the question in the title. :-)
I can see no clear reason in your code why you might be seeing the behavior you are seeing. But the scope of your question is a lot larger than the code you're presenting, and I cannot pretend that this is a complete problem diagnosis.
It is level triggered. POLLIN fires if there is data in the socket receive buffer when you poll, and POLLOUT fires if there is room in the socket send buffer (which there almost always is).
Based on your own assessment of the problem (that is, you are blocked on poll when you expect to be able to read the acknowledgement), then you will eventually get a timeout.
If the customer's machine is more than 50ms away from your server, then you will always timeout on the connection before receiving the acknowledgement, since you only wait 100ms. This is because it will take a minimum of 50ms for the data to reach the customer, and a minimum of 50ms for the acknowledgement to return.
Well i make a IOCP for handling client connections with the following details:
- Threads = (CPU cores * 2)
- Assigning an completion port to each socket
- Accessing the socket context by Client Index or overlapped struct (either way is the same)
So i am trying to debug the incoming packets, its works like a charm, except for a little but nasty detail... I set a break point on WorkersThread function (where i recv the packet) i am watching the buffer with the packet i recv, when suddenly the buffer gets overwritten with a new packet that i got from client.
Why is that? according to what i read, IOCP should wait till i process the packet, send a response to client before recv any other packet. So i set a flag on my socket context called "Processing" and still got the overwritten buffer with an incoming packet. So it doesn't let me debug at all and its driving me crazy
Is ollydbg (debugger) fault that let the other threads running while i set a break point? Or is some error in my IOCP implementation?
Here is how my WorkerThread is coded:
DWORD WINAPI WorkerThread(void* argument)
{
int BytesTransfer;
int BytesRecv;
int ClientID;
int result;
OVERLAPPED* overlapped = 0;
ClientInfo* clientinfo = 0;
WSABUF wsabuf;
int flags;
//Exit only when shutdown signal is recv
while (WaitForSingleObject(IOCPBase::internaldata->sockcontext.ShutDownSignal, NULL) != WAIT_OBJECT_0)
{
flags = 0; BytesTransfer = 0; BytesRecv = 0; ClientID = 0;
//Get from queued list
if (GetQueuedCompletionStatus(IOCPBase::internaldata->sockcontext.CompletionPort, (LPDWORD)&BytesTransfer, (PULONG_PTR)&ClientID, &overlapped, INFINITE) == TRUE)
{
if (overlapped == 0)
{
//Fatal error
break;
}
clientinfo = (ClientInfo*)overlapped;
if (BytesTransfer != 0)
{
//Assign the buffer pointer and buffer len to WSABUF local
clientinfo->RecvContext.RecvBytes = BytesTransfer;
wsabuf.buf = (char*)clientinfo->RecvContext.Buffer;
wsabuf.len = clientinfo->RecvContext.Len;
//Switch for OperationCode
//switch (IOCPBase::internaldata->ClientContext[ClientID].OperationCode)
switch (clientinfo->OperationCode)
{
case FD_READ:
// Check if we have send all data to the client from a previous send
if (clientinfo->SendContext.SendBytes < clientinfo->SendContext.TotalBytes)
{
clientinfo->OperationCode = FD_READ; //We set FD_READ caused on the next send, there could still be bytes left to send
wsabuf.buf += clientinfo->SendContext.SendBytes; //The buffer position is + sended bytes
wsabuf.len = clientinfo->SendContext.TotalBytes - clientinfo->SendContext.SendBytes; //the buffer len is total - sended bytes
//Send the remain bytes
result = WSASend(clientinfo->sock, &wsabuf, 1, (LPDWORD)&BytesRecv, flags, &clientinfo->overlapped, NULL);
if (result == SOCKET_ERROR && (WSAGetLastError() != WSA_IO_PENDING))
{
CloseClient(ClientID);
}
clientinfo->SendContext.SendBytes += BytesRecv;
}
else
{
if (clientinfo->Processing == 0)
{
clientinfo->OperationCode = FD_WRITE; //If no more bytes left to send now we can set the operation code to write (in fact is read)
memset(clientinfo->RecvContext.Buffer, NULL, MAX_DATA_BUFFER_SIZE); //Clean the buffer for recv new data
//Recv data from our client
clientinfo->RecvContext.RecvBytes = WSARecv(clientinfo->sock, &wsabuf, 1, (LPDWORD)&BytesRecv, (LPDWORD)&flags, &clientinfo->overlapped, NULL);
if (clientinfo->RecvContext.RecvBytes == SOCKET_ERROR && WSAGetLastError() != WSA_IO_PENDING)
{
CloseClient(ClientID);
break;
}
}
}
break;
case FD_WRITE:
//Send data to the RecvProtocol
clientinfo->Processing = 1;
IOCPBase::internaldata->callback.RecvProtocol(clientinfo->RecvContext.Buffer, clientinfo->RecvContext.Len, ClientID);
clientinfo->Processing = 0;
default:
break;
}
}
}
}
return false;
}
The problem appears when looking at clientinfo->RecvContext.Buffer. I am watching the packet, past a few seconds and boom the buffer is overwritten with a new packet.
Thanks !
Never mind i fix the debug problem by copy the packet to the stack frame of the function i use to analyze the packet, this way i have no overwritten problem.