AMQP-CPP: Broken pipe error in TCP Handler - c++

Unfortunately, in my project I always end up in the onError function in the event handler with the error message "Broken pipe". Unfortunately, I never get into the onConnected state. The monitor Funktion in the Event Handler is called twice with the Flag AMQP::readable. After that, it is called with no flags set, that is the time when my pipe gets broken.
Here is what I do in my code.
First I open the connection:
int Communicator_RabbitMQ::Open(string device)
{
AMQP::Address address(AMQP::Address("amqp://test:test#localhost/"));
// make a connection
m_connection = std::make_shared< AMQP::TcpConnection> (&oCommunicator_RabbitMQ_Handler, address);
// we need a channel too
m_channel = std::make_shared <AMQP::TcpChannel> (m_connection.get());
m_channel->declareExchange("my-exchange", AMQP::fanout);
m_channel->declareQueue("my-queue");
m_channel->bindQueue("my-exchange", "my-queue", "my-routing-key");
m_channel->declareExchange("cyos_tx_exchange", AMQP::direct);
m_channel->declareQueue("cyos_queue");
m_channel->bindQueue("cyos_tx_exchange", "cyos_queue", "");
return true;
}
then I call the read function cyclically in my thread:
string Communicator_RabbitMQ::Read()
{
int result = 0;
int maxfd = 1;
struct timeval tv
{
1, 0
};
string returnValue; //Rückgabe der Methode
string message; // Nachricht aus RabbitMQ
try
{
FD_ZERO(&oCommunicator_RabbitMQ_Handler.m_set);
FD_SET(oCommunicator_RabbitMQ_Handler.m_fd, &oCommunicator_RabbitMQ_Handler.m_set);
if (oCommunicator_RabbitMQ_Handler.m_fd != -1)
{
maxfd = oCommunicator_RabbitMQ_Handler.m_fd + 1;
}
result = select(FD_SETSIZE, &oCommunicator_RabbitMQ_Handler.m_set, NULL, NULL, &tv);
if ((result == -1) && errno == EINTR)
{
TRACE(L"Error in socket");
}
else if (result > 0)
{
if (oCommunicator_RabbitMQ_Handler.m_flags & AMQP::readable)
TRACE(L"Got something");
if (FD_ISSET(oCommunicator_RabbitMQ_Handler.m_fd, &oCommunicator_RabbitMQ_Handler.m_set))
{
m_connection->process(oCommunicator_RabbitMQ_Handler.m_fd, oCommunicator_RabbitMQ_Handler.m_flags);
}
}
}
catch (exception e)
{
cout << e.what();
}
return "";
}
Here is the TCP Event Handler:
#pragma once
class Communicator_RabbitMQ_Handler : public AMQP::TcpHandler
{
private:
/**
* Method that is called when the connection succeeded
* #param socket Pointer to the socket
*/
virtual void onConnected(AMQP::TcpConnection* connection)
{
std::cout << "connected" << std::endl;
}
/**
* When the connection ends up in an error state this method is called.
* This happens when data comes in that does not match the AMQP protocol
*
* After this method is called, the connection no longer is in a valid
* state and can be used. In normal circumstances this method is not called.
*
* #param connection The connection that entered the error state
* #param message Error message
*/
virtual void onError(AMQP::TcpConnection* connection, const char* message)
{
// report error
std::cout << "AMQP TCPConnection error: " << message << std::endl;
}
/**
* Method that is called when the connection was closed.
* #param connection The connection that was closed and that is now unusable
*/
virtual void onClosed(AMQP::TcpConnection* connection)
{
std::cout << "closed" << std::endl;
}
/**
* Method that is called by AMQP-CPP to register a filedescriptor for readability or writability
* #param connection The TCP connection object that is reporting
* #param fd The filedescriptor to be monitored
* #param flags Should the object be monitored for readability or writability?
*/
virtual void monitor(AMQP::TcpConnection* connection, int fd, int flags)
{
//TRACE(L"Communicator_RabbitMQ_Handler, monitor called, %d, %d, %x", fd, flags, &m_set);
// we did not yet have this watcher - but that is ok if no filedescriptor was registered
if (flags == 0)
return;
if (flags & AMQP::readable)
{
FD_SET(fd, &m_set);
m_fd = fd;
m_flags = flags;
}
}
public:
Communicator_RabbitMQ_Handler() = default;
int m_fd = -1;
int m_flags = 0;
fd_set m_set;
};
RabbitMQ Log entry:
2018-07-02 07:04:50.272 [info] <0.9653.0> accepting AMQP connection <0.9653.0> ([::1]:39602 -> [::1]:5672)
2018-07-02 07:04:50.273 [warning] <0.9653.0> closing AMQP connection <0.9653.0> ([::1]:39602 -> [::1]:5672):
{handshake_timeout,handshake}

I finally fixed this problem by increasing the hanshake timeout to 20 seconds in the rabbitmq.config file. I just added the following in that file:
handshake_timeout = 20000
The value is given in milliseconds, an the default is 10 seconds, which seems not enough for my solution.

Related

QUdpSocket client connected to echo server not working

I am developing a QT 6 Widget based UDP audio application that repeatedly sends out a single UDP audio frame sample (4K bytes sine wave tone) to a remote UDP echo server at a predetermined rate - (right now the echo server is hosted locally though).
The UDP echo server is based on the asynchronous UDP echo server sample developed by the asio author (not me). This is shown below (slightly modified to include a hard coded 4K block for testing purposes). The application is also launched with a port parameter 1234 - so it listens on port 1234 for the incoming audio packet that it will echo back to client.
//
// async_udp_echo_server.cpp
// ~~~~~~~~~~~~~~~~~~~~~~~~~
//
// Copyright (c) 2003-2022 Christopher M. Kohlhoff (chris at kohlhoff dot com)
//
// Distributed under the Boost Software License, Version 1.0. (See accompanying
// file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
//
#include <cstdlib>
#include <iostream>
#include <asio/ts/buffer.hpp>
#include <asio/ts/internet.hpp>
using asio::ip::udp;
class server {
public:
server(asio::io_context& io_context, short port)
: socket_(io_context, udp::endpoint(udp::v4(), port)) {
do_receive();
}
void do_receive() {
socket_.async_receive_from(
asio::buffer(data_, max_length), sender_endpoint_,
[this](std::error_code ec, std::size_t bytes_recvd) {
if (!ec && bytes_recvd > 0) {
do_send(bytes_recvd);
} else {
do_receive();
}
});
}
void do_send(std::size_t length) {
socket_.async_send_to(
asio::buffer(data_, length), sender_endpoint_,
[this](std::error_code /*ec*/, std::size_t /*bytes_sent*/) {
do_receive();
});
}
private:
udp::socket socket_;
udp::endpoint sender_endpoint_;
enum { max_length = 4096 };
char data_[max_length]{};
};
int main(int argc, char* argv[]) {
try {
if (argc != 2) {
std::cerr << "Usage: async_udp_echo_server <port>\n";
return 1;
}
asio::io_context io_context;
server s(io_context, std::atoi(argv[1]));
io_context.run();
} catch (std::exception& e) {
std::cerr << "Exception: " << e.what() << "\n";
}
return 0;
}
I currently have this working successfully in the client as a stand alone asio worker thread, however since I need to graphically display the returned audio packets, I cannot use the stand alone asio thread approach; I need to use QT with its signals/slots async magic instead.
For the purposes of illustration, I also include my working asio client code that runs in a separate joinable thread. This client thread uses a asio::steady_timer that fires an asynchronous 4k UDP packet repeatedly to the echo server. The code also compares the echoed back contents to this outgoing audio sample successfully.
void
RTPClient::start() {
mpSendEndpoint = std::make_unique<ip::udp::endpoint>(
ip::address::from_string(mConfig.mHostName),
mConfig.mPortNum);
mpSocket = std::make_unique<ip::udp::socket>(
mIOContext, mpSendEndpoint->protocol());
mpSocketTimer = std::make_unique<steady_timer>(
mIOContext);
mWorker = std::thread([this]() {
mIOContext.run();
});
if (!mShutdownFlag) {
// kick off the async chain by immediate timeout
mpSocketTimer->expires_after(std::chrono::seconds(0));
mpSocketTimer->async_wait([this]<typename T0>(T0&& ec) {
handle_timeout(std::forward<T0>(ec));
});
}
}
void
RTPClient::handle_timeout(const error_code& ec)
{
if (!ec && !mShutdownFlag) {
if (!mpAudioOutput) {
// check to see if there is new audio test data waiting in queue
if (const auto audioData = mIPCQueue->try_pop(); audioData) {
// new audio waiting, copy the data to mpAudioTXData and allocate an identically
// sized receive buffer to receive the echo replies from the server
mpAudioInput = std::make_unique<AudioDatagram>(audioData->first.size());
mpAudioOutput = std::make_unique<AudioDatagram>(std::move(audioData->first));
mAudioBlockUSecs = audioData->second;
} else {
mpSocketTimer->expires_after(seconds(1));
mpSocketTimer->async_wait([this]<typename T0>(T0&& ec) {
handle_timeout(std::forward<T0>(ec));
});
// nothing to send as waveform data not received from GUI.
// short circuit return with a 1 sec poll
return;
}
}
mpSocket->async_send_to(asio::buffer(
mpAudioOutput.get(), mpAudioOutput->size()),
*mpSendEndpoint, [this]<typename T0, typename T1>(T0&& ec, T1&& bytes_transferred) {
handle_send_to(std::forward<T0>(ec), std::forward<T1>(bytes_transferred));
});
}
}
void
RTPClient::handle_send_to(const error_code& ec, std::size_t bytes_transferred) {
if (!ec && bytes_transferred > 0 && !mShutdownFlag) {
mpSocketTimer->expires_after(microseconds(mAudioBlockUSecs));
mpSocketTimer->async_wait([this]<typename T0>(T0&& ec) {
handle_timeout(std::forward<T0>(ec));
});
mpSocket->async_receive_from(asio::buffer(
mpAudioInput.get(), mpAudioInput->size()), *mpSendEndpoint,
[this]<typename T0, typename T1>(T0&& ec, T1&& bytes_transferred) {
handle_receive(std::forward<T0>(ec), std::forward<T1>(bytes_transferred));
});
}
}
void
RTPClient::handle_receive(const error_code& ec, std::size_t bytes_transferred) {
if (!ec && bytes_transferred > 0) {
double foo = 0.0;
for (const auto next : *mpAudioOutput) {
foo += (double)next;
}
double bar = 0.0;
for (const auto next : *mpAudioInput) {
bar += (double)next;
}
if (foo != bar)
{
auto baz = 0;
(void)baz;
}
}
}
/**
* Shutdown the protocol instance by shutting down the IPC
* queue and closing the socket and associated timers etc.
*
* <p>This is achieved by setting a flag which is read by the
* busy loop as an exit condition.
*/
void
RTPClient::shutdown() {
// set the shared shutdown flag
mShutdownFlag = true;
// wake up any locked threads so they can see the above flag
if (mIPCQueue) {
mIPCQueue->shutdown();
}
// stop the socket timer - do not reset it
// as there are some time sensitive parts in the code
// where mpSocketTimer is dereferenced
if (mpSocketTimer) {
mpSocketTimer->cancel();
}
std::error_code ignoredError;
// close the socket if we created & opened it, making
// sure that we close down both ends of the socket.
if (mpSocket && mpSocket->is_open()) {
mpSocket->shutdown(ip::udp::socket::shutdown_both, ignoredError);
// reset so we will reallocate and then reopen
// via boost::async_connect(...) later.
mpSocket.reset();
}
// wait for the any other detached threads to see mShutdownFlag
// as it is running in a detached mWorkerThread which sleeps
// for 50ms CDU key polling requests.
std::this_thread::sleep_for(milliseconds(200));
}
I need to replace this separate asio client thread code with a QUdpSocket based client code to do the equivalent, as I need to use signals/slots to notify the GUI when the blocks arrive and display the returned waveform in a widget. To this end I have the following QT worker thread. I can see that the asio echo server receives the datagram, however I do not know how to receive the echoed contents back into the client. Is there some bind or connect call that I need to do on the client side. I am totally confused with when to call bind and when to call connect on UDP sockets.
// SYSTEM INCLUDES
//#include <..>
// APPLICATION INCLUDES
#include "RTPSession.h"
// DEFINES
// EXTERNAL FUNCTIONS
// EXTERNAL VARIABLES
// CONSTANTS
// STRUCTS
// FUNCTIONS
// NAMESPACE USAGE
using namespace std::chrono;
// STATIC VARIABLE INITIALIZATIONS
std::mutex RTPSession::gMutexGuard;
RTPSession::RTPSession(QObject* parent)
: QObject(parent)
, mpSocket{ std::make_unique<QUdpSocket>(parent) }
{
mpSocket->bind(45454, QUdpSocket::DefaultForPlatform);
connect(mpSocket.get(), &QUdpSocket::readyRead,
this, &RTPSession::processPendingDatagrams);
}
/**
* Thread function that listens RTP session updates.
*
* <p>The implementation polls for shutdown every second.
*
* #param rRTPInfo [in] qt thread parameters.
*/
void
RTPSession::doWork(
const std::tuple<int32_t, int32_t, int32_t>& /*rRTPInfo*/)
{
try {
// just dispatched, so reset exit flag
mExitWorkLoop = false;
int frameCounter = 0;
while (!mExitWorkLoop) {
constexpr auto gPollMillis = 1000;
// poll using shortest (non zero) interval in schedule
std::unique_lock<std::mutex> lk(gMutexGuard);
mCondVariable.wait_for(lk, milliseconds(gPollMillis),
[this] { return mExitWorkLoop; });
QByteArray datagram = "Broadcast message " + QByteArray::number(frameCounter++);
mpSocket->writeDatagram(datagram.data(), datagram.size(),
QHostAddress::LocalHost, 1234);
if (mpSocket->hasPendingDatagrams()) {
//mpSocket->readDatagram()
int t = 0;
(void)t;
}
// update GUI with the audio stats - add more later
emit updateProgress(frameCounter++);
}
} catch (const std::exception& rEx) {
// exit thread with the exception details
emit finishWork(tr("exiting worker, error:") + rEx.what());
}
// exit thread with status bar message
emit finishWork(tr("finished"));
}
void
RTPSession::shutdown()
{
// Critical section.
std::scoped_lock<std::mutex> lock(gMutexGuard);
mExitWorkLoop = true;
// Notify the potentially sleeping thread that is
// waiting for up to 1 second
mCondVariable.notify_one();
}
void
RTPSession::processPendingDatagrams() {
QByteArray datagram;
while (mpSocket->hasPendingDatagrams()) {
datagram.resize(int(mpSocket->pendingDatagramSize()));
mpSocket->readDatagram(datagram.data(), datagram.size());
//statusLabel->setText(tr("Received datagram: \"%1\"")
// .arg(datagram.constData()));
}
}

Event loop handling for sd-bus in libuv

We have an eventloop from libuv to handle unixsockets and TCP sockets. The program now also must handle DBus, and we decided to use sd-bus for that.
Lennart wrote on his blog:
Note that our APIs, including sd-bus, integrate nicely into sd-event
event loops, but do not require it, and may be integrated into other
event loops too, as long as they support watching for time and I/O events.
So i assume, it must be possible.
I can get the dbus socket fd via sd_bus_get_fd (sd_bus *bus).
But I can't find any obvious way to stop sd-bus from using its bus_poll method to wait for events internally.
For example when calling a method with sd_bus_call(...) will block with ppoll.
So: How do I handle the dbus events in libuv?
I figured it out, here's an example on how to unite C++, libuv and sd-bus:
I recommend that you read http://0pointer.de/blog/the-new-sd-bus-api-of-systemd.html to understand sd-bus in general.
These are code snippets from my implementation at https://github.com/TheJJ/horst
Method calls can then be done with sd_bus_call_async which does not block (opposed to sd_bus_call).
Don't forget to call update_events() after sd_bus_call_async so the call is sent out over the socket!
/**
* Callback function that is invoked from libuv
* once dbus events flowed in.
*/
static void on_dbus_ready(uv_poll_t *handle, int /*status*/, int /*events*/) {
DBusConnection *connection = (DBusConnection *)handle->data;
sd_bus *bus = connection->get_bus();
// let dbus handle the available events request
while (true) {
// this will trigger the dbus vtable-registered functions
int r = sd_bus_process(bus, nullptr);
if (r < 0) {
printf("[dbus] Failed to process bus: %s", strerror(-r));
break;
}
else if (r > 0) {
// try to process another request!
continue;
}
else {
// no more progress, wait for the next callback.
break;
}
}
// update the events we watch for on the socket.
connection->update_events();
}
/**
* Convert the sdbus-returned poll flags to
* corresponding libuv flags.
*/
int poll_to_libuv_events(int pollflags) {
int ret = 0;
if (pollflags & (POLLIN | POLLPRI)) {
ret |= UV_READABLE;
}
if (pollflags & POLLOUT) {
ret |= UV_WRITABLE;
}
// we also have the non-corresponding UV_DISCONNECT
return ret;
}
class DBusConnection {
public:
DBusConnection(Satellite *sat);
virtual ~DBusConnection() = default;
/** connect to dbus */
int connect() {
int r = sd_bus_open_system(&this->bus);
if (r < 0) {
printf("[dbus] Failed to connect to bus: %s", strerror(-r));
goto clean_return;
}
r = sd_bus_add_object_vtable(
this->bus,
&this->bus_slot,
"/rofl/lol", // object path
"rofl.lol", // interface name
your_vtable,
this // this is the userdata that'll be passed
// to the dbus methods
);
if (r < 0) {
printf("[dbus] Failed to install the horst sdbus object: %s", strerror(-r));
goto clean_return;
}
// register our service name
r = sd_bus_request_name(this->bus, "moveii.horst", 0);
if (r < 0) {
printf("[dbus] Failed to acquire service name: %s", strerror(-r));
goto clean_return;
}
// register the filedescriptor from
// sd_bus_get_fd(bus) to libuv
uv_poll_init(this->loop, &this->connection, sd_bus_get_fd(this->bus));
// make `this` reachable in callbacks.
this->connection.data = this;
// init the dbus-event-timer
uv_timer_init(this->loop, &this->timer);
this->timer.data = this;
// process initial events and set up the
// events and timers for subsequent calls
on_dbus_ready(&this->connection, 0, 0);
printf("[dbus] Listener initialized");
return 0;
clean_return:
sd_bus_slot_unref(this->bus_slot);
sd_bus_unref(this->bus);
this->bus = nullptr;
return 1;
}
/** update the events watched for on the filedescriptor */
void update_events() {
sd_bus *bus = this->get_bus();
// prepare the callback for calling us the next time.
int new_events = poll_to_libuv_events(
sd_bus_get_events(bus)
);
uint64_t usec;
int r = sd_bus_get_timeout(bus, &usec);
if (not r) {
// if the timer is running already, it is stopped automatically
// inside uv_timer_start.
uv_timer_start(
&this->timer,
[] (uv_timer_t *handle) {
// yes, handle is not a poll_t, but
// we just care for its -> data member anyway.
on_dbus_ready((uv_poll_t *)handle, 0, 0);
},
usec / 1000, // time in milliseconds, sd_bus provides µseconds
0 // don't repeat
);
}
// always watch for disconnects:
new_events |= UV_DISCONNECT;
// activate the socket watching,
// and if active, invoke the callback function
uv_poll_start(&this->connection, new_events, &on_dbus_ready);
}
/** close the connections */
int close() {
// TODO: maybe this memoryerrors when the loop actually
// does the cleanup. we have to wait for the callback.
uv_close((uv_handle_t *) &this->timer, nullptr);
uv_poll_stop(&this->connection);
sd_bus_close(this->bus);
sd_bus_slot_unref(this->bus_slot);
sd_bus_unref(this->bus);
return 0;
}
/**
* Return the bus handle.
*/
sd_bus *get_bus() const {
return this->bus;
}
protected:
/**
* loop handle
*/
uv_loop_t *loop;
/**
* polling object for dbus events
*/
uv_poll_t connection;
/**
* dbus also wants to be called periodically
*/
uv_timer_t timer;
/**
* dbus bus handle
*/
sd_bus *bus;
/**
* dbus slot handle
*/
sd_bus_slot *bus_slot;
};

Communication between client and server is erratic

I modified thegeekinthecorner examples to be able to continuously send data.
I am using g++4.9.2.
I tried uninstalling the oficial latest OFED from here http://downloads.openfabrics.org/OFED/
OFED Distribution Software Installation Menu
1) View OFED Installation Guide
2) Install OFED Software
3) Show Installed Software
4) Configure IPoIB
5) Uninstall OFED Software
Q) Exit
Select Option [1-5]:5
Uninstalling the previous version of OFED
Running rpm -e --allmatches libibverbs libibverbs-devel libibverbs-utils libmthca libmlx4 libcxgb3 libnes libipathverbs libibcm libibumad libibumad-devel libibmad ibacm librdmacm librdmacm-utils librdmacm-devel opensm opensm-libs dapl perftest mstflint ibutils infiniband-diags qperf infinipath-psm opensm opensm-libs libipathverbs dapl libibcm libibmad libibumad libibumad-devel libibverbs libibverbs-devel libibverbs-utils libipathverbs libmthca libmlx4 librdmacm librdmacm-devel librdmacm-utils ibacm ibutils ibutils-libs libnes infinipath-psm
Failed to uninstall the previous installation
See /tmp/OFED.22320.logs/ofed_uninstall.log
[idf#node1 OFED-1.5.4-20110726-0732]$
[idf#node1 OFED-1.5.4-20110726-0732]$
If instead I just try to install it, I get this:
OFED Distribution Software Installation Menu
1) Basic (OFED modules and basic user level libraries)
2) HPC (OFED modules and libraries, MPI and diagnostic tools)
3) All packages (all of Basic, HPC)
4) Customize
Q) Exit
Select Option [1-4]:3
Please choose an implementation of MVAPICH2:
1) OFA (IB and iWARP)
2) uDAPL
Implementation [1]: 1
Enable ROMIO support [Y/n]:
Enable shared library support [Y/n]:
Enable Checkpoint-Restart support [y/N]:
Kernel 3.10.0-229.7.2.el7.x86_64 is not supported.
For the list of Supported Platforms and Operating Systems see
/mnt/gluster/Downloads/OFED-1.5.4-20110726-0732/docs/OFED_release_notes.txt
[idf#node1 OFED-1.5.4-20110726-0732]$
[idf#node2 Release]$ lspci | grep -i mel
02:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0)
[idf#node2 Release]$
[idf#node1 Release]$ ibv_devinfo
hca_id: mlx4_0
transport: InfiniBand (0)
fw_ver: 2.7.200
node_guid: 0025:90ff:ff1a:081c
sys_image_guid: 0025:90ff:ff1a:081f
vendor_id: 0x02c9
vendor_part_id: 26428
hw_ver: 0xB0
board_id: SM_2092000001000
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 1
port_lid: 2
port_lmc: 0x00
link_layer: InfiniBand
[idf#node1 Release]$ ifconfig -a
ib0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2044
inet 192.168.0.1 netmask 255.255.255.0 broadcast 192.168.0.255
inet6 fe80::225:90ff:ff1a:71 prefixlen 64 scopeid 0x20<link>
Infiniband hardware address can be incorrect! Please read BUGS section in ifconfig(8).
infiniband 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 txqueuelen 256 (InfiniBand)
RX packets 5 bytes 280 (280.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 27 overruns 0 carrier 0 collisions 0
Below is the client and server. When I run this programs, the clients will send messages, but the number of messages it sends is erratic, error messages are often
Client:
#include <iostream>
#include <thread>
#include <netdb.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <rdma/rdma_cma.h>
#define TEST_NZ(x) do { if ( (x)) die("error: " #x " failed (returned non-zero)." ); } while (0)
#define TEST_Z(x) do { if (!(x)) die("error: " #x " failed (returned zero/null)."); } while (0)
const int BUFFER_SIZE = 2048;
const int TIMEOUT_IN_MS = 500; /* ms */
struct context
{
struct ibv_context *ctx;
struct ibv_pd *pd;
struct ibv_cq *cq;
struct ibv_comp_channel *comp_channel;
pthread_t cq_poller_thread;
};
struct connection
{
struct rdma_cm_id *id;
struct ibv_qp *qp;
struct ibv_mr *recv_mr;
struct ibv_mr *send_mr;
char *recv_region;
char *send_region;
int num_completions;
};
static pthread_t msgThread;
static void die(const char *reason);
static void build_context(struct ibv_context *verbs);
static void build_qp_attr(struct ibv_qp_init_attr *qp_attr);
static void * poll_cq(void *);
static void post_receives(struct connection *conn);
static void register_memory(struct connection *conn);
static int on_addr_resolved(struct rdma_cm_id *id);
static void on_completion(struct ibv_wc *wc);
static int on_connection(void *context);
static int on_disconnect(struct rdma_cm_id *id);
static int on_event(struct rdma_cm_event *event);
static int on_route_resolved(struct rdma_cm_id *id);
static struct context *s_ctx = NULL;
#include <mutex> // std::mutex, std::unique_lock
#include <condition_variable> // std::condition_variable
std::mutex mtx;
std::condition_variable cv;
bool ok_to_send_next_message = 1;
bool message_available()
{
return 0 != ok_to_send_next_message;
}
int main(int argc, char **argv)
{
struct addrinfo *addr;
struct rdma_cm_event *event = NULL;
struct rdma_cm_id *conn= NULL;
struct rdma_event_channel *ec = NULL;
if (argc != 3)
die("usage: client <server-address> <server-port>");
TEST_NZ(getaddrinfo(argv[1], argv[2], NULL, &addr));
TEST_Z(ec = rdma_create_event_channel());
TEST_NZ(rdma_create_id(ec, &conn, NULL, RDMA_PS_TCP));
TEST_NZ(rdma_resolve_addr(conn, NULL, addr->ai_addr, TIMEOUT_IN_MS));
freeaddrinfo(addr);
while (0 == rdma_get_cm_event(ec, &event))
//while (rdma_get_cm_event(ec, &event))
{
std::cout << "rdma_get_cm_event\n";
struct rdma_cm_event event_copy;
memcpy(&event_copy, event, sizeof(*event));
rdma_ack_cm_event(event);
if (on_event(&event_copy))
break;
}
rdma_destroy_event_channel(ec);
return 0;
}
void die(const char *reason)
{
fprintf(stderr, "%s\n", reason);
exit(EXIT_FAILURE);
}
void build_context(struct ibv_context *verbs)
{
if (s_ctx)
{
if (s_ctx->ctx != verbs)
die("cannot handle events in more than one context.");
return;
}
s_ctx = (struct context *)malloc(sizeof(struct context));
s_ctx->ctx = verbs;
TEST_Z(s_ctx->pd = ibv_alloc_pd(s_ctx->ctx));
TEST_Z(s_ctx->comp_channel = ibv_create_comp_channel(s_ctx->ctx));
TEST_Z(s_ctx->cq = ibv_create_cq(s_ctx->ctx, 100, NULL, s_ctx->comp_channel, 0)); /* cqe=10 is arbitrary */
TEST_NZ(ibv_req_notify_cq(s_ctx->cq, 0));
TEST_NZ(pthread_create(&s_ctx->cq_poller_thread, NULL, poll_cq, NULL));
}
void *SendMessages(void *context)
{
static int loopcount = 0;
while(1)
{
std::unique_lock<std::mutex> lck(mtx);
cv.wait(lck, message_available);
//std::this_thread::sleep_for(std::chrono::microseconds(50));
ok_to_send_next_message = 0;
struct connection *conn = (struct connection *)context;
struct ibv_send_wr wr, *bad_wr = NULL;
struct ibv_sge sge;
std::cout << "looping send..." << loopcount << '\n' << std::flush;
memset(&wr, 0, sizeof(wr));
wr.wr_id = (uintptr_t)conn;
wr.opcode = IBV_WR_SEND;
wr.sg_list = &sge;
wr.num_sge = 1;
wr.send_flags = IBV_SEND_SIGNALED;
sge.addr = (uintptr_t)conn->send_region;
sge.length = BUFFER_SIZE;
sge.lkey = conn->send_mr->lkey;
snprintf(conn->send_region, BUFFER_SIZE, "message from active/client side with count %d", loopcount++);
TEST_NZ(ibv_post_send(conn->qp, &wr, &bad_wr));
}
}
void build_qp_attr(struct ibv_qp_init_attr *qp_attr)
{
std::cout << "build_qp_attr\n";
memset(qp_attr, 0, sizeof(*qp_attr));
qp_attr->send_cq = s_ctx->cq;
qp_attr->recv_cq = s_ctx->cq;
qp_attr->qp_type = IBV_QPT_RC;
qp_attr->cap.max_send_wr = 100;
qp_attr->cap.max_recv_wr = 100;
qp_attr->cap.max_send_sge = 1;
qp_attr->cap.max_recv_sge = 1;
}
void * poll_cq(void *ctx)
{
struct ibv_cq *cq;
struct ibv_wc wc;
while (1)
{
TEST_NZ(ibv_get_cq_event(s_ctx->comp_channel, &cq, &ctx));
ibv_ack_cq_events(cq, 1);
TEST_NZ(ibv_req_notify_cq(cq, 0));
int ne;
struct ibv_wc wc;
do
{
std::cout << "polling\n";
ne = ibv_poll_cq(cq, 1, &wc);
}
while(ne == 0);
on_completion(&wc);
//if (wc.opcode == IBV_WC_SEND)
if (wc.status == IBV_WC_SUCCESS)
{
{
ok_to_send_next_message = 1;
//while (message_available()) std::this_thread::yield();
//std::cout << "past yield\n";
std::unique_lock<std::mutex> lck(mtx);
cv.notify_one();
}
}
}
return NULL;
}
void post_receives(struct connection *conn)
{
std::cout << "post_receives\n";
struct ibv_recv_wr wr, *bad_wr = NULL;
struct ibv_sge sge;
wr.wr_id = (uintptr_t)conn;
wr.next = NULL;
wr.sg_list = &sge;
wr.num_sge = 1;
sge.addr = (uintptr_t)conn->recv_region;
sge.length = BUFFER_SIZE;
sge.lkey = conn->recv_mr->lkey;
TEST_NZ(ibv_post_recv(conn->qp, &wr, &bad_wr));
}
void register_memory(struct connection *conn)
{
std::cout << "register_memory\n";
conn->send_region = (char *)malloc(BUFFER_SIZE);
conn->recv_region = (char *)malloc(BUFFER_SIZE);
TEST_Z(conn->send_mr = ibv_reg_mr(
s_ctx->pd,
conn->send_region,
BUFFER_SIZE,
IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_WRITE));
TEST_Z(conn->recv_mr = ibv_reg_mr(
s_ctx->pd,
conn->recv_region,
BUFFER_SIZE,
IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_WRITE));
}
int on_addr_resolved(struct rdma_cm_id *id)
{
std::cout << "on_addr_resolved\n";
struct ibv_qp_init_attr qp_attr;
struct connection *conn;
build_context(id->verbs);
build_qp_attr(&qp_attr);
TEST_NZ(rdma_create_qp(id, s_ctx->pd, &qp_attr));
id->context = conn = (struct connection *)malloc(sizeof(struct connection));
conn->id = id;
conn->qp = id->qp;
conn->num_completions = 0;
register_memory(conn);
post_receives(conn);
TEST_NZ(rdma_resolve_route(id, TIMEOUT_IN_MS));
return 0;
}
void on_completion(struct ibv_wc *wc)
{
std::cout << "on_completion\n";
struct connection *conn = (struct connection *)(uintptr_t)wc->wr_id;
if (wc->status != IBV_WC_SUCCESS)
{
//die("\ton_completion: status is not IBV_WC_SUCCESS.");
printf("\ton_completion: status is not IBV_WC_SUCCESS.");
printf("\t it is %d ", wc->status);
}
printf("\n");
if (wc->opcode & IBV_WC_RECV)
printf("\treceived message: %s\n", conn->recv_region);
else if (wc->opcode == IBV_WC_SEND)
printf("\tsend completed successfully.\n");
else
die("\ton_completion: completion isn't a send or a receive.");
if (5 == ++conn->num_completions)
rdma_disconnect(conn->id);
}
int on_connection(void *context)
{
std::cout << "on_connection\n";
TEST_NZ(pthread_create(&msgThread, NULL, SendMessages, context));
return 0;
}
int on_disconnect(struct rdma_cm_id *id)
{
struct connection *conn = (struct connection *)id->context;
printf("disconnected.\n");
rdma_destroy_qp(id);
ibv_dereg_mr(conn->send_mr);
ibv_dereg_mr(conn->recv_mr);
free(conn->send_region);
free(conn->recv_region);
free(conn);
rdma_destroy_id(id);
return 1; /* exit event loop */
}
int on_route_resolved(struct rdma_cm_id *id)
{
struct rdma_conn_param cm_params;
printf("route resolved.\n");
memset(&cm_params, 0, sizeof(cm_params));
TEST_NZ(rdma_connect(id, &cm_params));
return 0;
}
int on_event(struct rdma_cm_event *event)
{
int r = 0;
if (event->event == RDMA_CM_EVENT_ADDR_RESOLVED)
r = on_addr_resolved(event->id);
else if (event->event == RDMA_CM_EVENT_ROUTE_RESOLVED)
r = on_route_resolved(event->id);
else if (event->event == RDMA_CM_EVENT_ESTABLISHED)
r = on_connection(event->id->context);
else if (event->event == RDMA_CM_EVENT_DISCONNECTED)
r = on_disconnect(event->id);
else
die("on_event: unknown event.");
return r;
}
Server:
#include <iostream>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <inttypes.h>
#include <rdma/rdma_cma.h>
#define TEST_NZ(x) do { if ( (x)) die("error: " #x " failed (returned non-zero)." ); } while (0)
#define TEST_Z(x) do { if (!(x)) die("error: " #x " failed (returned zero/null)."); } while (0)
const int BUFFER_SIZE = 2048;
struct context
{
struct ibv_context *ctx;
struct ibv_pd *pd;
struct ibv_cq *cq;
struct ibv_comp_channel *comp_channel;
pthread_t cq_poller_thread;
};
struct connection
{
struct ibv_qp *qp;
struct ibv_mr *recv_mr;
struct ibv_mr *send_mr;
char *recv_region;
char *send_region;
};
static void die(const char *reason);
static void build_context(struct ibv_context *verbs);
static void build_qp_attr(struct ibv_qp_init_attr *qp_attr);
static void * poll_cq(void *);
static void post_receives(struct connection *conn);
static void register_memory(struct connection *conn);
static void on_completion(struct ibv_wc *wc);
static int on_connect_request(struct rdma_cm_id *id);
static int on_connection(void *context);
static int on_disconnect(struct rdma_cm_id *id);
static int on_event(struct rdma_cm_event *event);
static struct context *s_ctx = NULL;
int main(int argc, char **argv)
{
struct sockaddr_in6 addr;
struct rdma_cm_event *event = NULL;
struct rdma_cm_id *listener = NULL;
struct rdma_event_channel *ec = NULL;
uint16_t port = 0;
memset(&addr, 0, sizeof(addr));
addr.sin6_family = AF_INET6;
TEST_Z(ec = rdma_create_event_channel());
TEST_NZ(rdma_create_id(ec, &listener, NULL, RDMA_PS_TCP));
TEST_NZ(rdma_bind_addr(listener, (struct sockaddr *)&addr));
TEST_NZ(rdma_listen(listener, 100)); /* backlog=10 is arbitrary */
//printf("[ %"PRIu32" ]\n", *addr.sin6_addr.s6_addr32);
port = ntohs(rdma_get_src_port(listener));
printf("listening on port %d.\n", port);
while (rdma_get_cm_event(ec, &event) == 0)
{
struct rdma_cm_event event_copy;
memcpy(&event_copy, event, sizeof(*event));
rdma_ack_cm_event(event);
if (on_event(&event_copy))
break;
}
rdma_destroy_id(listener);
rdma_destroy_event_channel(ec);
return 0;
}
void die(const char *reason)
{
fprintf(stderr, "%s\n", reason);
exit(EXIT_FAILURE);
}
void build_context(struct ibv_context *verbs)
{
if (s_ctx)
{
if (s_ctx->ctx != verbs)
die("cannot handle events in more than one context.");
return;
}
s_ctx = (struct context *)malloc(sizeof(struct context));
s_ctx->ctx = verbs;
TEST_Z(s_ctx->pd = ibv_alloc_pd(s_ctx->ctx));
TEST_Z(s_ctx->comp_channel = ibv_create_comp_channel(s_ctx->ctx));
TEST_Z(s_ctx->cq = ibv_create_cq(s_ctx->ctx, 100, NULL, s_ctx->comp_channel, 0)); /* cqe=10 is arbitrary */
TEST_NZ(ibv_req_notify_cq(s_ctx->cq, 0));
TEST_NZ(pthread_create(&s_ctx->cq_poller_thread, NULL, poll_cq, NULL));
}
void build_qp_attr(struct ibv_qp_init_attr *qp_attr)
{
memset(qp_attr, 0, sizeof(*qp_attr));
qp_attr->send_cq = s_ctx->cq;
qp_attr->recv_cq = s_ctx->cq;
qp_attr->qp_type = IBV_QPT_RC;
qp_attr->cap.max_send_wr = 100;
qp_attr->cap.max_recv_wr = 100;
qp_attr->cap.max_send_sge = 1;
qp_attr->cap.max_recv_sge = 1;
}
void * poll_cq(void *ctx)
{
struct ibv_cq *cq;
struct ibv_wc wc;
while (1)
{
TEST_NZ(ibv_get_cq_event(s_ctx->comp_channel, &cq, &ctx));
ibv_ack_cq_events(cq, 1);
TEST_NZ(ibv_req_notify_cq(cq, 0));
while (ibv_poll_cq(cq, 1, &wc))
{
std::cout << "polling\n";
on_completion(&wc);
}
}
return NULL;
}
void post_receives(struct connection *conn)
{
std::cout << "post_receives\n";
struct ibv_recv_wr wr, *bad_wr = NULL;
struct ibv_sge sge;
wr.wr_id = (uintptr_t)conn;
wr.next = NULL;
wr.sg_list = &sge;
wr.num_sge = 1;
sge.addr = (uintptr_t)conn->recv_region;
sge.length = BUFFER_SIZE;
sge.lkey = conn->recv_mr->lkey;
TEST_NZ(ibv_post_recv(conn->qp, &wr, &bad_wr));
}
void register_memory(struct connection *conn)
{
conn->send_region = (char *)malloc(BUFFER_SIZE);
conn->recv_region = (char *)malloc(BUFFER_SIZE);
TEST_Z(conn->send_mr = ibv_reg_mr(
s_ctx->pd,
conn->send_region,
BUFFER_SIZE,
IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_WRITE));
TEST_Z(conn->recv_mr = ibv_reg_mr(
s_ctx->pd,
conn->recv_region,
BUFFER_SIZE,
IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_WRITE));
}
void on_completion(struct ibv_wc *wc)
{
if (wc->status != IBV_WC_SUCCESS)
die("on_completion: status is not IBV_WC_SUCCESS.");
if (wc->opcode & IBV_WC_RECV)
{
struct connection *conn = (struct connection *)(uintptr_t)wc->wr_id;
post_receives(conn);
printf("received message: %s\n", conn->recv_region);
}
else if (wc->opcode == IBV_WC_SEND)
{
printf("send completed successfully.\n");
}
}
int on_connect_request(struct rdma_cm_id *id)
{
struct ibv_qp_init_attr qp_attr;
struct rdma_conn_param cm_params;
struct connection *conn;
printf("received connection request.\n");
build_context(id->verbs);
build_qp_attr(&qp_attr);
TEST_NZ(rdma_create_qp(id, s_ctx->pd, &qp_attr));
id->context = conn = (struct connection *)malloc(sizeof(struct connection));
conn->qp = id->qp;
register_memory(conn);
post_receives(conn);
memset(&cm_params, 0, sizeof(cm_params));
TEST_NZ(rdma_accept(id, &cm_params));
return 0;
}
int on_connection(void *context)
{
struct connection *conn = (struct connection *)context;
struct ibv_send_wr wr, *bad_wr = NULL;
struct ibv_sge sge;
snprintf(conn->send_region, BUFFER_SIZE, "message from passive/server side with pid %d", getpid());
printf("connected. posting send...\n");
memset(&wr, 0, sizeof(wr));
wr.opcode = IBV_WR_SEND;
wr.sg_list = &sge;
wr.num_sge = 1;
wr.send_flags = IBV_SEND_SIGNALED;
sge.addr = (uintptr_t)conn->send_region;
sge.length = BUFFER_SIZE;
sge.lkey = conn->send_mr->lkey;
TEST_NZ(ibv_post_send(conn->qp, &wr, &bad_wr));
return 0;
}
int on_disconnect(struct rdma_cm_id *id)
{
struct connection *conn = (struct connection *)id->context;
printf("peer disconnected.\n");
rdma_destroy_qp(id);
ibv_dereg_mr(conn->send_mr);
ibv_dereg_mr(conn->recv_mr);
free(conn->send_region);
free(conn->recv_region);
free(conn);
rdma_destroy_id(id);
return 0;
}
int on_event(struct rdma_cm_event *event)
{
std::cout << "on_event\n";
int r = 0;
if (event->event == RDMA_CM_EVENT_CONNECT_REQUEST)
r = on_connect_request(event->id);
else if (event->event == RDMA_CM_EVENT_ESTABLISHED)
r = on_connection(event->id->context);
else if (event->event == RDMA_CM_EVENT_DISCONNECTED)
r = on_disconnect(event->id);
else
die("on_event: unknown event.");
return r;
}
Here are a couple of runs. Totally random the number of message sent:
[idf#node1 Release]$ ./TGKITCClient 192.168.0.1 47819
rdma_get_cm_event
on_addr_resolved
build_qp_attr
register_memory
post_receives
rdma_get_cm_event
route resolved.
rdma_get_cm_event
on_connection
looping send...0
polling
on_completion
received message: message from passive/server side with pid 4188
polling
on_completion
send completed successfully.
looping send...1
polling
on_completion
send completed successfully.
^C
[idf#node1 Release]$
And then
[idf#node1 Release]$ ./TGKITCClient 192.168.0.1 55148
rdma_get_cm_event
on_addr_resolved
build_qp_attr
register_memory
post_receives
rdma_get_cm_event
route resolved.
rdma_get_cm_event
on_connection
looping send...0
polling
on_completion
received message: message from passive/server side with pid 4279
polling
on_completion
send completed successfully.
looping send...1
polling
on_completion
send completed successfully.
looping send...2
polling
on_completion
send completed successfully.
looping send...3
polling
on_completion
send completed successfully.
looping send...4
polling
on_completion
send completed successfully.
looping send...5
polling
on_completion
send completed successfully.
looping send...6
polling
on_completion
send completed successfully.
looping send...7
polling
on_completion
send completed successfully.
looping send...8
rdma_get_cm_event
disconnected.
polling
on_completion
send completed successfully.
on_completion: status is not IBV_WC_SUCCESS. it is 5 [idf#node1 Release]$
Here is the server side:
on_event
peer disconnected.
on_event
received connection request.
post_receives
on_event
connected. posting send...
polling
send completed successfully.
polling
post_receives
received message: message from active/client side with count 0
polling
post_receives
received message: message from active/client side with count 1
polling
post_receives
received message: message from active/client side with count 2
polling
post_receives
received message: message from active/client side with count 3
polling
post_receives
received message: message from active/client side with count 4
polling
post_receives
received message: message from active/client side with count 5
polling
post_receives
received message: message from active/client side with count 6
polling
post_receives
received message: message from active/client side with count 7
on_event
peer disconnected.
Make sure that the most recent drivers and firmware on the cards are installed. Beyond that, using the RDMA packages included with most OS distributions when trying to run IB is a dangerous game to play.
It is strongly recommended that for applications like these the Open Fabrics Enterprise Distribution should be used to provide openib, opensm and a variety of other useful infiniband related packages for analysis diagnostics and tuning of a network. The official OFED packages can be found on the OpenFabrics website.
Based on the question it looks like IPoIB is being used but the specific configuration is not mentioned. IPoIB is not necessarily the best way to take advantages of the hardware resources available in the IB cards.
In addition to those considerations making sure that the subnetmanager is setup and configured correctly. Some switches have built-in subnet managers that can be access and configured through a management interface, in other cases it might make more sense to run and configure the subnet manager on one of the nodes that you are using. OpenSM is a common subnet manager that is included with OFED distributions and there are many online guides available for setting up and configuring a subnet manager based on the type of network being setup.
OFED also includes a variety of IB testing and profiling tools. ibdiagnet is a useful tool for debugging IB network issues. And there are many guides available online that show different ways to make use of the tool as well as the other tools included in OFED.
Depending on the type of IB switch used there may additionally be some network management and diagnostic tools that would allow for further analysis of the network. The configuration of IB hardware and the low-level software that manages it is sometimes more critical to overall performance than the actual code being run. But with that being said recompiling and linking to relevant libraries from the correct version of OFED might be advisable if significant software of hardware configuration changes are made.

how to wakeup select() within timeout from another thread

According to the "man select" information:
"On success, select() and pselect() return the number of file descrip‐
tors contained in the three returned descriptor sets which may be zero
if the timeout expires before anything interesting happens. On error,
-1 is returned, and errno is set appropriately; the sets and timeout become
undefined, so do not rely on their contents after an error."
Select will wakup because of:
1)read/write availability
2)select error
3)descriptoris closed.
However, how can we wake up the select() from another thread if there is no data available and the select is still within timeout?
[update]
Pseudo Code
// Thread blocks on Select
void *SocketReadThread(void *param){
...
while(!(ReadThread*)param->ExitThread()) {
struct timeval timeout;
timeout.tv_sec = 60; //one minute
timeout.tv_usec = 0;
fd_set rds;
FD_ZERO(&rds);
FD_SET(sockfd, &rds)'
//actually, the first parameter of select() is
//ignored on windows, though on linux this parameter
//should be (maximum socket value + 1)
int ret = select(sockfd + 1, &rds, NULL, NULL, &timeout );
//handle the result
//might break from here
}
return NULL;
}
//main Thread
int main(){
//create the SocketReadThread
ReaderThread* rthread = new ReaderThread;
pthread_create(&pthreadid, NULL, SocketReaderThread,
NULL, (void*)rthread);
// do lots of things here
............................
//now main thread wants to exit SocketReaderThread
//it sets the internal state of ReadThread as true
rthread->SetExitFlag(true);
//but how to wake up select ??????????????????
//if SocketReaderThread currently blocks on select
}
[UPDATE]
1) #trojanfoe provides a method to achieve this, his method writes socket data (maybe dirty data or exit message data) to wakeup select. I am going to have a test and update the result there.
2) Another thing to mention, closing a socket doesn't guarantee to wake up select function call, please see this post.
[UPDATE2]
After doing many tests, here are some facts about waking up select:
1) If the socket watched by select is closed by another application, then select() calling
will wakeup immediately. Hereafter, reading from or writing to the socket will get return value of 0 with an errno = 0
2) If the socket watched by select is closed by another thread of the same application,
then select() won't wake up until timeout if there is no data to read or write. After select timeouts, making read/write operation results in an error with errno = EBADF
(because the socket has been closed by another thread during timeout period)
I use an event object based on pipe():
IoEvent.h:
#pragma once
class IoEvent {
protected:
int m_pipe[2];
bool m_ownsFDs;
public:
IoEvent(); // Creates a user event
IoEvent(int fd); // Create a file event
IoEvent(const IoEvent &other);
virtual ~IoEvent();
/**
* Set the event to signalled state.
*/
void set();
/**
* Reset the event from signalled state.
*/
void reset();
inline int fd() const {
return m_pipe[0];
}
};
IoEvent.cpp:
#include "IoEvent.h"
#include <string.h>
#include <unistd.h>
#include <errno.h>
#include <fcntl.h>
#include <poll.h>
using namespace std;
IoEvent::IoEvent() :
m_ownsFDs(true) {
if (pipe(m_pipe) < 0)
throw MyException("Failed to create pipe: %s (%d)", strerror(errno), errno);
if (fcntl(m_pipe[0], F_SETFL, O_NONBLOCK) < 0)
throw MyException("Failed to set pipe non-blocking mode: %s (%d)", strerror(errno), errno);
}
IoEvent::IoEvent(int fd) :
m_ownsFDs(false) {
m_pipe[0] = fd;
m_pipe[1] = -1;
}
IoEvent::IoEvent(const IoEvent &other) {
m_pipe[0] = other.m_pipe[0];
m_pipe[1] = other.m_pipe[1];
m_ownsFDs = false;
}
IoEvent::~IoEvent() {
if (m_pipe[0] >= 0) {
if (m_ownsFDs)
close(m_pipe[0]);
m_pipe[0] = -1;
}
if (m_pipe[1] >= 0) {
if (m_ownsFDs)
close(m_pipe[1]);
m_pipe[1] = -1;
}
}
void IoEvent::set() {
if (m_ownsFDs)
write(m_pipe[1], "x", 1);
}
void IoEvent::reset() {
if (m_ownsFDs) {
uint8_t buf;
while (read(m_pipe[0], &buf, 1) == 1)
;
}
}
You could ditch the m_ownsFDs member; I'm not even sure I use that any more.

Socket class Error

ListenSocket.h
// class does not contain WSASTARTUP () and WSACLEANUP ()
#ifndef LISTENTHREAD_H
#define LISTENTHREAD_H
#include "includes.h"
#include "LOGMSGs.h"
// 1, CListenSocket: class is used to create the listener thread local variable.
// This class can be reused. When you call Close () is closed, re-calling Open () the new listening port. But the system did not use the feature.
class CListenSocket
{
public:
// main method:
// BIND each object only to a port.
CListenSocket(u_short nPort, int nSndSize = 0);
// to release SOCKET
~CListenSocket(){};
// Create server listening SOCKET, specific options see the code. Fails to return false.
bool Open(); // call can be repeated
// error return INVALID_SOCKET
SOCKET Accept(u_long & nClientIP);
// repeated calls. Usually not, can be used to take the initiative to close the SOCKET.
// close the re-call after Open () re-use the object.
void Close(); // call can be repeated
bool IsOpen() { return m_bState; }
bool Rebuild();
public:
SOCKET Socket() { return m_sockListen; }
protected:
// main member variables:
const u_short m_nPort;
const int m_nSndBuf;
SOCKET m_sockListen;
// network status is normal sign.
// When the value is false that the object is not available. May not have Open (), may also be a network error.
bool m_bState;
time_t m_tCloseTime; // SOCKET last closed the time delay for the automatic re-SOCKET
};
#endif // LISTENTHREAD_H
ListenSocket.cpp
#include "ListenSocket.h"
long s_nSocketCount = 0;
int REBUILDLISTENDELAYSEC;
CListenSocket::CListenSocket(u_short nPort, int nSndBuf /*= 0*/) // 0: Default
: m_nPort(nPort), m_nSndBuf(nSndBuf)
{
m_sockListen = INVALID_SOCKET;
m_bState = false;
// m_nPort = nPort;
m_tCloseTime = 0;
}
// Error returned INVALID_SOCKET
SOCKET CListenSocket::Accept(u_long & nClientIP)
{
/*
// Reconstruction SOCKET
if(!m_bState)
{
if(clock() < m_tCloseTime + REBUILDLISTENDELAYSEC*CLOCKS_PER_SEC)
return INVALID_SOCKET;
else
{
LOGMSG("Anti-crash system start listening SOCKET [%d] re under construction...", m_nPort);
if(Open())
{
LOGMSG("... listen SOCKET reconstruction success.");
PrintText("Listen SOCKET [%d] failed to rebuild SOCKET success. Server continues to run in the ...", m_nPort);
}
else
{
Error("... listen SOCKET reconstruction has failed. Server will not accept new connections");
PrintText("Listen SOCKET [%d] error, [%d] seconds after the re-SOCKET. Server continues to run in the ...", m_nPort, REBUILDLISTENDELAYSEC); // nDelaySec);
}
m_tCloseTime = clock();
}
}
//*/
if(!m_bState)
{
Error("ACCEPT inner exception a1");
return INVALID_SOCKET;
}
// ACCEPT
struct sockaddr_in addr;
memset(&addr, 0, sizeof(addr));
int len = sizeof(addr);
SOCKET newsock = accept(m_sockListen, (sockaddr*)&addr, (int*)&len); // receive to the other side of the map, you can use
#ifdef PROFILE_X
// Analysis Accept speed (cycle speed)
const int nTimes2 = ACCEPTPROFILESEC; // Statistics once every 30 seconds the speed ACCEPT
static clock_t tNextTime2 = clock() + nTimes2 * CLOCKS_PER_SEC; //? Only one monitor thread, no sharing violation
static long nCount2 = 0; //? Only one monitor thread, no sharing violation
if(clock() >= tNextTime2)
{
PrintText("Each [%d] seconds to execute a [%d] times Accept ()", nTimes2, InterlockedExchange(&nCount2, 0));
tNextTime2 = clock() + nTimes2 * CLOCKS_PER_SEC;
}
else
{
InterlockedIncrement(&nCount2);
}
#endif // PROFILE
if(newsock == INVALID_SOCKET)
{
// Network Error
int err = WSAGetLastError();
if(err != WSAEWOULDBLOCK)
{
PrintText("Listen SOCKET %d failed, %s seconds after the re-SOCKET.", m_nPort, REBUILDLISTENDELAYSEC);
Error("Listen SOCKET [%d] failed [%d], [%s] seconds after the re-SOCKET.", m_nPort, err, REBUILDLISTENDELAYSEC);
Close();
}
else
Error("ACCEPT inner exception a2");
return INVALID_SOCKET;
}
else
{
nClientIP = addr.sin_addr.S_un.S_addr;
InterlockedIncrement(&s_nSocketCount);
}
// Check whether the SOCKET closed
fd_set readmask;
FD_ZERO(&readmask);
FD_SET(newsock, &readmask);
struct timeval timeout = {0, 0};
/*
char nTemp;
if(select(FD_SETSIZE, &readmask, (fd_set *) 0, (fd_set *) 0, &timeout)
&& recv(newsock, &nTemp, 1, MSG_PEEK) == 0)
{
#ifdef ALPHA_X
LOGMSG("ACCEPT a new SOCKET is invalid .");
#endif
closesocket(newsock);
InterlockedDecrement(&s_nSocketCount);
return INVALID_SOCKET;
}
//else*/
//*
fd_set exceptmask;
FD_ZERO(&exceptmask);
FD_SET(newsock, &exceptmask);
int ret = select(FD_SETSIZE, &readmask, (fd_set *) 0, (fd_set *) &exceptmask, &timeout);
if(ret < 0)
{
Error("ACCEPT a new SOCKET is invalid . can't read"); // Not trigger
closesocket(newsock);
InterlockedDecrement(&s_nSocketCount);
return INVALID_SOCKET;
}
else if(ret > 0)
{
if(FD_ISSET(newsock, &exceptmask))
{
LOGMSG("ACCEPT a new SOCKET is invalid.except"); // Not trigger
closesocket(newsock);
InterlockedDecrement(&s_nSocketCount);
return INVALID_SOCKET;
}
else if(FD_ISSET(newsock, &readmask))
{
char nTemp;
if(recv(newsock, &nTemp, 1, MSG_PEEK) == 0)
{
#ifdef ALPHA_X
LOGMSG("ACCEPT a new SOCKET is invalid. recv==0"); // Not trigger
#endif
closesocket(newsock);
InterlockedDecrement(&s_nSocketCount);
return INVALID_SOCKET;
}
}
}
//*/
#ifdef PROFILE_X
// analysis Accept speed (received valid SOCKET)
const int nTimes = ACCEPTPROFILESEC; // Statistics once every 10 seconds the speed ACCEPT
static clock_t tNextTime = clock() + nTimes * CLOCKS_PER_SEC; //? Only one monitor thread, no sharing violation
static long nCount = 0; //? Only one monitor thread, no sharing violation
if(clock() >= tNextTime)
{
LOGPROFILE("Port [%d] for every [%d] seconds, the successful implementation of the [%d] times Accept()",
m_nPort, nTimes, InterlockedExchange(&nCount, 0));
tNextTime = clock() + nTimes * CLOCKS_PER_SEC;
}
else
{
InterlockedIncrement(&nCount);
}
#endif // PROFILE
return newsock;
}
Main.cpp
#include "includes.h"
#include "IniFile.h"
#include "LOGMSGs.h"
#include "ListenSocket.h"
CListenSocket Sock(9985);
int main()
{
Sock.Open();
if(!Sock.Open())
{
Sock.Rebuild();
}
if(Sock.IsOpen())
PrintText("okey");
Sock.Socket();
u_long ip;
Sock.Accept(ip);
}
but i always got this error : ACCEPT inner exception a2 while it should work anyclue why?
CListenSocket Sock(9985);
int main()
{
Sock.Open();
if(!Sock.Open())
/* I think you meant 'IsOpen()' */
{
Sock.Rebuild();
}
if(Sock.IsOpen())
PrintText("okey");
Sock.Socket();
u_long ip;
Sock.Accept(ip);
}
Incidentally, this code sure reads funny. It feels like a generic toolkit programmed without a specific goal in mind. Maybe I'm missing it, but I have to think you'd have better results if you just wrote the network code that actually needed, and then abstract out the common bits into some helper routines later. There's no point in trying too hard to make the be-all and end-all network helper library, but there is a huge point in making tools that collapse common cases.
Feel free to ignore that last paragraph if you know what you're doing :) but if you're just starting out, I'd like to suggest writing a few smaller clients and servers, and then try writing your abstraction layer.