I am newcomer in area of network and internet,therefore want to apologize for may be stupid question. I do not understand whether there are other ways to send data from client socket to server's axcept putting data into a stream using method QIODevice::write(QByteArray& ). If that is the only way how server should recognize what exactly data has been sent to it ? For example, we may have QString message as a usual input data, but also sometimes QString as the name of further receiver of future data. It is possible to describe all variants but the slot connected to readyRead() signal seems to be of enormous
size at this case.
Eventually, is there way to direct data to some exact server's functions ?
Qt Solutions has a library to make Qt servers easily:
Qt Solutions
And Json format it is a beautiful way to communicate
You need to define comman data type both side( client and server ). before you sent data packet you should write size of data packet to first four byte of data packet. at the server side check size of data receiving from client with first four bytes. and deserialize data which you how did you serialize at client side. I used this method a long time and there is any problem occured to today. i will give you sample code for you.
Client Side:
QBuffer buffer;
buffer.open(QIODevice::ReadWrite);
QDataStream in(&buffer);
in.setVersion(QDataStream::Qt_5_2);
in << int(0); // for packet size
in << int(3); // int may be this your data type or command
in << double(4); // double data
in << QString("asdsdffdggfh"); //
in << QVariant("");
in << .... // any data you can serialize which QDatastream accept
in.device()->seek(0); // seek packet fisrt byte
in << buffer.data().size(); // and write packet size
array = buffer.data();
this->socket->write(arr);
this->socket->waitForBytesWritten();
Server Side:
QDatastream in(socket);
//define this out of this scope and globally
int expectedByte = -1;
if( expectedByte < socket->bytesAvailable() && expectedByte == -1 )
{
in >> expectedByte;
}
if(expectedByte - socket->bytesAvailable()- (int)sizeof(int) != 0){
return;
}
// if code here, your packet received completely
int commandOrDataType;
in >> commandOrDataType;
double anyDoubleValue;
in >> anyDoubleValue;
QString anyStringValue;
in >> anyStringValue;
QVariant anyVariant;
in >> anyVariant;
// and whatever ...
// do something with above data
//you must set expectedByte = -1;
// if your proccessing doing any thing at this time there is no any data will be received while expectedByte != -1, but may be socket buffer will be filling. you should comfirm at the begining of this function
expectedByte = -1;
Hope this helpfully! :)
Related
I am trying to write reverse proxy with nonblocking socket and epoll. That seems ok at first, but when I tried to open a big jpg file, I got stuck.
When I try to write into client sometimes It may not writable and how can I handle proper way.
Additional Notes:
this->getFd() = ProxyFd
this->clientHandler->getFd = clientFd
I am using EPOLLET flag both proxy and client
if( (flag & EPOLLIN) ){
char buffer[1025] = {'\0'};
int readSize;
while( (readSize = read(this->getFd(),buffer,1024)) > 0){
this->headerParse(buffer);
this->readSize += readSize;
int check = 0;
do{
check = write(this->clientHandler->getFd(),buffer,readSize);
}while(check < 0);
}
if(this->headerEnd == 1 && this->readSize >= this->headerLenght ){
close(this->clientHandler->getFd());
close(this->getFd());
delete this->clientHandler;
delete this;
}
}
Thanks for taking time to read.
Assuming your headerParse() method doesn't change buffer in a size-extending way (you'd need to update readsize, at least, not to mention the buffer full scenario), it seems like your write() path is broken.
if the socket you're writing to is also in nonblocking mode, it's perfectly legal for write() to return -1 (and set errno to EGAIN or EWOULDBLOCK or whatever your platform has) before you wrote all data.
In that case, you must store the remaining data (the remainder of buffer minus what was written if one or more calls to write() succeeded), program epoll to notify the clientHandler->getFd() descriptor for writeability, if not already, and when you get subsequent "write ready" event, you write the data you stored. On this case, the write() can again be unable to flush all your data, so you must cycle until all data is sent.
I got a client and a server application which will send each other data by using the Asio (Standalone) library. Both applications consists of two (logical) parts:
A high level part: dealing with complex objects e.g. users, permissions,...
A low level part: sending data over network between client and server
Let's assume the complex objects are already serialized by using Protocoll Buffers and the low level part of the application receives the data as std::string from the high level part. I would like to use this function from Protocoll Buffers for this job:
bool SerializeToString(string* output) const;: serializes the message
and stores the bytes in the given string. Note that the bytes are
binary, not text; we only use the string class as a convenient
container.
And say I transfer this data with async_write on the client side:
size_t dataLength = strlen(data);
//writes a certain number of bytes of data to a stream.
asio::async_write(mSocket,
asio::buffer(data, dataLength),
std::bind(&Client::writeCallback, this,
std::placeholders::_1,
std::placeholders::_2));
How can I read this data on the server side? I don't know how much data I will have to read. Therefore this will not work (length is unknown):
asio::async_read(mSocket,
asio::buffer(mResponse, length),
std::bind(&Server::readCallback, this,
std::placeholders::_1,
std::placeholders::_2));
What is the best way to solve this problem? I could think of two solutions:
Append a 'special' character at the end of data and read until I reach this 'end of data signal'. The problem is, what if this character appears in data somehow? I don't know how Protocoll Buffers serializes my data.
Send a binary string with size_of_data + data instead of data. But I don't know how to serialize the size in an platform independent way, add it to the binary data and extract it again.
Edit: Maybe I could use this:
uint64_t length = strlen(data);
uint64_t nwlength = htonl(length);
uint8_t len[8];
len[0] = nwlength >> 56;
len[1] = nwlength >> 48;
len[2] = nwlength >> 40;
len[3] = nwlength >> 32;
len[4] = nwlength >> 24;
len[5] = nwlength >> 16;
len[6] = nwlength >> 8;
len[7] = nwlength >> 0;
std::string test(len);
mRequest = data;
mRequest.insert(0, test);
and send mRequest to the server? Any traps or caveats with this code?
How could I read the length on server side and the content afterwards?
Maybe like this:
void Server::readHeader(){
asio::async_read(mSocket,
asio::buffer(header, HEADER_LENGTH),
std::bind(&Server::readHeaderCallback, this,
std::placeholders::_1,
std::placeholders::_2),
asio::transfer_exactly(HEADER_LENGTH));
}
void Server::readHeaderCallback(const asio::error_code& error,
size_t bytes_transferred){
if(!error && decodeHeader(header, mResponseLength)){
//reading header finished, now read the content
readContent();
}
else{
if(error) std::cout << "Read failed: " << error.message() << "\n";
else std::cout << "decodeHeader failed \n";
}
}
void Server::readContent(){
asio::async_read(mSocket,
asio::buffer(mResponse, mResponseLength),
std::bind(&Server::readContentCallback, this,
std::placeholders::_1,
std::placeholders::_2),
asio::transfer_exactly(mResponseLength));
}
void Server::readContentCallback(const asio::error_code& error,
size_t bytes_transferred){
if (!error){
//handle content
}
else{
//#todo remove this cout
std::cout << "Read failed: " << error.message() << "\n";
}
}
Please note that I try to use transfer_exactly. Will this work?
When sending variable length messages over a stream-based protocol, there are generally three solutions to indicate message boundaries:
Use a delimiter to specify message boundaries. The async_read_until() operations provide a convenient way to read variable length delimited messages. When using a delimiter, one needs to consider the potential of a delimiter collision, where the delimiter appears within the contents of a message, but does not indicate a boundary. There are various techniques to handle delimiter collisions, such as escape characters or escape sequences.
Use a fixed-length header with a variable-length body protocol. The header will provide meta-information about the message, such as the length of the body. The official Asio chat example demonstrates one way to handle fixed-length header and variable-length body protocols.
If binary data is being sent, then one will need to consider handling byte-ordering. The hton() and ntoh() family of functions can help with byte-ordering. For example, consider a protocol that defines the field as two bytes in network-byte-order (big-endian) and a client reads the field as a uint16_t. If the value 10 is sent, and a little-endian machine reads it without converting from network-order to local-order, then the client will read the value as 2560. The Asio chat example avoids handling endianness by encoding the body length to string instead of a binary form.
Use the connection's end-of-file to indicate the end of a message. While this makes sending and receiving messages easy, it limits the sender to only one message per connection. To send an additional message, one would need to established another connection.
A few observations about the code:
The Protocol Buffers' SerializeToString() function serializes a message to a binary form. One should avoid using text based functions, such as strlen(), on the serialized string. For instance, strlen() may incorrectly determine the length, as it will treat the first byte with a value of 0 as the terminating null byte, even if that byte is part of the encoded value.
When providing an explicitly sized buffer to an operation via asio::buffer(buffer, n), the default completion condition of transfer_all will function the same as transfer_exactly(n). As such, the duplicate use of variables can be removed:
asio::async_read(mSocket,
asio::buffer(header, HEADER_LENGTH),
std::bind(&Server::readHeaderCallback, this,
std::placeholders::_1,
std::placeholders::_2));
The htonl() overloads support uint16_t and uint_32t, not uint64_t.
Asio supports scatter/gather operations, allowing a receive operation to scatter-read into multiple buffers, and transmit operations can gather-write from multiple buffers. As such, one does not necessarily need to have both the fixed-length header and message-body contained with a single buffer.
std::string body_buffer;
body.SerializeToString(&body_buffer);
std::string header_buffer = encode_header(body_buffer.size());
// Use "gather-write" to send both the header and data in a
// single write operation.
std::vector<boost::asio::const_buffer> buffers;
buffers.push_back(boost::asio::buffer(header_buffer));
buffers.push_back(boost::asio::buffer(body_buffer));
boost::asio::write(socket_, buffers);
client must call
socket.shutdown(asio::ip::tcp::socket::shutdown_both);
socket.close();
on the server size read until EOF detected
std::string reveive_complete_message(tcp::socket& sock)
{
std::string json_msg;
asio::error_code error;
char buf[255];
while (1)
{
//read some data up to buffer size
size_t len = sock.read_some(asio::buffer(buf), error);
//store the received buffer and increment the total return message
std::string str(buf, len);
json_msg += str;
if (error == asio::error::eof)
{
//EOF received, the connection was closed by client
break;
}
else if (error)
{
throw asio::system_error(error);
}
}
return json_msg;
}
I am capturing some audio from my microphone using SFML.
The data is being stored in samples of type Int16*.
Int16* samples;
My question is. What should I do to this samples to stream it over a socket to be played in another place? I ask in relation of data type. Do I need to convert this Int16 array to another type? Or can I just send this Int16* as it is?
EDIT
void BroadcastRecorder::loadBufferFromSamples()
{
//m_samples is of type vector<Int16*>
if (!m_samples.empty()){
m_buffer.loadFromSamples(&m_samples[0], m_samples.size(), 1, getSampleRate());
m_samples.clear();
}
}
void Broadcaster::Send()
{
//load the buffer with the samples
if(!m_recorder->empty()){
m_recorder->loadBufferFromSamples();
const sf::SoundBuffer& buffer = m_recorder->getBuffer();
size_t dataLength = m_recorder->GetSamplesSize();
wxSocketClient * socket = new wxSocketClient(wxSOCKET_NOWAIT);
socket->Notify(false);
// ------------- DATA----------------------
wxString data = "";
wxString strToPrepend(_("--myboundary\r\nContent-Type: audio/wav\r\n"));
wxString strToAppend(_("\r\n--myboundary\r\n"));
// ------------- HEADER -----------------------
wxString header = "";
header.append("POST ");
header.append("/cgi-bin/operator/transmit");
header.append(" HTTP/1.0\r\n");
header.append("Content-Type: multipart/form-data; boundary=--myboundary\r\n");
header.append("Content-Length: " + wxString::Format(wxT("%i"),(dataLength + strToPrepend.Len() + strToAppend.Len()) ) + "\r\n");
header.append("Authorization: Basic keykeykeykey\r\n");
header.append("\r\n");
//-------------- CONNECTION ---------------
wxString host = _("192.168.50.11");
wxIPV4address * address = new wxIPV4address();
address->Hostname(host);
address->Service(8084);
if (socket->Connect(*address)){
//Write header
socket->Write(header.c_str(),header.Len());
//Write data
socket->Write(strToPrepend.c_str(),strToPrepend.Len());
const sf::Int16* samples = buffer.getSamples();
const char* bytesData = reinterpret_cast<const char*>(samples);
socket->Write(bytesData,dataLength);
socket->Write(strToAppend.c_str(),strToAppend.Len());
socket->Close();
}
delete socket;
delete address;
}
}
I am getting only some noises between gaps.
BTW. The audio is being sent to an IP camera p2 connector.
The data format is just the way your application treats them. After all you send raw bytes over a socket. And you can do it with anything you want
Int16 data;
const char* pBytesOfData = (const char*) &data;
int size = sizeof (Int16);
send( socket, pBytesOfdata, size, flags);
When the bytes arrive on the second end it is up to you to interpret them correctly. Probably you will want again treat them as Int16. You need to have a protocol (common way of communication) to do it right (maybe send size of the data at the begining of the transmission, etc).
You can also take a look on libraries that ease serialization: Boost.Asio and Boost.Serialization.
Firstly, You need to create and bind a socket. Then you have to send the data stored in "samples" to another peer by using socket API. For using socket API to send the data, you need to convert this data to char*. As send API of socket takes input of data you need to send as char*. For more information about sending you can go through this link. This is for windows. For Unix you can check the manpage for send API for unix.
Int16* is a pointer. The samples you get should also have an associated length. Your data will likely be between addresses: [samples, samples + length) (where samples is the address to the first sample).
To play the samples remotely (actual code will depend on what APIs you use):
open socket
in a loop
get samples from your microphone
transmit the data over socket
on the server, you will have to read samples in a loop and send them to whatever sound output API you use.
Sockets work with bytes, so in the end you will send bytes. As long as the way you interpret these bytes on the receiving side matches the data you sent, you can send whatever you want in those bytes.
In this case sending the samples directly without conversion seems the most trivial thing to do, but you will probably need to send the size of the sample before, most likely in a fixed length format, for example:
[size on 4 bytes][sample on `size` bytes]
[] [] [] [][] [] [] [] [] []
I'm having issues with receiving a transfer.
QTcpSocket->readAll() does not read enough bytes when I'm sending to it. When I send 15k bytes, it reads only some part of it and then does nothing. What am I doing wrong?
QByteArray array;
array = socket->readAll(); //just reads some part, not fully.
Why does this happen?
Most probably the socket didn't receive all data yet when you call readAll(). This is because of TCP communication happens in small packets (each having around 1KB of data, depending on a lot of things). These packets make up a stream in which the other end of the communication line writes bytes into. You have to assemble them on the receiving side. How you assemble them has to be defined in a protocol.
To solve this issue, you have to wait for all expected data before assembling it. Sometimes it is not known how much data is expected unless you read it (depending on the protocol).
Let's say you want to implement a protocol which says "everything until a line break is something we call a message". Now you want to receive such a message. This is done by successively reading and appending to a target buffer (like your QByteArray) until there comes a line break. However, there is another thing: When you expect a second message, it can be immediately after the first one in the TCP stream, so you just read not only the end of the first message, but also the beginning of the second. Just keep this in mind.
When not dealing with signal slot connection, you can write a synchronous receiver for such newline-separated messages like this:
QByteArray array;
while(!array.contains('\n')) {
socket->waitForReadyRead();
array += socket->readAll();
}
int bytes = array.indexOf('\n') + 1; // Find the end of message
QByteArray message = array.left(bytes); // Cut the message
array = array.mid(bytes); // Keep the data read too early
processMessage(message);
When handling QTcpSocket::readyRead(), you can do something similar.
void MyClass::socketReadyRead() // connected to QTcpSocket::readyRead() signal
{
array += socket->readAll();
if(array.contains('\n')) {
int bytes = array.indexOf('\n') + 1; // Find the end of message
QByteArray message = array.left(bytes); // Cut the message
array = array.mid(bytes); // Keep the data read too early
processMessage(message);
socketReadyRead(); // re-call myself to process more
}
}
When you want to read everything sent via one TCP connection (until it gets closed by the peer), you can wait for this event either in a blocking way or process the data in a slot connected to the proper signal: QTcpSocket::disconnected.
Blocking:
socket->waitForDisconnected();
QByteArray array = socket->readAll();
Non-blocking (handling signals using slots):
void MyClass::socketReadyRead() // connected to QTcpSocket::readyRead() signal
{
array += socket->readAll();
// Do NOT process yet!
}
void MyClass::socketDisconnected() // connected to QTcpSocket::disconnected() signal
{
processMessage(array);
}
Alternative non-blocking solution (essentially the same):
// You don't have to connect to QTcpSocket::readyRead() signal in this case
void MyClass::socketDisconnected()
{
processMessage(socket->readAll());
}
Currently, I'm learning how to build a transparent HTTP proxy in C++. There had two issues on the proxy client side I couldn't resolve for long time. Hope someone can point out the root causes based on following scenarios. Thanks a lot. :D
The HTTP proxy I built right now is somehow work partially only. For example, I could access google's main page through proxy while I couldn't get any search result after I typed keyword(the google instant is also not working at all). On the other hand, youtube is working perfectly includes searching, loading video and commenting. What's more, there also got some websites like yahoo even couldn't display main page after I keyed in its URL.
The reason why I said the issues are on the proxy client side at the begining is because I traced the data flow of my program. I found out the written size returned by socket programming function write() was smaller than the data size I passed to my write back function. The most weird observation for me was the data losing issue is independent from the size of data. The socket write() function could work properly for youtube video data which is nearly 2MB while it would loss data for google search request which is just 20KB.
Furthermore, there also had another situation that browser displayed blank when the data size I passed to my write back function and the written size returned by socket write function() are the same. I used wireshark to trace the flow of communication and compared mine with pure IP communication without proxy involved. I found out that browser didn't continuously send out HTTP requests after it received certain HTTP responses comparing with pure IP communication flow. I couldn't find out why the browser didn't send out rest of HTTP requests.
Following is my code for write back function:
void Proxy::get_data(char* buffer, size_t length)
{
cout<<"Length:"<<length<<endl;
int connfd;
size_t ret;
// get connfd from buffer
memset(&connfd, 0, sizeof(int));
memcpy(&connfd, buffer, sizeof(int));
cout<<"Get Connection FD:"<<connfd<<endl;
// get receive data size
size_t rData_length = length-sizeof(int);
cout<<"Data Size:"<<rData_length<<endl;
// create receive buffer
char* rBuf = new char[rData_length];
// allocate memory to receive buffer
memset(rBuf, 0, rData_length);
// copy data to buffer
memcpy(rBuf, buffer+sizeof(int), rData_length);
ret = write(connfd, rBuf, rData_length);
if(ret < 0)
{
cout<< "received data failed"<< endl;
close(connfd);
delete[] rBuf;
exit(1);
}
else
{
printf("Write Data[%d] to Socket\n", ret);
}
close(connfd);
delete[] rBuf;
}
May be you could try this
int curr = 0;
while( curr < rData_length ) {
ret = write( connfd, rBuf + curr, rData_length - curr );
if( ret == -1 ) { /* ERROR */ }
else
curr += ret;
}
instead of
ret = write(connfd, rBuf, rData_length);
In general, the number of bytes written by write() could differ from what you ask to write. You should better read some manual. Say, http://linux.die.net/man/2/write
Copying bytes between an input socket and an output socket is much simpler than this. You don't need to dynamically allocate buffers according to how much data was read by the last read. You just need to read into a char[] array and write from that array to the target, taking due account of the length value returned by the read.