Differences between libpcap packet results and bsd socket recv data using C++ - c++

I have relatively limited network programming experience and was hoping someone could help me with perhaps a n00bish question, thanks in advance for any help! :)
I have two C++ programs running separately. One is a simple packet sniffer written with the libpcap library that is filtering such that it only listens to packets received on a specific port, and then parses the packet such that it writes out the payload length and payload content to a binary file.
The other program is a simple bsd socket server program that is set to receives messages on that specific port with the recv() function, and then similarly writes the received message length and message content out to a binary file.
Everything appears to be running fine, so when I check the binary output files I expect the content to be the same... but unfortunately, I am finding they are only somewhat similar. It seems that the packet sniffer is catching the same data that the server is catching, but it is also recording extra packets with 6-byte long payloads, that apparently aren't being received by the server program.
So, either the packet sniffer is picking up packets that the server isn't supposed to know about, or the server program is missing packets that it's supposed to be receiving. However, I feel somewhat confident that the server program is functioning properly and is receiving the right messages, so these extra packets being detected by the packet sniffer baffle me. Does anyone know what these extra packets may be, and how I might filter them out?
Notes:
The client that is sending data to the ports I'm listening on is an old Windows NT machine in a small network, it's basically passing binary data to the server program for processing.
Unfortunately, I can't upload the code thanks to workplace policies. However, examples of what my code is like may be found here:
http://www.tcpdump.org/pcap.html
http://beej.us/guide/bgnet/output/html/singlepage/bgnet.html
Thanks again!

One is a simple packet sniffer written with the libpcap library that is filtering such that it only listens to packets received on a specific port, and then parses the packet such that it writes out the payload length and payload content to a binary file.
TCP port or UDP port?
If it's a TCP port, your program might receive the 3-way handshake packets and ACK-only packets.
How are you calculating the payload length? You should do it by taking the IP packet's total length, subtracting from it the IP header length (not 20 bytes, but the length from the version/header length field, so that you handle IP options; if that length is < 5, i.e. < 20 bytes, just discard the packet, it's bogus), to get the total TCP packet length, and then subtract from that the TCP header length (not 20 bytes, but the "Data Offset" field, so you handle TCP options; if that length is < 5, i.e. < 20 bytes, just discard the packet, it's bogus) to get the TCP payload length.
That should handle the options to which mfontanini refers, as well as the IP options.
If it's a UDP port, you should still handle IP options and get the total UDP packet length, and then:
if that's less than the length in the UDP header, discard the packet, it's bogus;
if the length in the UDP header is < 8, discard the packet, it's bogus;
otherwise, use the length in the UDP header, minus 8, as the payload length.
The other program is a simple bsd socket server program that is set to receives messages on that specific port with the recv() function, and then similarly writes the received message length and message content out to a binary file.
That won't see the initial handshake or ACK-only packets.

My magic ball indicates those 6 bytes long payloads might be just TCP options. You should skip those options if you only want to work with the payload. Note that some packets might contain both a payload and some options.
The data offset field in the TCP packet indicates at which point the payload starts. You can have a look at that here in section 3.1.
I'd anyway recommend you using some higher level library which deals with that stuff internally. Using libtins you could do that task using some short snippet such as:
#include <tins/tins.h>
using namespace Tins;
bool callback(const PDU &pdu) {
const RawPDU &raw = pdu.rfind_pdu<RawPDU>();
// raw.payload() returns a std::vector<uint8_t>
process_payload(raw.payload());
return true;
}
int main() {
Sniffer some_sniffer(
"eth0",
Sniffer::NON_PROMISC,
"some pcap filter"
);
some_sniffer.sniff_loop(callback);
}

Related

rtp analyzer written in C++ in linux environment

I need to write (c++, linux) rtp analyzer like:http://mmlab.disi.unitn.it/wiki/index.php/Analyzer. But I can't understand some technical moments, such as how server, wich receives udp-datagrams, determines that one udp-datagram had finished and another udp-datagram started and how it identifies udp-datagrams, containing rtp. It would be great to see the code of analyzer, but I haven't find it.
Could you help me to find the answers to my questions (I wonder to know how it can be implemented via sockets). Thank you very much, I will really appreciate it!
If I do understand your question correctly, you can follow the steps stated below:
Write a server program in c++ using one of the socket libraries, and create a UDP socket.
Listen to the port to which your client is sending the RTP packets.
While you listen to the specified UDP port, you will get UDP packets one by one.
When you get a UDP packet parse the length field which is a 2 byte field, and starts at 4th byte.
Length - 8 (UDP header size in bytes) is what you looking for, and actually the size of the RTP packet.
Get the RTP packet by getting the bits between 8th byte of the UDP packet to Length'the byte.
Then you have your RTP packet, you can also parse it in the same way by looking at how it looks like in RFC3550 by IETF.
(Also good to specify that UDP packets might arrive out of order, and you can use the sequence number field in the RTP header, to reorder them)

How to receive more than 1440 through the socket

I wrote two simple programs server and a client using sockets in C++ (Linux). And initially it was a sample client-server application (echo-message sending and receiving the answer). Next, I changed the client in order to implement HTTP GET (now I do not use my sample server anymore). It works, but whatever buffer size I set, the client receives only 1440 bytes. I want to receive whole page into the buffer. I think that this is concerned with the TCP properties and I should implement some kind of cycle inside my client's code to capture all the parts of the answer. But I don't know what exactly I should do.
This is my code:
...
int bytesSent = send(sock, tmpCharArr, message.size()+1, 0);
// Wait for the answer. Receive it into the buffer defined.
int bytesRecieved = recv(sock, resultBuf, 2048*100, 0);
...
2048*100 is a buffer size and I think this is more than enough for the relatively small WEB-page used for testing. But as I mentioned, I receive only 1440 bytes.
What can I do with recv() function call to capture all the reply "parts" when the server's response is larger then 1440 bytes?
Thanks in advance.
The buffer size is determined by factors outside your control (routers, ADSL links, IP stacks, etc.). The standard way to transmit large volumes of data is to call recv() repeatedly.
HTTP works over TCP, and to understand the working of TCP sockets better you have to think of them as a stream rather than packets.
For further clarity, read my earlier post: recombine split TCP packet with flash sockets
As to why you receive only 1400 (or so) bytes, you have to understand MTU and Fragmentation. To sum it up, MTU (Maximum Transmission Unit) is the ability of the network to transfer a single packet of a certain maximum size. MTU of a entire network is the lowest MTU of all the routers involved. Fragmentation is splitting up of the packets if you try to send a single packet of size larger than the MTU of that network.
For a better understanding of MTU and Fragmentation, read: http://www.miislita.com/internet-engineering/ip-packet-fragmentation-tutorial.pdf
Now as for how to receive the entire page in the buffer, one alternative is to keep calling recv() and appending the data you get in a buffer, until recv() returns zero. This will work because typically a web-server will close the TCP connection after it sends you the response. However, this technique will fail to work if the web-server doesn't close the session (maybe keep-alives are configures).
Therefore, the correct solution would be to keep receiving until you have received the HTTP header. Take a peek and determine the length of the entire HTTP response (Content-Length:) and then you can keep on receiving until you have received the exact amount of bytes you were supposed to receive.

Receive packet by packet data from TCP socket?

I have a tcp socket on which I receive video stream. I want to receive data as packet by packet from socket so that I could remove the packet header and keep the only stream data. How can I do this??
any help will be appreciated.
You can't. TCP doesn't work with packets / messages etc. TCP works with bytes. You get a stream of bytes. The problem is that there's no guarantee reagarding the number of bytes you'll get each time you read from a socket. The usual way to handle this:
When you want to send a "packet" include as the first thing a length
When you read stuff from a socket make sure you read at least that length
Your message could be:
|Message Length:4bytes|Additional header Information:whatever1|Message Data:whatever2|
What you'll then have to do is read 4 bytes and then read as much as those 4 bytes tell you. Then you'll be able to strip the header and get the data.
As others have mentioned, TCP is a streaming protocol. This means from an API point of view, there is no concept of "packet". As a user, all you can expect is a stream of data.
Internally, TCP will break the stream into segments that can be placed into IP packets. These packets will be sent along with control data, over IP, to the remote end. The remote end will receive these IP packets. It may discard certain IP packets (in the case of duplicates), reorder the packets or withhold data until earlier packets have arrived. All this is internal to TCP meaning the concept of a "TCP packet" is meaningless.
You might be able to use raw sockets to receive the raw IP packets but this will mean you will have to reimplement much of the TCP stack (like sending ACKs and adjusting window size) to get the remote end to perform correctly. You do not want to do this.
UDP, on the other hand, is a datagram protocol. This means that the user is made aware of how the data is sent over the network. If the concept of packets or datagrams are important to you, you will need to build your own protocol on top of UDP.
TCP is a stream protocol and it doesn't guaranty that when you call socket read function you will receive one, complete packet. UDP or SCTP are packet oriented protocols and guaranty this. For TCP you can get part of the packet or few packet at once. You have to build your own application protocol on top of TCP and fragment/defragment messages manually.
TCP is a streaming protocol. You get bytes with no message boundaries. The solution is to buffer all your reads and extract/process full video packets from the buffer.
Algorithm:
Initialize an empty buffer.
Examine buffer for a complete packet.
If found, remove complete packet from beginning of buffer and process it.
If not found, append data from a recv() to the buffer and go to #2.
What a "complete packet" contains should be defined by the video streaming protocol.
Are you pretty sure about this approach? In my opinion these "preprocessing" will introduce an additional overhead to the system. And of course this is handled by a lower layer (Read about OSI model) so it is not easy to change. Note that most of the existing streaming protocols are already optimized for the best performance.

Receiving data in packets on TCP client

Does recv() call intercepts data in packets or can i get data packets with timestamps?
On a datagram socket (like UDP), recv gets data in datagrams. TCP is a stream-mode socket, however, and recv gets a collection of bytes with no regard for packets.
It's possible, using low-level APIs, to get the packets, but if you were hoping to see boundaries between send calls you are out of luck... that information is not present in the packets.
Recv gets data from a socket that has been successfully received. It does not tell you when that happened; i.e. no timestamp.
Would you elaborate on what problem you're trying to solve ("why do you need this?") instead of your attempted solution? (Or have I completely misunderstood your question?)
If your own code is sending data to the remote machine where you are receiving data...then you can make you r own application level data format...such as sending the data after sending timestamp (some specified number of bytes).
This information can be extracted at the receiving end. Although as mentioned connection is TCP ...the data would be in stream format not as a complete packet as in case of UDP.

Count the number of packets sent to a server from a client?

So I'm almost done an assignment involving Win32 programming and sockets, but I have to generate and analyze some statistics about the transfers. The only part I'm having trouble with is how to figure out the number of packets that were sent to the server from the client.
The data sent can be variable-length, so I can't just divide the total bytes received by a #define'd value.
We have to use asynchronous calls to do everything, so I've been trying to increment a counter with every FD_READ message I get for the server's socket. However, because I have to be able to accept a potentially large file size, I have to call recv/recvfrom with a buffer size around 64k. If I send a small packet (a-z), there are no problems. But if I send a string of 1024 characters 10x, the server reports 2 or 3 packets received, but 0% data loss in terms of bytes sent/received.
Any idea how to get the number of packets?
Thanks in advance :)
This really boils down to what you mean by 'packet.'
As you are probably aware, when a TCP/UDP message is sent on the wire, the data being sent is 'wrapped,' or prepended, with a corresponding TCP/UDP header. This is then 'wrapped' in an IP header, which is in turn 'wrapped' in an Ethernet frame. You can see this breakout if you use a sniffing package like Wireshark.
The point is this. When I hear the term 'packet,' I think of data at the IP level. IP data is truly packetized on the wire, so packet counts make sense when talking about IP. However, if you're using regular sockets to send and receive your data, the IP headers, as well as the TCP/UDP headers, are stripped off, i.e., you don't get this information from the socket. And without that information, it is impossible to determine the number of 'packets' (again, I'm thinking IP) that were transmitted.
You could do what others are suggesting by adding your own header with a length and a counter. This information will help you accurately size your receive buffers, but it won't help you determine the number of packets (again, IP...), especially if you're doing TCP.
If you want to accurately determine the number of packets using Winsock sockets, I would suggest creating a 'raw' socket as suggested here. This socket will collect all IP traffic seen by your local NIC. Use the IP and TCP/UDP headers to filter the data based on your client and server sockets, i.e., IP addresses and port numbers. This will give an accurate picture of how many IP packets were actually used to transmit your data.
Not a direct answer to your question but rather a suggestion for a different solution.
What if you send a length-descriptor in front of the data you want to transfer? That way you can already allocate the correct buffer size (not too much, not too little) on the client and also check if there were any losses when the transfer is over.
With TCP you should have no problem at all because the protocol itself handles the error-free transmission or otherwise you should get a meaningful error.
Maybe with UDP you could also split up your transfer into fixed-size chunks with a propper sequence-id. You'd have to accumulate all incoming packages before you sort them (UDP makes no guarantee on the receive-order) and paste the data together.
On the other hand you should think about it if it is really necessary to support UDP as there is quite some manual overhead if you want to get that protocol error-safe... (see the Wikipedia Article on TCP for a list of the problems to get around)
Do your packets have a fixed header, or are you allowed to define your own. If you can define your own, include a packet counter in the header, along with the length. You'll have to keep a running total that accounts for rollover in your counter, but this will ensure you're counting packets sent, rather than packets received. For an simple assignment, you probably won't be encountering loss (with UDP, obviously) but if you were, a packet counter would make sure your statistics reflected the sent message accurately.