Application Scenario:
I receive many network packets, and group them by tcp stream. I can make sure that the packets that from one specific. The first received packet is the head of gzip content(0x1f 0x8b 0x08...), so I initialize the zlib_steam and then can decompress this packet successfully, but I cannot make sure the following received packet is Arrive in order(they may be out-of-order or lost), When the received packet is not the next packet of pre-packet, zlib_steam mostly failed to decompress the packet.
So I have the question: If you give me a partial compressed gzip data(for example one HTTP packet with gzip content-encoding but it's just a part of the complete gzip content), can zlib decompress it successfully?
Related
I'm using the boost::asio::ip::udp::socket to receive UDP packets via socket's async_receive_from method.
The code works fine, the only problem is that in the time I process a packet, lots more come creating a queue (the buffer) to process. My program though must sink all the packets received since the start of the processing, so that it listens only to the most recent ones.
Example:
packet 1 is sent
packet 1 is being processed
packets 2, 3, 4 are sent
packet 1 ends the computation
flush the buffer
packet 5 is sent
packet 5 is being processed
etc
Is there any way to discard the packets in the middle?
Thanks!
Use a buffer that holds only a single datagram.
Keep reading into the buffer until there are no more datagrams to read.
If you read at least one packet, process the datagram in the buffer.
Go to step 2.
Note that UDP is a datagram protocol, not a packet protocol. A single UDP datagram can be split over multiple packets.
I think it can be done quite simple:
async_receive_from until got a packet.
Check available method to determine is more data in the socket.
If we have more data, discard buffer and goto 1; else process packet
I have Ogg format file containing OPUS frames. As per my requirement, I need to parse this file (frames/packets), and send OPUS compressed data to a remote device through RTP.
My question is in one RTP packet (assuming 48khz sampling rate)
1. One OPUS frame will be sent
2. Or multiple OPUS frames will be sent
3. Or one packet as per Ogg file format specification, which may be one frame, 2 frames or arbitrary number of frames will be sent
Each Opus RTP packet contains only one Opus packet, as defined by the Opus specification. That may contain more than one Opus frame internally, but it must have the correct header bytes to signal this and conform to other rules, so make sure you mean the same thing the spec does by "frame".
Basically, you want to send each Opus packet out of the Ogg file in its own RTP packet. There's no packing at the RTP payload level. Don't send the Id or Comment headers in the first two packets of the .opus Ogg stream, and of course you need to prepend RTP headers with the appropriate flags, timestamp and so on.
See https://git.xiph.org/?p=opus-tools.git;a=blob;f=src/opusrtp.c#l517 for a toy implementation of this.
I have an application that compresses and sends data via socket and data received is written in remote machine. During recovery, this data is decompressed and retrieved. Compression/Decompression is done using "zlib".But during decompression I face the following problem randomly:
zlib inflate() fails with error "Z_DATA_ERROR" for binary files like .xls,.qbw etc.
The application compresses data in blocks say "1024" bytes in a loop with data read from the file and decompresses in the same way.From the forum posts, I found that one reason for Z_DATA_ERROR is due to data corruption. As of now, to avoid this problem, we have introduced CRC check of data compressed during send and what is received.
Any possible reasons on why this happens is really appreciated! (as this occurs randomly and for the same file, it works the other time around).Is it bcoz of incorrect handling of zlib inflate() and deflate() ?
Note: If needed,will post the exact code snippet for further analysis!
Thanks...Udhai
You didn't mention if the socket was TCP or UDP; but based on the blocking and checksumming, I'm going out on a limb and guessing it's UDP.
If you're sending the compressed packets over UDP they could be received out-of-order on the other end, or the packets could be lost in transit.
Getting things like out-of-sequencing and lost packets right ends up being a lot of the work that is all fixed by using the TCP protocol - you have a simple pipe that guarantees the data arrives in-order and as-expected.
Also I'd make sure that the code on the receiving side is simple, and receives into buffers allocated on the heap and not on the stack (I've seen many a bug triggered by this).
Again, this is just an educated guess based on the detail of the question.
I have a tcp socket on which I receive video stream. I want to receive data as packet by packet from socket so that I could remove the packet header and keep the only stream data. How can I do this??
any help will be appreciated.
You can't. TCP doesn't work with packets / messages etc. TCP works with bytes. You get a stream of bytes. The problem is that there's no guarantee reagarding the number of bytes you'll get each time you read from a socket. The usual way to handle this:
When you want to send a "packet" include as the first thing a length
When you read stuff from a socket make sure you read at least that length
Your message could be:
|Message Length:4bytes|Additional header Information:whatever1|Message Data:whatever2|
What you'll then have to do is read 4 bytes and then read as much as those 4 bytes tell you. Then you'll be able to strip the header and get the data.
As others have mentioned, TCP is a streaming protocol. This means from an API point of view, there is no concept of "packet". As a user, all you can expect is a stream of data.
Internally, TCP will break the stream into segments that can be placed into IP packets. These packets will be sent along with control data, over IP, to the remote end. The remote end will receive these IP packets. It may discard certain IP packets (in the case of duplicates), reorder the packets or withhold data until earlier packets have arrived. All this is internal to TCP meaning the concept of a "TCP packet" is meaningless.
You might be able to use raw sockets to receive the raw IP packets but this will mean you will have to reimplement much of the TCP stack (like sending ACKs and adjusting window size) to get the remote end to perform correctly. You do not want to do this.
UDP, on the other hand, is a datagram protocol. This means that the user is made aware of how the data is sent over the network. If the concept of packets or datagrams are important to you, you will need to build your own protocol on top of UDP.
TCP is a stream protocol and it doesn't guaranty that when you call socket read function you will receive one, complete packet. UDP or SCTP are packet oriented protocols and guaranty this. For TCP you can get part of the packet or few packet at once. You have to build your own application protocol on top of TCP and fragment/defragment messages manually.
TCP is a streaming protocol. You get bytes with no message boundaries. The solution is to buffer all your reads and extract/process full video packets from the buffer.
Algorithm:
Initialize an empty buffer.
Examine buffer for a complete packet.
If found, remove complete packet from beginning of buffer and process it.
If not found, append data from a recv() to the buffer and go to #2.
What a "complete packet" contains should be defined by the video streaming protocol.
Are you pretty sure about this approach? In my opinion these "preprocessing" will introduce an additional overhead to the system. And of course this is handled by a lower layer (Read about OSI model) so it is not easy to change. Note that most of the existing streaming protocols are already optimized for the best performance.
Does recv() call intercepts data in packets or can i get data packets with timestamps?
On a datagram socket (like UDP), recv gets data in datagrams. TCP is a stream-mode socket, however, and recv gets a collection of bytes with no regard for packets.
It's possible, using low-level APIs, to get the packets, but if you were hoping to see boundaries between send calls you are out of luck... that information is not present in the packets.
Recv gets data from a socket that has been successfully received. It does not tell you when that happened; i.e. no timestamp.
Would you elaborate on what problem you're trying to solve ("why do you need this?") instead of your attempted solution? (Or have I completely misunderstood your question?)
If your own code is sending data to the remote machine where you are receiving data...then you can make you r own application level data format...such as sending the data after sending timestamp (some specified number of bytes).
This information can be extracted at the receiving end. Although as mentioned connection is TCP ...the data would be in stream format not as a complete packet as in case of UDP.