I am developing MQTT server in Clojure. I use Gloss to parse binary data. However, I'm stuck with Variable Byte Integer, not sure how to create codec for this. My gut feeling it should use header frame somehow, but I'm not sure how to put it together. So my questions are: 1. Is it possible to represent this as Gloss codec, or I should use some other approach? 2. Can you please provide codec for this? Thanks!
Reading the docs, it looks like you'll need to define something new to process the VBI encoding.
Since Gloss appears to expect only fixed-length representations, your algorithm will probably need to accept a block of bytes with a VBI at the beginning, then decode the VBI and return
the decoded value
the remaining bytes
The bytes from (2) can then be input to Gloss as normal. If the VBI is not the first item in the overall input, you can strip off any leading bytes using Gloss or any other means.
Ended up creating custom codec for Gloss https://github.com/selentium/clj-mqtt/blob/master/src/clj_mqtt/varint.clj
Related
The professor gave me a research paper that shows a way to efficiently compress some kind of data.
It's not worth to eplain the full algorithm since the question is not about that, I just introduce a little example that should allow you to undestand what the real question is about.
Our compression algorithm have is own dictionary which is a table (no matter how it is calculated, just assume that both compressor and decompressor have it), each table row has a string.
The compressor in order to compress a message will open it and start from begining, it will search for a match in the dictionary and eventually send a MATCH message with the row id, if nothing is found then a SET message with the message to set is sent.
Note that MATCH do not really have to be complete match, they can be followed by many MISSMATCH message each containing the byte offset wrong and the correct byte.
So for example the compressor might want to encode:
Now, in the paper they say that they entropy encode this "stream" of data using LZMA and they assume it's a trivial thing to do without giving further details.
I've searched online but I didn't come up with anything. Do you have any idea on how this last step could be done? Do you have any reference?
There is a stream compression algorithm with preset dictionary using LZMA as part of this open-source project: Zip-Ada . The preset dictionary is called there "training data".
How to encode Vessel data (position, course, speed & ect.) with AIS (Automatic Information System) AIVDM/AIVDO sentence.
You can find a php version for encoding at my repo here - phpais
It should be pretty straight forward to understand the code if you understand how AIS strings are composed
This is a very vague question it depends on which AIS message you want to send? (See the second link for the different msg.)
As you might see, it is a rather complexed task, but basically you have to do what is done here it-digin.com in reversed order. But there are several different message types, you can read more about the standard here United States Coast Guard
did you try ais2csv ? It works well with NMEA sentences
https://github.com/dma-ais/AisLib/tree/master/ais-lib-cli/launch4j/ais2csv
I have an char* array of binary data.
It is binary media-stream encoded with h.264.
It has next structure: ...
stream_header is 64 bytes struct.
I've already done reinterpret_cast(charArray) where chararray represents first 64 bytes of stream. I'm successfully get all header data. In this header there is an nLength variable, which tell us how many bytes of media data is in next stream_data.
For example 1024 bytes.
I read next 1024 bytes in char* data array, and here my question begins: how I can get from this data set of video frames (in structure i have info about resolution of this frames), and save it in *.jpg files such as (1.jpg 2.jpg 3.jpg .....)
Maybe someone has already done something simmilar??? Help me plz..
You need an H264 decoder library, best option is ffmpeg
But even then it's a bit complicated to use the library - although decoding is simpler since you have less options to worry about.
Do you really need to do this in a program? It's very simple to use the 'ffmpeg' executable to save a video as jpegs
If you just want to get a sequence of JPEGs from a video file, GStreamer can do that among many other things.
If you want to write code from scratch to convert H.264 video into JPEGs, let me warn you that you have many hundreds of pages of specifications documents and some very serious mathematics to understand and then implement. It would be months of work for a reasonably skilled programmer mathematician. Understanding the MP4 format is the easy part, the video compression will blow your mind.
The input data is a byte array which represents a h.264 frame. The frame consists of a single slice (not multislice frame).
So, as I understood I can cope with this frame as with slice. The slice has header, and slice data - macroblocks, each macroblock with its own header.
So I have to parse that byte array to extract frame number, frame type, quantisation coefficient (as I understood each macroblock has its own coefficient? or I'm wrong?)
Could You advise me, where I can get more detailed information about parsing h.264 frame bytes.
(In fact I've read the standard, but it wasn't very specific, and I'm lost.)
Thanks
The H.264 Standard is a bit hard to read, so here are some tips.
Read Annex B; make sure your input starts with a start code
Read section 9.1: you will need it for all of the following
Slice header is described in section 7.3.3
"Frame number" is not encoded explicitly in the slice header; frame_num is close to what you probably want.
"Frame type" probably corresponds to slice_type (the second value in the slice header, so most easy to parse; you should definitely start with this one)
"Quantization coefficient" - do you mean "quantization parameter"? If yes, be prepared to write a full H.264 parser (or reuse an existing one). Look in section 9.3 to get an idea on a complexity of a H.264 parser.
Standard is very hard to read. You can try to analyze source code of existing H.264 video stream decoding software such as ffmpeg with it's C (C99) libraries. For example there is avcodec_decode_video2 function documented here. You can get full working C (open file, get H.264 stream, iterate thru frames, dump information, get colorspace, save frames as raw PPM images etc.) here. Alternatively there is great "The H.264 Advanced Video Compression Standard" book, which explains standard in "human language". Another option is to try Elecard StreamEye Pro software (there is trial version), which could give you some additional (visual) perspective.
Actually much better and easier (it is only my opinion) to read H.264 video coding documentation.
ffmpeg is very good library but it contain a lot of optimized code. Better to look at reference implementation of the H.264 codec and official documentation.
http://iphome.hhi.de/suehring/tml/download/ - this is link to the JM codec implementation.
Try to separate levels of decoding process, like transport layer that contains NAL units (SPS, PPS, SEI, IDR, SLICE, etc). Than you need to implement VLC engine (mostly exp-Golomb codes of 0 range). Than very difficult and powerful codec called CABAC (Context Adaptive Arithmetic Binary Codec). It is quite tricky task. Demuxing process (goes after unpacking of a video data) also complicated. You need completely understand each of such modules.
Good luck.
So all I need is a simple function that sets it up (eating incoming PCM RATE (for example: rate near to 44100) It's channels (for example: 2) and -bits (for example: 16) and desirable 128 kb\s rate) and another one that takes PCM data and encodes it into pure MP3 frames.
I know it looks like a silly homework task but I assure you - it is not.
I hope it will be of help to all C++ developers starting with MP3s.
So can anybody please help me with that?
See the example I gave in your other question for the basic usage of Lame. It should contain everything you need.
It's a long time since i messed with this, but Lame lib contains all you need to do it, check out lame.h , there is some test code you could look into.
Personally, I'd be looking at ffmpeg's libavcodec. There's an example file containing an audio_encode_example which ought to be more or less what you're looking for.
See also this question.
Note that not all ffmpeg packagings include mp3 codec support (patent issues) by default, although there's usually some simple way of enabling it.
GStreamer should definitely be able to handle this.
If you are on Windows you can use the GStreamer Winbuilds to get started.