Android MediaPlayer / Stagefright crashes reading SDP from RTSP server - c++

I am trying to use Android's MediaPlayer class to play an RTSP stream. MediaPlayer calls to a native library called stagefright to make this happen. This library, and the entire media framework, crash on the following error:
15988-23172/? A/APacketSource﹕ frameworks/av/media/libstagefright/rtsp/APacketSource.cpp:143 CHECK_EQ( (unsigned)nalType,7u) failed: 22 vs. 7
So it seems pretty obvious what the error is - the assert fails when nalType is supposed to be 7, but instead evaluates to 22. I don't understand why this is though. For context, the RTSP stream's SDP contains an attribute sprop-parameter-sets, which is a base64-encoded set of bytes. nalType seems to refer to the first byte of the result of base64-decoding sprop-parameter-sets. In this instance, this is the base64-encoded version:
NjdkMDAxNmRhMDI4MGY2YzA1NTIwMDAwMDAzMDA0MDAwMDAwNzhjMDgwMDNlODAwMDBhOGMzN2JkZjBiYzIy
Decoded, it should look like this:
67d0016da0280f6c0552000000300400000078c08003e80000a8c37bdf0bc22
Below is the relevant excerpt from APacketSource.cpp, from the Android 4.4 source code:
// Appears to assign value of "sprop-parameter-sets" from SDP to 'val'
if (!GetAttribute(params, "sprop-parameter-sets", &val)) {
return NULL;
}
size_t start = 0;
for (;;) {
ssize_t commaPos = val.find(",", start);
size_t end = (commaPos < 0) ? val.size() : commaPos;
AString nalString(val, start, end - start);
sp<ABuffer> nal = decodeBase64(nalString);
CHECK(nal != NULL);
CHECK_GT(nal->size(), 0u);
CHECK_LE(nal->size(), 65535u);
// Mask away everything but 0001 1111 from the first byte
uint8_t nalType = nal->data()[0] & 0x1f;
if (numSeqParameterSets == 0) {
// Line 143, where the failure happens.
CHECK_EQ((unsigned)nalType, 7u);
}
...
}
The first byte, 0x67, after being masked, should evaluate to 0x07. Somehow, it valuates to 0x16. I have no idea why this may be, though I suspect maybe something unusual is being returned by nal->data().
I can provide the stacktrace that gets printed to the log, but it's quite large and I'm not sure it's relevant. Why is this assert failing?

Related

IMFTransfomer::ProcessInput() and MF_E_TRANSFORM_NEED_MORE_INPUT

I have code that decodes AAC-encoded audio using IMFTransform. It works well for various test inputs. But I observed that in some cases IMFTransform::ProcessOutput() returns MF_E_TRANSFORM_NEED_MORE_INPUT when according to my reading of MS documentation it should return a valid data sample.
Basically the code has the following structure:
IMFTransform* transformer;
MFT_OUTPUT_DATA_BUFFER output_data_buffer;
...
bool try_to_get_output = false;
for (;;) {
if (try_to_get_output) {
// Try to get the outpu sample.
try_to_get_output = false;
output_data_buffer.dwStatus = 0;
...
hr = transformer->ProcessOutput(...&output_data_buffer);
if (success) {
// process sample
if (output_data_buffer.dwStatus & MFT_OUTPUT_DATA_BUFFER_INCOMPLETE) {
// We have more data
try_to_get_output = true;
}
} else if (hr == MF_E_TRANSFORM_NEED_MORE_INPUT) {
Log("Unnecessary ProcessOutput()");
} else {
// Process other errors
}
continue;
}
// Send more encoded AAC data to MFT.
hr->ProcessInput();
}
What happens is that ProcessOutput() sets MFT_OUTPUT_DATA_BUFFER_INCOMPLETE in MFT_OUTPUT_DATA_BUFFER.dwStatus but then the following ProcessOutput() always returns MF_E_TRANSFORM_NEED_MORE_INPUT contradicting the documentation.
Again, so far it seems harmless and things works. But then what exactly does AAC decoder want to tell the caller via setting MFT_OUTPUT_DATA_BUFFER_INCOMPLETE?
This might be a small glitch in the decoder implementation. Quite possible that if you happen to drain the MFT it would spit out some data, so the incompletion flag migth indicate, a bit confusingly, some data even though not immediately accessible.
However overall the idea is to do ProcessOutput sucking the output data for as long as possible until yщu get MF_E_TRANSFORM_NEED_MORE_INPUT, and then proceed with feeding new input (or draining). That is, I would say MF_E_TRANSFORM_NEED_MORE_INPUT is much more important compared to MFT_OUTPUT_DATA_BUFFER_INCOMPLETE. After all this is what Microsoft's own code over MFTs does.
Also keep in mind that AAC decoder is an "old", "first generation" MFT and so over years its update could be such that it diverted a bit from the current docs.

pjsip capture and play pcm data

I have some embedded Devices that have no audio device by default. They communicate with each other via a FPGA. So my question is, how do I capture/play back audio from pjsip in pcm in order to send/receive it with the FPGA?
I know that there is pjmedia_mem_player_create() and pjmedia_mem_capture_create() but I can't seem to find any good info towards using these functions.
I tried the following piece of code, but an assertion failed cause one of the function's parameter is "empty".
Error:
pjmedia_mem_capture_create: Assertion `pool && buffer && size && clock_rate && channel_count && samples_per_frame && bits_per_sample && p_port' failed.
Note: I'm mainly using pjsua2 for everything else like registrations, transports etc. Also the default audio is set to null with ep.audDevManager().setNullDev(); as without this, making/receiving a call would simply fail?!
void MyCall::onCallMediaState(OnCallMediaStateParam &prm){
CallInfo ci = getInfo();
pj_caching_pool_init(&cp, &pj_pool_factory_default_policy, 0);
pj_pool_t *pool = pj_pool_create(&cp.factory, "POOLNAME", 2000, 2000, NULL);
void *buffer;
pjmedia_port *prt;
#define CLOCK_RATE 8000
#define CHANELS 1
#define SAMPLES_PER_FRAME 480
#define BITS_PER_SAMPLE 16
pjmedia_mem_capture_create( pool, //Pool
buffer, //Buffer
2000, //Buffer Size
CLOCK_RATE,
CHANELS,
SAMPLES_PER_FRAME,
BITS_PER_SAMPLE,
0, //Options
&prt); //The return port}
UPDATE
The assertion failed cause the buffer variable doesn't have any memory allocated to it. Allocate with twice the amount of samples per frame to have sufficient memory.
buffer = pj_pool_zalloc(pool, 960);
Also a callback needs to be registered with pjmedia_mem_capture_set_eof_cb2() (The two at the end is necessary for PJSIP 2.10 or later) Apparently from there the buffer can be used. Just that my implementation atm doesn't execute the callback.
Looks like I found the solution, I have modified your code and wrote a simple code in C with pjsua API to dump every frame to file. Sorry for mess, I'm not proficient in C:
pjsua_call_info ci;
pjsua_call_get_info(call_id, &ci);
pjsua_conf_port_info cpi;
pjsua_conf_get_port_info(ci.conf_slot, &cpi);
pj_pool_t *pool = pjsua_pool_create("POOLNAME", 2000, 2000);
pjmedia_port *prt;
uint buf_size = cpi.bits_per_sample*cpi.samples_per_frame/8;
void *buffer = pj_pool_zalloc(pool, buf_size);
pjsua_conf_port_id port_id;
pjmedia_mem_capture_create( pool,
buffer,
buf_size,
cpi.clock_rate,
cpi.channel_count,
cpi.samples_per_frame,
cpi.bits_per_sample,
0,
&prt);
pjmedia_mem_capture_set_eof_cb(prt, buffer, dump_incoming_frames);
pjsua_conf_add_port(pool, prt, &port_id);
pjsua_conf_connect(ci.conf_slot, port_id); //connect port with conference
///////dumping frames///
static pj_status_t dump_incoming_frames(pjmedia_port * port, void * usr_data){
pj_size_t buf_size = pjmedia_mem_capture_get_size(port);
char * data = usr_data;
...
fwrite(data,sizeof(data[0]),buf_size,fptr);
...
}
Documenation says pjmedia_mem_capture_set_eof_cb is deprecated but I couldn't make work pjmedia_mem_capture_set_eof_cb2, buf_size is 0 for every call of dump_incoming_frames so just left with deprecated function. I also succeed the same result with creating custom port.
I hope you can modify it easily to your C++/pjsua2 code
UPD:
I have modified the PJSIP and packed audio in-out streaming into proper PJSUA2/Media classes so it can be called from Python. Full code is here.

recieving Frame of 8 bytes in QT Creator GUI via serial port

I'm working to send frame of 8 bytes to Micro-controller Xmega128a1 (via RS232) the frame looks like this
{header1,header2,CMD,D1,D2,D3,D4,CRC},
for example
{0x55,0xaa,0xFF,0x59,0xfd,0x64,0x68,0x32},
Micro-controller has to resend the frame back to PC, if it's 'correct'.
I built GUI in QT Creator I defined the Headers (header0=0x55, header1=0xaa) and CMD=01 also calculated the CRC,
the user has to enter the data field in the Line_Edit which is value in RPM(Real value) The Micro-controller Receive the frame byte byte and resend the full frame, so I have to send the frame in the form of bytes, when I send the frame I receive the headers, command and CRC correctly, but data field Received not in proper way such in the picture below, my problem is with converting the input value in the Line_Edit to bytes to be send inside the frame, when I tried to send the value 1265 RPM I received the frame {55aa0100209e44fb} but I want to receive the frame look like this {55aa014F109e44fb}, where: (1265)DC=(4F1)HEX, I couldn't figure what's the problem with my code:
the way I read data from serial port:
void MainWindow::read()
{
uint64_t size = serial->bytesAvailable();
if (size > 0)
{
QByteArray data;
data.append(serial->readAll());
ui->termial_textEdit->append(data.toHex());
}
}
the send value in RPM code:
#define CMD_SPEED_REF2 0x01
void MainWindow::on_speed_ref2_lineEdit_returnPressed()
{
uint8_t frame2[8];
frame2[0] = 0x55;
frame2[1] = 0xAA;
frame2[2] = CMD_SPEED_REF2;
float fdata2 = 0.0f;
fdata2 = ui->speed_ref2_lineEdit->text().toFloat();
uint8_t *data2 = new uint8_t();
data2 = (uint8_t*)&fdata2;
frame2[3] = data2[0];
frame2[4] = data2[1];
frame2[5] = data2[2];
frame2[6] = data2[3];
frame2[7] = frame2[2] ^ frame2[3] ^ frame2[4] ^ frame2[5] ^ frame2[6];
serial->write((char*)frame2, 8);
}
this Image Illustrate what happens:recived frame
I think your code mostly looks ok. The one area that looks very suspect is your conversion of the text/string back into binary.
Since you convert your binary into a string with:
ui->termial_textEdit->append(data.toHex());
You should in theory be able to use the following to convert it back:
// Convert back...
QByteArray binaryData = QByteArray::fromHex(ui->speed_ref2_lineEdit->text().toLatin1());
// Print to debug to check it...
qDebug("d1: %02x, d2: %02x...etc...\n", binaryData[0], binaryData[1]);
// or just
qDebug() << "data:" << binaryData.toHex() << endl;
Not on my qt PC until Monday so I can't verify this code, so there may be a bug in there somewhere... I'll check it on Monday!
For serial comms I always use QByteArray's instead of char/uint8_t arrays (when using Qt) because they are so easy to use. You can re-build your array like this:
QByteArray frame2;
frame2.append((char) 0x55); // not sure you need to cast it here
frame2.append((char) 0xAA);
frame2.append((char) CMD_SPEED_REF2);
:
etc
:
If you MUST send as a char * then just do:
serial->write(frame2.data(), 8);
//or
serial->write(frame2.data(), frame2.size()); // if you want to send the whole thing

How to read YUV8 data from avi file?

I have avi file that contains uncompressed gray video data. I need to extract frames from it. The size of file is 22 Gb.
How do i do that?
I have already tried ffmpeg, but it gives me "could not find codec parameters for video stream" message - because there is no codec at work, just frames.
Since Opencv just uses ffmpeg to read video, that rules out opencv as well.
The only path that seems to be left is to try and dig into the raw data, but i do not know how.
Edit: this is the code i use to read from the file with opencv. The failure occurs inside the second if. Running ffmpeg binary on the file also fails with the message above (could not find codec aprameters etc)
/* register all formats and codecs */
av_register_all();
/* open input file, and allocate format context */
if (avformat_open_input(&fmt_ctx, src_filename, NULL, NULL) < 0) {
fprintf(stderr, "Could not open source file %s\n", src_filename);
ret = 1;
goto end;
}
fmt_ctx->seek2any = true;
/* retrieve stream information */
int res = avformat_find_stream_info(fmt_ctx, NULL);
if (res < 0) {
fprintf(stderr, "Could not find stream information\n");
ret = 1;
goto end;
}
Edit:
Here is sample code i have tried to make the extraction: pastebin. The result i get is an unchanging buffer after every call to AVIStreamRead.
If you do not need cross platform functionality Video for Windows (VFW) API is a good alternative (http://msdn.microsoft.com/en-us/library/windows/desktop/dd756808(v=vs.85).aspx), i will not put an entire code block, since there's quite much to do, but you should be able to figure it out from the reference link. Basically, you do a AVIFileOpen, then get the video stream via AVIFileGetStream with streamtypeVIDEO, or alternatively do it at once with AVIStreamOpenFromFile and then read samples from the stream with AVIStreamRead. If you get to a point where you fail I can try to help, but it should be pretty straightforward.
Also, not sure why ffmpeg is failing, I have been doing raw AVI reading with ffmpeg without any codecs involved, can you post what call to ffpeg actually fails?
EDIT:
For the issue that you are seeing when the read data size is 0. The AVI file has N slots for frames in each second where N is the fps of the video. In real life the samples won't come exactly at that speed (e.g. IP surveillance cameras) so the actual data sample indexes can be non continuous like 1,5,11,... and VFW would insert empty samples between them (that is from where you read a sample with a zero size). What you have to do is call AVIStreamRead with NULL as buffer and 0 as size until the bRead is not 0 or you run past last sample. When you get an actual size, then you can again call AVIStreamRead on that sample index with the buffer pointer and size. I usually do compressed video so i don't use the suggested size, but at least according to your code snipplet I would do something like this:
...
bRead = 0;
do
{
aviOpRes = AVIStreamRead(ppavi,smpS,1,NULL,0,&bRead,&smpN);
} while (bRead == 0 && ++smpS < si.dwLength + si.dwStart);
if(smpS >= si.dwLength + si.dwStart)
break;
PUCHAR tempBuffer = new UCHAR[bRead];
aviOpRes = AVIStreamRead(ppavi,smpS,1,tempBuffer,bRead,&bRead,&smpN);
/* do whatever you need */
delete tempBuffer;
...
EDIT 2:
Since this may come in handy to someone or yourself to make a choice between VFW and FFMPEG I also updated your FFMPEG example so that it parsed the same file (sorry for the code quality since it lacks error checking but i guess you can see the logical flow):
/* register all formats and codecs */
av_register_all();
AVFormatContext* fmt_ctx = NULL;
/* open input file, and allocate format context */
const char *src_filename = "E:\\Output.avi";
if (avformat_open_input(&fmt_ctx, src_filename, NULL, NULL) < 0) {
fprintf(stderr, "Could not open source file %s\n", src_filename);
abort();
}
/* retrieve stream information */
int res = avformat_find_stream_info(fmt_ctx, NULL);
if (res < 0) {
fprintf(stderr, "Could not find stream information\n");
abort();
}
int video_stream_index = 0; /* video stream is usualy 0 but still better to lookup in case it's not present */
for(; video_stream_index < fmt_ctx->nb_streams; ++video_stream_index)
{
if(fmt_ctx->streams[video_stream_index]->codec->codec_type == AVMEDIA_TYPE_VIDEO)
break;
}
if(video_stream_index == fmt_ctx->nb_streams)
abort();
AVPacket *packet = new AVPacket;
while(av_read_frame(fmt_ctx, packet) == 0)
{
if (packet->stream_index == video_stream_index)
printf("Sample nr %d\n", packet->pts);
av_free_packet(packet);
}
Basically you open the context and read packets from it. You will get both audio and video packets so you should check if the packet belongs to the stream of interest. FFMPEG will save you the trouble with empty frames and give only those samples that have data in them.

Losing characters in TCP Telnet transmission

I'm using Winsock to send commands through Telnet ; but for some reason when I try to send a string, a few characters get dropped occasionally. I use send:
int SendData(const string & text)
{
send(hSocket,text.c_str(),static_cast<int>(text.size()),0);
Sleep(100);
send(hSocket,"\r",1,0);
Sleep(100);
return 0;
}
Any suggestions?
Update:
I checked and the error still occurs even if all the characters are sent. So I decided to change the Send function so that it sends individual characters and checks if they have been sent:
void SafeSend(const string &text)
{
char char_text[1];
for(size_t i = 0; i <text.size(); ++i)
{
char_text[0] = text[i];
while(send(hSocket,char_text,1,0) != 1);
}
}
Also, it drops characters in a peculiar way ; i.e. in the middle of the sentence. E.g.
set variable [fp]exit_flag = true
is sent as
ariable [fp]exit_flag = true
Or
set variable [fp]app_flag = true
is sent as
setrable [fp]app_flag = true
As mentioned in the comments you absolutely need to check the return value of send as it can return after sending only a part of your buffer.
You nearly always want to call send in a loop similar to the following (not tested as I don't have a Windows development environment available at the moment):
bool SendString(const std::string& text) {
int remaining = text.length();
const char* buf = text.data();
while (remaining > 0) {
int sent = send(hSocket, buf, remaining, 0);
if (sent == SOCKET_ERROR) {
/* Error occurred check WSAGetLastError() */
return false;
}
remaining -= sent;
buf += sent;
}
return true;
}
Update:
This is not relevant for the OP, but calls to recv should also structured in the same way as above.
To debug the problem further, Wireshark (or equivalent software) is excellent in tracking down the source of the problem.
Filter the packets you want to look at (it has lots of options) and check if they include what you think they include.
Also note that telnet is a protocol with numerous RFCs. Most of the time you can get away with just sending raw text, but it's not really guaranteed to work.
You mention that the windows telnet client sends different bytes from you, capture a minimal sequence from both clients and compare them. Use the RFCs to figure out what the other client does different and why. You can use "View -> Packet Bytes" to bring up the data of the packet and can easily inspect and copy/paste the hex dump.