Capturing a picture in c++ using v4l2, explaining the process - c++

I have been struggling with making a c++ code for capturing a picture from web-camera. I successfully did it, but I would like some clarification about the process i took.
So my code can be described in 6 steps, i will write them here along with my questions:
I. step: Initialize the device and set image format.
const char* dev_name = "/dev/video0";
int width=320;
int height=240;
int fd = v4l2_open(dev_name, O_RDWR | O_NONBLOCK, 0);
struct v4l2_format format = {0};
format.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
format.fmt.pix.width = width;
format.fmt.pix.height = height;
format.fmt.pix.pixelformat = V4L2_PIX_FMT_YUYV;//V4L2_PIX_FMT_YUYV //V4L2_PIX_FMT_RGB24
format.fmt.pix.field = V4L2_FIELD_NONE; //V4L2_FIELD_NONE
xioctl(fd, VIDIOC_S_FMT, &format);
II. step: Request for a buffer.
struct v4l2_requestbuffers req = {0};
req.count = 2;
req.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
req.memory = V4L2_MEMORY_MMAP;
xioctl(fd, VIDIOC_REQBUFS, &req);
question: What exactly does this "request a buffer" do?
question: Is this buffer physically located inside the camera or on my pc?
III. step: Query the buffer and mapping.
struct v4l2_buffer buf;
buffer* buffers;
unsigned int i;
buffers = (buffer*) calloc(req.count, sizeof(*buffers));
for (i = 0; i < req.count; i++) {
clear_memmory(&(buf));
(buf).type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
(buf).memory = V4L2_MEMORY_MMAP;
(buf).index = i;
xioctl(fd, VIDIOC_QUERYBUF, &buf);
buffers[i].length = (buf).length;
printf("A buff has a len of: %i\n",buffers[i].length);
buffers[i].start = v4l2_mmap(NULL, (buf).length, PROT_READ | PROT_WRITE, MAP_SHARED,fd, (buf).m.offset);
if (MAP_FAILED == buffers[i].start) {
perror("Can not map the buffers.");
exit(EXIT_FAILURE);
}
}
question: I understand that i do some mapping here, but can somebody explain why is this needed and what exactly is mapped where. To me it sounds like actual buffer is somewhere else and then i mapp it into my own buffer so i can read things from my buffer. Is this correct and where is the actual buffer?
question: Could i also avoid the mapping?
IV. step: Start streaming. Queue the buffer.
for (i = 0; i < 1; i++) {
clear_memmory(&(buf));
(buf).type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
(buf).memory = V4L2_MEMORY_MMAP;
(buf).index = i;
xioctl(fd,VIDIOC_QBUF, &(buf));
}
enum v4l2_buf_type type;
type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
xioctl(fd,VIDIOC_STREAMON, &type);
question: What does the VIDIOC_QBUF do? Please be very precise here and using mostly child words in answer to this since this is very confusing.
question: Should VIDIOC_QBUF happen after VIDIOC_STREAMON? Does it matter?
V. step: DeQueue the buffer and save a frame.
do {
FD_ZERO(&fds);
FD_SET(fd, &fds);
// Timeout.
tv.tv_sec = 2;
tv.tv_usec = 0;
r = select(fd + 1, &fds, NULL, NULL, &tv);
} while ((r == -1 && (errno = EINTR)));
if (r == -1) {
perror("select");
exit(EXIT_FAILURE);
}
clear_memmory(&(buf));
(buf).type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
(buf).memory = V4L2_MEMORY_MMAP;
xioctl(fd,VIDIOC_DQBUF, &(buf));
printf("Buff index: %i\n",(buf).index);
sprintf(out_name, "image%03d.ppm",pic_count);
fout = fopen(out_name, "w");
if (!fout) {
perror("Cannot open image");
exit(EXIT_FAILURE);
}
fwrite(buffers[(buf).index].start, (buf).bytesused, 1, fout);
fclose(fout);
pic_count++;
question: Why is VIDIOC_DQBUF needed and what does it do?
question: If i want to capture 3 frames and if i have only 1 buffer for 1 picture, do i need to queue, dequeue 3 times?
question: If i want 1000000 pictures, can I make that buffer big enough to hold them all? What is limiting the size of that buffer? Is that buffer on the camera and that means all this pictures would be sitting in it?
question: If i want 10 pictures taken at moments I chose, and if i go with making a buffer large enough for 10 pictures. What should i call at the moment i want to take a picture? only VIDIOC_QBUF? VIDIOC_QBUF and VIDIOC_DQBUF?, only VIDIOC_DQBUF?
Please don't point me to https://01.org/linuxgraphics/gfx-docs/drm/media/uapi/v4l/vidioc-qbuf.html or other sites, as I have read everything i was able to found and I am still left with above unclarity. I really want detailed explanations about these questions. I thank you in advance for all helpful answers.

Related

Capturing YUYV in c++ using v4l2

I have a webcam connected to beaglebone via usb. I am coding in c++ and my goal is to capture raw UNCOMPRESSED picture from the webcam.
Firstly i checked what formats are supported via command v4l2-ctl --list-formats and the result was:
Index : 0
Type : Video Capture
Pixel Format: 'MJPG' (compressed)
Name : Motion-JPEG
Index : 1
Type : Video Capture
Pixel Format: 'YUYV'
Name : YUYV 4:2:2
So from this I assume it has to be possible to get an uncompressed picture if i try to use YUYV format.
Knowing this I started writing a program in c++. I successfully written a program to capture a compressed picture, but when trying to capture using format YUYV it doesnt work and i really need some help to get this done.
Here is my code:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <fcntl.h>
#include <errno.h>
#include <sys/ioctl.h>
#include <sys/types.h>
#include <sys/time.h>
#include <sys/mman.h>
#include <linux/videodev2.h>
#include <libv4l2.h>
template <typename typeXX>
void clear_memmory(typeXX* x) {
memset(x, 0, sizeof(*x));
}
void xioctl(int cd, int request, void *arg){
int response;
do{
//ensures we get the correct response.
response = v4l2_ioctl(cd, request, arg);
}
while (response == -1 && ((errno == EINTR) || (errno == EAGAIN)));
if (response == -1) {
fprintf(stderr, "error %d, %s\n", errno, strerror(errno));
exit(EXIT_FAILURE);
}
}
struct LMSBBB_buffer{
void* start;
size_t length;
};
int main(){
const char* dev_name = "/dev/video0";
int width=1920;
int height=1080;
int fd = v4l2_open(dev_name, O_RDWR | O_NONBLOCK, 0);
struct v4l2_format format = {0};
format.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
format.fmt.pix.width = width;
format.fmt.pix.height = height;
format.fmt.pix.pixelformat = V4L2_PIX_FMT_RGB24;//V4L2_PIX_FMT_YUYV //V4L2_PIX_FMT_RGB24
format.fmt.pix.field = V4L2_FIELD_NONE; //V4L2_FIELD_NONE
xioctl(fd, VIDIOC_S_FMT, &format);
printf("Device initialized.\n");
///request buffers
struct v4l2_requestbuffers req = {0};
req.count = 2;
req.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
req.memory = V4L2_MEMORY_MMAP;
xioctl(fd, VIDIOC_REQBUFS, &req);
printf("Buffers requested.\n");
///mapping buffers
struct v4l2_buffer buf;
LMSBBB_buffer* buffers;
unsigned int i;
buffers = (LMSBBB_buffer*) calloc(req.count, sizeof(*buffers));
for (i = 0; i < req.count; i++) {
clear_memmory(&(buf));
(buf).type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
(buf).memory = V4L2_MEMORY_MMAP;
(buf).index = i;
xioctl(fd, VIDIOC_QUERYBUF, &buf);
buffers[i].length = (buf).length;
printf("A buff has a len of: %i\n",buffers[i].length);
buffers[i].start = v4l2_mmap(NULL, (buf).length, PROT_READ | PROT_WRITE, MAP_SHARED,fd, (buf).m.offset);
if (MAP_FAILED == buffers[i].start) {
perror("Can not map the buffers.");
exit(EXIT_FAILURE);
}
}
printf("Buffers mapped.\n");
for (i = 0; i < req.count; i++) {
clear_memmory(&(buf));
(buf).type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
(buf).memory = V4L2_MEMORY_MMAP;
(buf).index = i;
ioctl(fd,VIDIOC_QBUF, &(buf));
}
enum v4l2_buf_type type;
type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
ioctl(fd,VIDIOC_STREAMON, &type);
printf("buffers queued and streaming.\n");
int pic_count=0;
///CAPTURE
fd_set fds;
struct timeval tv;
int r;
char out_name[256];
FILE* fout;
do {
FD_ZERO(&fds);
FD_SET(fd, &fds);
// Timeout.
tv.tv_sec = 2;
tv.tv_usec = 0;
r = select(fd + 1, &fds, NULL, NULL, &tv);
} while ((r == -1 && (errno = EINTR)));
if (r == -1) {
perror("select");
exit(EXIT_FAILURE);
}
clear_memmory(&(buf));
(buf).type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
(buf).memory = V4L2_MEMORY_MMAP;
xioctl(fd,VIDIOC_DQBUF, &(buf));
printf("Buff index: %i\n",(buf).index);
sprintf(out_name, "image%03d.ppm",pic_count);
fout = fopen(out_name, "w");
if (!fout) {
perror("Cannot open image");
exit(EXIT_FAILURE);
}
fprintf(fout, "P6\n%d %d 255\n",width, height);
fwrite(buffers[(buf).index].start, (buf).bytesused, 1, fout);
fclose(fout);
pic_count++;
clear_memmory(&(buf));
(buf).type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
(buf).memory = V4L2_MEMORY_MMAP;
xioctl(fd,VIDIOC_DQBUF, &(buf));
printf("Buff index: %i\n",(buf).index);
sprintf(out_name, "image%03d.ppm",pic_count);
fout = fopen(out_name, "w");
if (!fout) {
perror("Cannot open image");
exit(EXIT_FAILURE);
}
fprintf(fout, "P6\n%d %d 255\n",width, height);
fwrite(buffers[(buf).index].start, (buf).bytesused, 1, fout);
fclose(fout);
pic_count++;
///xioctl(fd,VIDIOC_QBUF, &(buf));
return 0;
}
in line 50, i can choose the format between V4L2_PIX_FMT_YUYV and V4L2_PIX_FMT_RGB24.
for V4L2_PIX_FMT_RGB24 i get the picture, but when using V4L2_PIX_FMT_YUYV I get this error:
libv4l2: error dequeuing buf: Resource temporarily unavailable
libv4l2: error dequeuing buf: Resource temporarily unavailable
libv4l2: error dequeuing buf: Resource temporarily unavailable
libv4l2: error dequeuing buf: Resource temporarily unavailable
libv4l2: error dequeuing buf: Resource temporarily unavailable
the error lines goes for ever until i end the program manually.
Does anyone have an idea what to do? I spent over 2 weeks on this and i can't move anywhere from here. I would really appreciate any advice.
From what I see you are requesting a FullHD (1920x1080) buffer in YUYV format from a camera. You did not mention the camera type/model/specs, but if it is a generic USB-attached hardware most likely you will not get a raw FullHD YUYV buffer as an output, only the MJPEG one (which you can decode to YUV, if you hack around with libjpeg) or the decoded RGB buffer (which is pretty much the decoded MJPEG with YUV->RGB conversion) which is not mmapped.
The exact list of formats with framerates can be requested by this command, which would probably tell you it does not provide a 1920x1080 YUYV, only something smaller, like 640x480:
v4l2-ctl --list-formats
If you need video processing with "true" zero-copy access to raw YUYV camera frames, you need direct access to hardware and that specific hardware in the first place. Once you have the USB interface between your software and the camera, you get an extra indirection and that means the speed goes down. Think for a moment, the YUYV frame at 1920x1080 takes up approximately 4 Megabytes of memory. At 30 FPS this is 120 Megabytes (or 960 Megabits) per second of bus throughput. If you have a USB2.0 camera, there is just no bandwidth to support this (thus the need for MJPEG). Even at 15FPS this is 480 Megabits, not counting the USB latency and protocol overhead.
To provide some "actionable feedback" I would advice to first concentrate on the algorithms (probably, you just don't want to loose the processing speed at the very first step) which you want to apply to the image. Don't hesitate to use OpenCV for camera input and basic image processing, later you can switch to some hardware interface and hand-written algorithms.
The easier way of getting raw frames would be to use Android's camera interface and try to process the incoming frames with GLSL shaders using the GL_TEXTURE_EXTERNAL_OES extension, about which there information and code samples available. There you can connect GL textures to AHardwareBuffer instances and then use AHardwareBuffer_lock function to get raw pointers. The exact supported formats also may vary across the hardware, so do not expect this to be super-easy.
I've recently had a similar issue. In my case the camera driver needed the VIDIOC_S_PARM ioctl in order to set the frame rate and initialize the camera for the selected capture mode.
You can try to add this code after the VIDIOC_S_FMT and see if it works for you as well:
struct v4l2_streamparm streamparam;
memset(&streamparam, 0, sizeof(streamparam));
streamparam.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
xioctl(fd, VIDIOC_G_PARM, &streamparam);
streamparam.parm.capture.timeperframe.numerator = 1;
streamparam.parm.capture.timeperframe.denominator = 5;
xioctl(fd, VIDIOC_S_PARM, &streamparam);

How to read h264 stream as a file from the USB webcam directly in c/c++ without using opencv?

I am able to read a video file of h264 format and doing some machine learning inference on top of it. The code works absolutely fine for input from a file. Below code is a sample code from Deepstream SDK
FileDataProvider(const char *szFilePath, simplelogger::Logger *logger)
: logger_(logger)
{
fp_ = fopen(szFilePath, "rb");
//fp_ = fopen("/dev/video0", "rb");
if (nullptr == fp_) {
LOG_ERROR(logger, "Failed to open file " << szFilePath);
exit(1);
}
pLoadBuf_ = new uint8_t[nLoadBuf_];
pPktBuf_ = new uint8_t[nPktBuf_];
assert(nullptr != pLoadBuf_);
}
~FileDataProvider() {
if (fp_) {
fclose(fp_);
}
if (pLoadBuf_) {
delete [] pLoadBuf_;
}
if (pPktBuf_) {
delete [] pPktBuf_;
}
}
What is requirement ?
Read from the Logitech c920 webcam instead for video file.
I know How to read from webcam using opencv. But I don't want to use opencv here.
My Research
Using v4l we can get the stream and display it in vlc.
Camera supports below formats.
#ubox:~$ v4l2-ctl --device=/dev/video1 --list-formats
ioctl: VIDIOC_ENUM_FMT Index : 0 Type : Video Capture
Pixel Format: 'YUYV' Name : YUYV 4:2:2
Index : 1 Type : Video Capture Pixel Format: 'H264'
(compressed) Name : H.264
Index : 2 Type : Video Capture Pixel Format: 'MJPG'
(compressed) Name : Motion-JPEG
Reading output of a USB webcam in Linux
vlc v4l2:///dev/video1 --v4l2-chroma=h264 - this displays the video from the webcam.
How to do this?
- Now how to feed this live stream into
above sample code such that it reads from the webcam rather than file?
[update-1]
- In otherwords, does v4l has some options to write the video stream as h264 formant ? So that, I can read that file like before(above code) when its(v4l) writing to disk.
[update-2]
- we can use ffmpeg instead of v4l. If any solutions for using ffmpeg to save the video stream into disk continuously, so that other programs reads that file ?
Before using ioctl to capture frames from camera, you need to set the format like below first.
fp_ = open("/dev/video0", O_RDWR);
struct v4l2_format fmt = {0};
fmt.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
fmt.fmt.pix.pixelformat = V4L2_PIX_FMT_H264;
ioctl(fp_, VIDIOC_S_FMT, &fmt);
then, initialize and map buffer
struct Buffer
{
void *start;
unsigned int length;
unsigned int flags;
};
int buffer_count_ = 4;
Buffer *buffers_;
bool AllocateBuffer()
{
struct v4l2_requestbuffers req = {0};
req.count = buffer_count_;
req.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
req.memory = V4L2_MEMORY_MMAP;
if (ioctl(fp_, VIDIOC_REQBUFS, &req) < 0)
{
perror("ioctl Requesting Buffer");
return false;
}
buffers_ = new Buffer[buffer_count_];
for (int i = 0; i < buffer_count_; i++)
{
struct v4l2_buffer buf = {0};
buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
buf.memory = V4L2_MEMORY_MMAP;
buf.index = i;
if (ioctl(fp_, VIDIOC_QUERYBUF, &buf) < 0)
{
perror("ioctl Querying Buffer");
return false;
}
buffers_[i].start = mmap(NULL, buf.length, PROT_READ | PROT_WRITE, MAP_SHARED, fd_, buf.m.offset);
buffers_[i].length = buf.length;
if (MAP_FAILED == buffers_[i].start)
{
printf("MAP FAILED: %d\n", i);
for (int j = 0; j < i; j++)
munmap(buffers_[j].start, buffers_[j].length);
return false;
}
if (ioctl(fp_, VIDIOC_QBUF, &buf) < 0)
{
perror("ioctl Queue Buffer");
return false;
}
}
return true;
}
STREAMON to start capturing
v4l2_buf_type type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
ioctl(fp_, VIDIOC_STREAMON, &type);
finally read a frame from the mapped buffer. Generally, CaptureImage() will be in the while loop.
Buffer CaptureImage()
{
fd_set fds;
FD_ZERO(&fds);
FD_SET(fd_, &fds);
struct timeval tv = {0};
tv.tv_sec = 1;
tv.tv_usec = 0;
int r = select(fd_ + 1, &fds, NULL, NULL, &tv);
if (r == 0)
{
// timeout
}
struct v4l2_buffer buf = {0};
buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
buf.memory = V4L2_MEMORY_MMAP;
while (ioctl(fp_, VIDIOC_DQBUF, &buf) < 0)
{
perror("Retrieving Frame");
}
struct Buffer buffer = {.start = buffers_[buf.index].start,
.length = buf.bytesused,
.flags = buf.flags};
if (ioctl(fp_, VIDIOC_QBUF, &buf) < 0)
{
perror("Queue buffer");
}
return buffer;
}

Is it possible to get good FPS using Raspberry Pi camera v4l2 in c++?

I'm trying to stream video on a Raspberry Pi using the official V4L2 driver with the Raspberry Pi camera, from C++ on raspbian (2015-02 release), and I'm having low FPS issues.
Currently I'm just creating a window and copying the buffer to the screen (which takes about 30ms) whereas the select() takes about 140ms (for a total of 5-6 fps). I also tried sleeping for 100ms and it decreases the select() time by a similar amount (resulting in the same fps). CPU load is about 5-15%.
I also tried changing the driver fps from console (or system()) but it only works downwards (for example, if I set the driver fps to 1fps, I'll get 1fps but if I set it to 90fps I still get 5-6fps, even though the driver confirms setting it to 90fps).
Also, when querying FPS modes for the used resolution I get 90fps.
I included the parts of the code related to V4L2 (code omitted between different parts) :
//////////////////
// Open device
//////////////////
mFD = open(mDevName, O_RDWR | O_NONBLOCK, 0);
if (mFD == -1) ErrnoExit("Open device failed");
//////////////////
// Setup format
//////////////////
struct v4l2_format fmt;
memset(&fmt, 0, sizeof(fmt));
fmt.fmt.pix.pixelformat = V4L2_PIX_FMT_YUYV;
fmt.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
Xioctl(VIDIOC_G_FMT, &fmt);
mImgWidth = fmt.fmt.pix.width;
mImgHeight = fmt.fmt.pix.height;
cout << "width=" << mImgWidth << " height=" << mImgHeight << "\nbytesperline=" << fmt.fmt.pix.bytesperline << " sizeimage=" << fmt.fmt.pix.sizeimage << "\n";
// For some reason querying the format always sets pixelformat to JPEG
// no matter the input, so set it back to YUYV
fmt.fmt.pix.pixelformat = V4L2_PIX_FMT_YUYV;
if (Xioctl(VIDIOC_S_FMT, &fmt) == -1)
{
cout << "Set video format failed : " << strerror(errno) << "\n";
}
//////////////////
// Setup streaming
//////////////////
struct v4l2_requestbuffers req;
memset(&req, 0, sizeof(req));
req.count = 20;
req.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
req.memory = V4L2_MEMORY_MMAP;
if (-1 == Xioctl(VIDIOC_REQBUFS, &req))
{
ErrnoExit("Reqbufs");
}
if (req.count < 2)
throw "Not enough buffer memory !";
mNBuffers = req.count;
mBuffers = new CBuffer[mNBuffers];
if (!mBuffers) throw "Out of memory !";
for (unsigned int i = 0; i < mNBuffers; i++)
{
struct v4l2_buffer buf;
memset(&buf, 0, sizeof(buf));
buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
buf.memory = V4L2_MEMORY_MMAP;
buf.index = i;
if (-1 == Xioctl(VIDIOC_QUERYBUF, &buf))
ErrnoExit("Querybuf");
mBuffers[i].mLength = buf.length;
mBuffers[i].pStart = mmap(NULL, buf.length, PROT_READ | PROT_WRITE, MAP_SHARED, mFD, buf.m.offset);
if (mBuffers[i].pStart == MAP_FAILED)
ErrnoExit("mmap");
}
//////////////////
// Start streaming
//////////////////
unsigned int i;
enum v4l2_buf_type type;
struct v4l2_buffer buf;
for (i = 0; i < mNBuffers; i++)
{
memset(&buf, 0, sizeof(buf));
buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
buf.memory = V4L2_MEMORY_MMAP;
buf.index = i;
if (-1 == Xioctl(VIDIOC_QBUF, &buf))
ErrnoExit("QBUF");
}
type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
if (-1==Xioctl(VIDIOC_STREAMON, &type))
ErrnoExit("STREAMON");
And the last two parts in the main loop :
//////////////////
// Get frame
//////////////////
FD_ZERO(&fds);
FD_SET(mFD, &fds);
tv.tv_sec = 3;
tv.tv_usec = 0;
struct timespec t0, t1;
clock_gettime(CLOCK_REALTIME, &t0);
// This line takes about 140ms which I don't get
r = select(mFD + 1, &fds, NULL, NULL, &tv);
clock_gettime(CLOCK_REALTIME, &t1);
cout << "select time : " << ((float)(t1.tv_sec - t0.tv_sec))*1000.0f + ((float)(t1.tv_nsec - t0.tv_nsec))/1000000.0f << "\n";
if (-1 == r)
{
if (EINTR == errno)
continue;
ErrnoExit("select");
}
if (r == 0)
throw "Select timeout\n";
// Read the frame
//~ struct v4l2_buffer buf;
memset(&mCurBuf, 0, sizeof(mCurBuf));
mCurBuf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
mCurBuf.memory = V4L2_MEMORY_MMAP;
// DQBUF about 2ms
if (-1 == Xioctl(VIDIOC_DQBUF, &mCurBuf))
{
if (errno == EAGAIN) continue;
ErrnoExit("DQBUF");
}
clock_gettime(CLOCK_REALTIME, &mCaptureTime);
// Manage frame in mBuffers[buf.index]
mCurBufIndex = mCurBuf.index;
break;
}
//////////////////
// Release frame
//////////////////
if (-1 == Xioctl(VIDIOC_QBUF, &mCurBuf))
ErrnoExit("VIDIOC_QBUF during mainloop");
I have been looking at the various methods of using the picamera and I'm hardly an expert, but it would seem that the default camera settings are what's holding you back. There are many modes and switches. I don't know if they are exposed through ioctls or how yet, I just started. But I had to use a program called v4l-ctl to get things ready for the mode I wanted. A deep look at that source and some code lifting should let you achieve greatness. Oh, and I doubt the select call is an issue, it's simply waiting on the descriptor which is slow to become readable. Depending on mode, etc. there can be a mandatory wait for autoexposure, etc.
Edit: I meant to say "a default setting" as you've changed some. There are also rules not codified in the driver.
The pixelformat matters. I encountered the similar low-fps problem, and I spent some time testing using my program in Go and C++ using V4L2 API. What I found is, Rpi Cam Module has good accelaration with H.264/MJPG pixelformat. I can easily get 60fps at 640*480, same as non-compressed formats like YUYV/RGB. However JPEG runs very slow. I can only get 4fps even at 320*240. And I also found the current is higher (>700mA) with JPEG compare to 500mA with H.264/MJPG.

WaveOutWrite callback creates choppy audio

I have four buffers that I am using for audio playback in a synthesizer. I submit two buffers initially, and then in the callback routine I write data into the next buffer and then submit that buffer.
When I generate each buffer I'm just putting a sine wave into it whose period is a multiple of the buffer length.
When I execute I hear brief pauses between each buffer. I've increased the buffer size to 16K samples at 44100 Hz so I can clearly hear that the whole buffer is playing, but there is an interruption between each.
What I think is happening is that the callback function is only called when ALL buffers that have been written are complete. I need the synthesis to stay ahead of the playback so I need a callback when each buffer is completed.
How do people usually solve this problem?
Update: I've been asked to add code. Here's what I have:
First I connect to the WaveOut device:
// Always grab the mapped wav device.
UINT deviceId = WAVE_MAPPER;
// This is an excelent tutorial:
// http://planet-source-code.com/vb/scripts/ShowCode.asp?txtCodeId=4422&lngWId=3
WAVEFORMATEX wfx;
wfx.nSamplesPerSec = 44100;
wfx.wBitsPerSample = 16;
wfx.nChannels = 1;
wfx.cbSize = 0;
wfx.wFormatTag = WAVE_FORMAT_PCM;
wfx.nBlockAlign = (wfx.wBitsPerSample >> 3) * wfx.nChannels;
wfx.nAvgBytesPerSec = wfx.nBlockAlign * wfx.nSamplesPerSec;
_waveChangeEventHandle = CreateMutex(NULL,false,NULL);
MMRESULT res;
res = waveOutOpen(&_wo, deviceId, &wfx, (DWORD_PTR)WavCallback,
(DWORD_PTR)this, CALLBACK_FUNCTION);
I initialize the four frames I'll be using:
for (int i=0; i<_numFrames; ++i)
{
WAVEHDR *header = _outputFrames+i;
ZeroMemory(header, sizeof(WAVEHDR));
// Block size is in bytes. We have 2 bytes per sample.
header->dwBufferLength = _codeSpec->OutputNumSamples*2;
header->lpData = (LPSTR)malloc(2 * _codeSpec->OutputNumSamples);
ZeroMemory(header->lpData, 2*_codeSpec->OutputNumSamples);
res = waveOutPrepareHeader(_wo, header, sizeof(WAVEHDR));
if (res != MMSYSERR_NOERROR)
{
printf("Error preparing header: %d\n", res - MMSYSERR_BASE);
}
}
SubmitBuffer();
SubmitBuffer();
Here is the SubmitBuffer code:
void Vodec::SubmitBuffer()
{
WAVEHDR *header = _outputFrames+_curFrame;
MMRESULT res;
res = waveOutWrite(_wo, header, sizeof(WAVEHDR));
if (res != MMSYSERR_NOERROR)
{
if (res = WAVERR_STILLPLAYING)
{
printf("Cannot write when still playing.");
}
else
{
printf("Error calling waveOutWrite: %d\n", res-WAVERR_BASE);
}
}
_curFrame = (_curFrame+1)&0x3;
if (_pointQueue != NULL)
{
RenderQueue();
_nextFrame = (_nextFrame + 1) & 0x3;
}
}
And here is my callback code:
void CALLBACK Vodec::WavCallback(HWAVEOUT hWaveOut,
UINT uMsg,
DWORD dwInstance,
DWORD dwParam1,
DWORD dwParam2 )
{
// Only listen for end of block messages.
if(uMsg != WOM_DONE) return;
Vodec *instance = (Vodec *)dwInstance;
instance->SubmitBuffer();
}
The RenderQueue code is pretty simple - just copies a piece of a template buffer into the output buffer:
void Vodec::RenderQueue()
{
double white = _pointQueue->White;
white = 10.0; // For now just override with a constant value
int numSamples = _codeSpec->OutputNumSamples;
signed short int *data = (signed short int *)_outputFrames[_nextFrame].lpData;
for (int i=0; i<numSamples; ++i)
{
Sample x = white * _noise->Samples[i];
data[i] = (signed short int)(x);
}
_sampleOffset += numSamples;
if (_sampleOffset >= _pointQueue->DurationInSamples)
{
_sampleOffset = 0;
_pointQueue = _pointQueue->next;
}
}
UPDATE: Mostly solved the issue. I need to increment _nextFrame along with _curFrame (not conditionally). The playback buffer was getting ahead of the writing buffer.
However, when I decrease the playback buffer to 1024 samples, it gets choppy again. At 2048 samples it is clear. This happens for both Debug and Release builds.
1024 samples is just about 23ms of audio data. wav is pretty high level API from Windows Vista onwards. If you want low-latency audio playback, you should use CoreAudio. You can get latencies down to 10 ms in shared mode and 3 ms in exclusive mode. Also, the audio depends upon the processes currently running on your system. In other words, it depends on how frequently your audio thread can run to get data. You should also look at MultiMedia Class Scheduler Service and AvSetMmThreadCharacteristics function.

Serial Port communication with Arduino and C++

I am having a problem with a Serial Port communication between Arduino Nano and C++, even though the problem is in C++ side. Basically I want to send integers (or long,...) from the Arduino to a C++ program to be processed.
First I did a test sending information from the Arduino to the computer using Matlab. The Arduino code is pretty simple:
int i = 0;
void setup() {
// start serial port at 9600 bps:
Serial.begin(9600);
establishContact();
}
void loop() {
Serial.println(i);
i=i+1;
delay(10);
}
void establishContact() {
while (Serial.available() <= 0) {
Serial.println('A', BYTE);
delay(10);
}
}
The Matlab side is also simple:
clc;
clear all;
numSec=2;
t=[];
v=[];
s1 = serial('COM3'); % define serial port
s1.BaudRate=9600; % define baud rate
set(s1, 'terminator', 'LF'); % define the terminator for println
fopen(s1);
try % use try catch to ensure fclose
% signal the arduino to start collection
w=fscanf(s1,'%s'); % must define the input % d or %s, etc.
if (w=='A')
display(['Collecting data']);
fprintf(s1,'%s\n','A'); % establishContact just wants
% something in the buffer
end
i=0;
t0=tic;
while (toc(t0)<=numSec)
i=i+1;
t(i)=toc(t0);
t(i)=t(i)-t(1);
v(i)=fscanf(s1,'%d');
end
fclose(s1);
plot(t,v,'*r')
catch me
fclose(s1);
end
My goal is, with C++, do the same that is done in Matlab using fscanf(s1, '%d').
Here is the current code that I am using (C++ code):
void main()
{
HANDLE hSerial;
hSerial = CreateFile(TEXT("COM3"),
GENERIC_READ | GENERIC_WRITE,
0,
NULL,
OPEN_EXISTING,
FILE_ATTRIBUTE_NORMAL,//FILE_FLAG_OVERLAPPED,
NULL);
if ( hSerial == INVALID_HANDLE_VALUE)
{
printf("Error initializing handler");
}
else
{
// Set the parameters of the handler to the serial port.
DCB dcb = {0};
dcb.DCBlength = sizeof(dcb);
if ( !GetCommState(hSerial, &dcb) )
{
printf("Error setting parameters");
}
FillMemory(&dcb, sizeof(dcb), 0);
dcb.BaudRate = CBR_9600;
dcb.ByteSize = 8;
dcb.StopBits = ONESTOPBIT;
dcb.Parity = NOPARITY;
if ( !SetCommState(hSerial, &dcb) )
{
// error setting serial port state.
}
// Tell the program not to wait for data to show up
COMMTIMEOUTS timeouts = {0};
timeouts.ReadIntervalTimeout = 0;//20;
timeouts.ReadTotalTimeoutConstant = 0;//20;
timeouts.ReadTotalTimeoutMultiplier = 0;//50;
timeouts.WriteTotalTimeoutConstant = 0;//100;
timeouts.WriteTotalTimeoutMultiplier = 0;//100;
if ( !SetCommTimeouts(hSerial, &timeouts) )
{
printf("Error setting the timeouts");
}
char szBuff[5] = "";
DWORD dwBytesRead = 0;
int i = 0;
char test[] = "B\n";
int maxSamples = 10;
DWORD dwCommStatus;
WriteFile(hSerial, test, 2, &dwBytesRead, NULL);
SetCommMask(hSerial,EV_RXCHAR);
while (i < maxSamples)
{
WaitCommEvent (hSerial, &dwCommStatus, 0);
if (dwCommStatus & EV_RXCHAR)
{
memset(szBuff,0,sizeof(szBuff));
ReadFile(hSerial, LPVOID(szBuff), 4, &dwBytesRead, NULL);
cout<<szBuff;
printf(" - %d - \n", atoi(szBuff));
}
i++;
}
scanf("%d", &i);
CloseHandle(hSerial);
}
}
The goal of my code would be something like num = ReadSerialCOM(hSerial, "%d");
My current C++ code reads the information from the buffer, but there is not an accepted end of line, which implies that my numbers (integers) are received cut.
Eg:
I send 8889 from the Arduino, which places it in the COM port. And the command ReadFile saves '88' into szBuff. At the next iteration '89\n' is saved into sZBuff. Basically I want to avoid to post-process sZBuff to concat '88' and '89\n'.
Anyone?
Thanks!
If I understand your question correctly, one way to avoid having to 'post-process' is to move the pointer passed to ReadFile to the end of the available data, so the ReadFile call is appending to the buffer, instead of overwriting.
Essentially, you would have two pointers. One to the buffer, the other to the end of the data in the buffer. So when your program starts, both pointers will be the same. Now, you read the first 2 bytes. You increment the end-of-data pointer by 2. You do another read, but instead of szBuff, you pass a pointer to the end of the previously read data. You read the next three bytes and you have the complete entry in szBuff.
If you need to wait until some delimiter to mark the end of an entry is received, you could just search the received data for it. If it's not there, you keep reading until you find it. If it is there, you can just return.
// Fill the buffer with 0
char szBuff[256] = {0};
// We have no data in the buffer, so the end of data points to the beginning
// of the buffer.
char* szEndOfData = szBuff;
while (i < maxSamples)
{
WaitCommEvent (hSerial, &dwCommStatus, 0);
if (dwCommStatus & EV_RXCHAR)
{
// Append up to 4 bytes from the serial port to the buffer
ReadFile(hSerial, LPVOID(szEndOfData), 4, &dwBytesRead, NULL);
// Increment the end of data pointer, so it points to the end of the
// data available in the buffer.
szEndOfData += dwBytesRead;
cout<<szBuff;
printf(" - %d - \n", atoi(szBuff));
}
i++;
}
// Output, assuming what you mentioned happens:
// - 88 -
// - 8889 -
If this approach is acceptable to you, it will require a bit more work. For example, you would have to ensure you don't overflow your buffer. When you remove data from the buffer, you'll have to move all of the data after the removed segment to the beginning, and fix the end of data pointer. Alternatively, you could use a circular buffer.
As Hans Passant and dauphic pointed, it doesn't seem to be a general solution for my question. I am writing, though, the code that I was trying to avoid, just in case somebody finds it useful or face the same problem that I had:
int i = 0;
DWORD dwBytesRead = 0;
DWORD dwCommStatus = 0;
char szBuff[2] = "";
int maxRead = 20;
int sizeNum = 1;
int *num = (int*)malloc(maxRead*sizeof(int));
char *currNum;
char *pastNum;
// Write something into the Serial Port to start receive
// information from the Arduino
WriteFile(hSerial, (LPCVOID)"A\0", 1, &dwBytesRead, NULL);
SetCommMask(hSerial, EV_RXCHAR);
// Start reading from the Serial Port
while ( i < maxRead )
{
WaitCommEvent (hSerial, &dwCommStatus, 0);
if (dwCommStatus & EV_RXCHAR) // if a char is received in the serial port
{
ReadFile(hSerial, LPVOID(szBuff), 1, &dwBytesRead, NULL);
if ( szBuff[0] > 47 && szBuff[0] < 58 )
{
sizeNum++;
if (sizeNum ==2)
{
currNum = (char*)malloc(sizeNum*sizeof(char));
strcpy(currNum, szBuff);
} else
{
if (pastNum != NULL)
free(pastNum);
pastNum = currNum;
currNum = (char*)malloc(sizeNum*sizeof(char));
strcpy(currNum, pastNum);
strcpy(currNum+(sizeNum-2)*sizeof(char), szBuff);
}
cout << szBuff<<endl;
} else if (szBuff[0] == '\n' && sizeNum > 1) // end of number
{
num[i] = atoi(currNum);
i++;
sizeNum = 1;
if (currNum!=NULL)
free(currNum);
}
}
}