Processing Video as a string of images - c++

I have a project where part of it is processing a video as a batch of images. I found a snippet online which helped me process a batch of images(the one attached below)
string Img_seq[] = {"flying_plane"};
int size_of_Img_seq = sizeof(Img_seq) / sizeof(Img_seq[0]);
imgpath = argv[1];
if (argc >= 3)
{
TrackingResPath = argv[2];
cout << "The image path is: " << imgpath << endl;
int Seg_num = 1; //for OPE
for (int j = 0; j < size_of_Img_seq; ++j)
{
folder = imgpath + Img_seq[j] + "/*.jpg";
string imgInFormat, imgOutFormat, imgOutFormat1, imgOutFormat2;
ofstream outBB;
imgInFormat = imgpath + Img_seq[j] + "/%05d.jpg";
if (argc >= 3)
{
//add process here
}
}
// get the total number of images in folder
glob(folder, filenames);
int Frames_per_seg = filenames.size(); // for OPE
cout << "Frames_per_seg = " << Frames_per_seg << endl;
for (int k = 1; k <= Seg_num; ++k)
{
sprintf(imgInPath, imgInFormat.c_str(), startFrame);// read image path
sprintf(imgOutPath, imgOutFormat.c_str(), startFrame);
}
I need to use this procedure to process a video from either a directory or from the webcam. I tried the one way which I've given below but it is spitting out errors
VideoCapture capture(videoFilename);
if(!capture.isOpened()){
cerr << "Unable to open video file:" << videoFilename << endl;
exit(EXIT_FAILURE);
}
while( (char)keyboard != 'q' && (char)keyboard != 27){
if(!capture.read(frame)){
cerr << "Unable to read next frame" << endl;
cerr << "Exiting" << endl;
exit(EXIT_FAILURE);
}}
stringstream ss;
rectangle(frame, cv::Point(10, 2), cv::Point(100, 20),
cv::Scalar(255, 255, 255), -1);
ss << capture.get(CV_CAP_PROP_POS_FRAMES);
string Img_seq[] = ss.str();
keyboard = waitKey( 30 );
The errors are basically because Img_seq[] is not linked properly between the above two codes.
Could anyone help me understand how I should proceed in using video for a batch processing logic.

Related

Synchronizing multiple RTSP streams with Cudacodec VideoReader?

Camera broadcast comes from 4 different fpga's with rtsp. I'm starting to get the broadcasts by connecting to all of them from different ip's. I made myself an ntp server for synchronization and connected FPGA's to this server and they all use the same time. However, when the connections come and I start pressing the screen, they are not synchronous. I can take any time stamp and sync using buffer. But I was not able to get any timestamp with videoreader. How can I achieve synchronization?
Ubuntu 20.04, opencv 4.5.3, QT creator, c++
Note: I have to use VideoReader. I cannot decode videos that come with another library.
void Widget::on_StartStream_clicked(){
cv::Ptr<cv::cudacodec::VideoReader> videoReader[4];
namedWindow("OpenGL Output", WINDOW_OPENGL);
setWindowProperty("OpenGL Output", WND_PROP_FULLSCREEN, WINDOW_FULLSCREEN);
cv::cuda::GpuMat MergedOutput(3120 * 2, 4208 * 2, CV_8UC4);
try{
int index = 0;
int i = 0;
for(int i = 1; i < 8; i+=2){
string rtspPathFpga = "rtsp://172.16.1.10" + to_string(i) + ":900" + to_string(i) + "/camera_1";
cout << "i: " << i << endl;
videoReader[index++] = cv::cudacodec::createVideoReader(rtspPathFpga);
}
while (true) {
tryReconnection:
i = 0;
try {
do{
videoReader[i]->nextFrame(videoFrames[i]);
i++;
}while(i < 4);
} catch (exception ex) {
cout << "Try Reconnection for: " << ((i*2) + 1) << endl;
string rtspPathFpga = "rtsp://172.16.1.10" + to_string((i*2) + 1) + ":900" + to_string((i*2) + 1) + "/camera_1";
videoReader[i].release();
videoReader[i] = cv::cudacodec::createVideoReader(rtspPathFpga);
goto tryReconnection;
}
videoFrames[0].copyTo(MergedOutput(cv::Rect(0, 0, videoFrames[0].cols, videoFrames[0].rows)));
videoFrames[1].copyTo(MergedOutput(cv::Rect(videoFrames[0].cols, 0, videoFrames[1].cols, videoFrames[1].rows)));
videoFrames[2].copyTo(MergedOutput(cv::Rect(0, videoFrames[1].rows, videoFrames[2].cols, videoFrames[2].rows)));
videoFrames[3].copyTo(MergedOutput(cv::Rect(videoFrames[2].cols, videoFrames[3].rows, videoFrames[3].cols, videoFrames[3].rows)));
//cout << fpga1.size() << endl;
imshow("OpenGL Output",MergedOutput);
if (waitKey(1) == 'q') {
break;
}
}
}catch(string ex){
cout << "camera 2: " << endl;
cout << "An exception occurred." << ex;
}
}

Convert images from Pylon to Opencv in c++

I want to convert stereo images captured by Basler cameras to opencv (Mat) format. In the below code i have converted images to opencv format, but in show stages, i can not show the images. please guide me.
Thanks
int main(int argc, char* argv[])
{
// The exit code of the sample application.
int exitCode = 0;
PylonInitialize();
Pylon::PylonAutoInitTerm autoInitTerm;//me
try
{
// Get the transport layer factory.
CTlFactory& tlFactory = CTlFactory::GetInstance();
// Get all attached devices and exit application if no device is found.
DeviceInfoList_t devices;
if (tlFactory.EnumerateDevices(devices) == 0)
{
throw RUNTIME_EXCEPTION("No camera present.");
}
CInstantCameraArray cameras(min(devices.size(), c_maxCamerasToUse));
// Create and attach all Pylon Devices.
for (size_t i = 0; i < cameras.GetSize(); ++i)
{
cameras[i].Attach(tlFactory.CreateDevice(devices[i]));
// Print the model name of the camera.
cout << "Using device " << cameras[i].GetDeviceInfo().GetModelName() << endl;
}
CGrabResultPtr ptrGrabResult;
CImageFormatConverter formatConverter;//me
formatConverter.OutputPixelFormat = PixelType_BGR8packed;//me
CPylonImage pylonImage;//me
// Create an OpenCV image
Mat openCvImage;//me
for (int i = 0; i < c_countOfImagesToGrab && cameras.IsGrabbing(); ++i)
{
cameras.RetrieveResult(5000, ptrGrabResult, TimeoutHandling_ThrowException);
intptr_t cameraContextValue = ptrGrabResult->GetCameraContext();
#ifdef PYLON_WIN_BUILD
#endif
// Print the index and the model name of the camera.
cout << "Camera " << cameraContextValue << ": " << cameras[cameraContextValue].GetDeviceInfo().GetModelName() << endl;
// Now, the image data can be processed.
cout << "GrabSucceeded: " << ptrGrabResult->GrabSucceeded() << endl;
cout << "SizeX: " << ptrGrabResult->GetWidth() << endl;
cout << "SizeY: " << ptrGrabResult->GetHeight() << endl;
const uint8_t *pImageBuffer = (uint8_t *)ptrGrabResult->GetBuffer();
cout << "Gray value of first pixel: " << (uint32_t)pImageBuffer[0] << endl << endl;
formatConverter.Convert(pylonImage, ptrGrabResult);//me
// Create an OpenCV image out of pylon image
openCvImage = cv::Mat(ptrGrabResult->GetHeight(), ptrGrabResult->GetWidth(), CV_8UC3, (uint8_t *)pylonImage.GetBuffer());//me
if (cameraContextValue == 0)
{
imshow("left camera", openCvImage);
imwrite("right_img.png", openCvImage);
}
else if (cameraContextValue == 1)
{
imshow("right camera", openCvImage);
imwrite("right_img.png", openCvImage);
}
Sleep(3000);
}
}
catch (const GenericException &e)
{
// Error handling
cerr << "An exception occurred." << endl
<< e.GetDescription() << endl;
exitCode = 1;
}
// Comment the following two lines to disable waiting on exit.
cerr << endl << "Press Enter to exit." << endl;
while (cin.get() != '\n');
// Releases all pylon resources.
PylonTerminate();
return exitCode;
}
You need to create a window to display an opencv image into, use :
namedWindow("left camera", CV_WINDOW_NORMAL);
imshow("left camera", openCvImage);
There is also a few mistakes in your code, i guess "right_img.png" should be change in "left_img.png", otherwise you will save only one image.
And this is redundant code
PylonInitialize();
Pylon::PylonAutoInitTerm autoInitTerm;
autoInitTerm is automatically calling PylonInitialize() and PylonTerminate(). So you should remove it or remove PylonInitialize() and PylonTerminate()
I think a waitKey(0) is required after the imshow to display the image.
add below piece of code. after completing for (size_t i = 0; i < cameras.GetSize(); ++i)
cameras.StartGrabbing(GrabStrategy_LatestImageOnly, GrabLoop_ProvidedByUser);
Add this to your code. and like above comments remove unnecessary code.

openCV video not registering final frame

I am displaying and manipulating a video with open CV as so
// looping through list of videos
for (unsigned int i = 0; i < img_loc.size(); i++)
{
string fileLoc = root_dir + "\\" + img_loc[i];
string name = img_loc[i].substr(0,img_loc[i].find("."));
cv::VideoCapture cap(fileLoc);
image_window win;
int cnt = 0;
while (!win.is_closed())
{
cv::Mat temp;
cap >> temp;
if (temp.empty())
{
break;
}
cout << "frame number ---- " << cap.get(CV_CAP_PROP_POS_FRAMES) << endl;
cv_image<bgr_pixel> cimg(temp);
// some image manipulations
win.clear_overlay();
win.set_image(cimg);
cout << cnt << endl;
// save some details
cnt++;
cout << "after cnt++ ------------ " << cnt << endl;
}
cout << endl << "finished with " << img_loc[i] << ", proceed to the next video?" << endl;
cin.get();
}
which works fine for the first for all frames except the very last. I know there are exactly 200 frames but cnt only ever reaches 198 and the frames only ever go up to 199. The final frame isn't entirely necessary but it means there's some extra data handling that I'd rather avoid. Any ideas?

Head Pose Estimation on Random Forest in G Fanelli's paper

I have been working on head pose estimation on depth data. And I have read G Fanelli's paper-"Real Time Head Pose Estimation from Consumer Depth Cameras" "Real Time Head Pose Estimation with Random Regression Forests". I test the data and the code Fanelli published on the website(http://www.vision.ee.ethz.ch/~gfanelli/head_pose/head_forest.html). However when I run the code, there is a problem. The error information is "usage: ./head_pose_estimation config_file depth_image". I think it is about file reading but I don't how to fix it.
and the code is like this:
int main(int argc, char* argv[])
{
if( argc != 3 )
{
cout << "usage: ./head_pose_estimation config_file depth_image" << endl;
exit(-1);
}
loadConfig(argv[1]);
CRForestEstimator estimator;
if( !estimator.loadForest(g_treepath.c_str(), g_ntrees) ){
cerr << "could not read forest!" << endl;
exit(-1);
}
string depth_fname(argv[2]);
//read calibration file (should be in the same directory as the depth image!)
string cal_filename = depth_fname.substr(0,depth_fname.find_last_of("/")+1);
cal_filename += "depth.cal";
ifstream is(cal_filename.c_str());
if (!is){
cerr << "depth.cal file not found in the same folder as the depth image! " << endl;
return -1;
}
//read intrinsics only
float depth_intrinsic[9]; for(int i =0; i<9; ++i) is >> depth_intrinsic[i];
is.close();
Mat depthImg;
//read depth image (compressed!)
if (!loadDepthImageCompressed( depthImg, depth_fname.c_str() ))
return -1;
Mat img3D;
img3D.create( depthImg.rows, depthImg.cols, CV_32FC3 );
//get 3D from depth
for(int y = 0; y < img3D.rows; y++)
{
Vec3f* img3Di = img3D.ptr<Vec3f>(y);
const int16_t* depthImgi = depthImg.ptr<int16_t>(y);
for(int x = 0; x < img3D.cols; x++){
float d = (float)depthImgi[x];
if ( d < g_max_z && d > 0 ){
img3Di[x][0] = d * (float(x) - depth_intrinsic[2])/depth_intrinsic[0];
img3Di[x][1] = d * (float(y) - depth_intrinsic[5])/depth_intrinsic[4];
img3Di[x][2] = d;
}
else{
img3Di[x] = 0;
}
}
}
g_means.clear();
g_votes.clear();
g_clusters.clear();
string pose_filename(depth_fname.substr(0,depth_fname.find_last_of('_')));
pose_filename += "_pose.bin";
cv::Vec<float,POSE_SIZE> gt;
bool have_gt = false;
//try to read in the ground truth from a binary file
FILE* pFile = fopen(pose_filename.c_str(), "rb");
if(pFile){
have_gt = true;
have_gt &= ( fread( &gt[0], sizeof(float),POSE_SIZE, pFile) == POSE_SIZE );
fclose(pFile);
}
//do the actual estimate
estimator.estimate( img3D,
g_means,
g_clusters,
g_votes,
g_stride,
g_maxv,
g_prob_th,
g_larger_radius_ratio,
g_smaller_radius_ratio,
false,
g_th
);
cout << "Heads found : " << g_means.size() << endl;
//assuming there's only one head in the image!
if(g_means.size()>0){
cout << "Estimated: " << g_means[0][0] << " " << g_means[0][1] << " " << g_means[0][2] << " " << g_means[0][3] << " " << g_means[0][4] << " " << g_means[0][5] <<endl;
float pt2d_est[2];
float pt2d_gt[2];
if(have_gt){
cout << "Ground T.: " << gt[0] << " " << gt[1] << " " << gt[2] << " " << gt[3] << " " << gt[4] << " " << gt[5] <<endl;
cv::Vec<float,POSE_SIZE> err = (gt-g_means[0]);
//multiply(err,err,err);
for(int n=0;n<POSE_SIZE;++n)
err[n] = err[n]*err[n];
float h_err = sqrt(err[0]+err[1]+err[2]);
float a_err = sqrt(err[3]+err[4]+err[5]);
cout << "Head error : " << h_err << " mm " << endl;
cout << "Angle error : " << a_err <<" degrees " << endl;
pt2d_gt[0] = depth_intrinsic[0]*gt[0]/gt[2] + depth_intrinsic[2];
pt2d_gt[1] = depth_intrinsic[4]*gt[1]/gt[2] + depth_intrinsic[5];
}
pt2d_est[0] = depth_intrinsic[0]*g_means[0][0]/g_means[0][2] + depth_intrinsic[2];
pt2d_est[1] = depth_intrinsic[4]*g_means[0][1]/g_means[0][2] + depth_intrinsic[5];
}
return 0;
}
can anyone could tell me how to fix the problem?Thanks so much!
You should always read the readme.txt (here attached in head_pose_estimation.tgz) before testing an application:
To run the example code, type ./head_pose_estimation config.txt
data/frame_XXXX_depth.bin. The config.txt file contains all parameters
needed for the head pose estimation, e.g., the path to the forest, the
stride, and z threshold used to segment the person from the
background.

OpenCV resize() Assertion failed

I'm having a problem with OpenCV and the Assertion failed error. It seems to be having a problem reading the image so the function imread() fails silently and then in resize() function it crashes because the input image is empty.
I've checked the path and it's ok, in fact if I print the path that I'm trying to read and do open path in a terminal image opens just fine. Also, if I take this path that I printed and put it in a string variable code works. So the problem seems to be with ss object. I just don't know what.
Here's my code:
int main(int argc, char** argv ){
cout << "Prueba OpenCV \n";
char* path;
Mat input;
if(argc >= 2 )
{
path = argv[1];
for(int j=0; j< numFiles; j++){
cout << "Character "<< OCR::strCharacters[1] << " file: " << j << "\n";
stringstream ss(stringstream::in | stringstream::out);
ss << path << OCR::strCharacters[1] << "/" << j << ".jpg" << "\n";
cout << "Character file: " << ss.str() << "\n";
Mat img=imread(ss.str(), 0);
/*if (img.cols == 0) {
cout << "Error reading file " << ss << endl;
return -1;
}*/
Mat lowData;
resize(img, lowData, Size(5, 5) );
}
}
}
run this with:
./PruebaOpenCV samples/
I found this link in stack that had a similar problem but the answer just was a piece of code to know if the image is properly read (commented code above). I know mine isn't but path is ok.