Periodically send data to MATLAB from mexFile - c++

Question

I'm working right now on a Data Acquisition Tool completely written
in MATLAB. It was the wish of my colleagues that i write this thing in MATLAB
so that they can expand and modify it.
The Software needs to grab a picture from two connected USB cameras.
The API for these cameras is written in C++ and is documented -> Here.
Here is the Problem:
When i write a mex file which grabs a picture it includes the
initialization and configuration-loading of the cameras which
takes a long time. When i want to grab the pictures
this way it takes MATLAB over 1 second to perform the task.
The cameras are able, once initialized, to record and send 100 fps.
The minimum frame rate i need is 10 fps.
I need to be able to send every recorded picture back
to MATLAB. Because the recording session for which the
Acquisition Tool is needed takes approx 12 hours and we
need a Live Screen with some slight PostProcessing.
Is it possible to generate a loop within the mex File which
sends data to MATLAB, then waits for a return signal from MATLAB
and continues ?
This way i could initialize the cameras and send periodically
the images to MATLAB.
I'am a Beginner in C++ and it is quite possible that i
don't understand a fundamental concept why this
is not possible.
Thank you for any advice or sources where i could look.
Please find below the Code which initializes the Cameras
using the Pylon API provided by Basler.
// Based on the Grab_MultipleCameras.cpp Routine from Basler
/*
This routine grabs one frame from 2 cameras connected
via two USB3 ports. It directs the Output to MATLAB.
*/
// Include files to use the PYLON API.
#include <pylon/PylonIncludes.h>
#include <pylon/usb/PylonUsbIncludes.h>
#include <pylon/usb/BaslerUsbInstantCamera.h>
#include <pylon/PylonUtilityIncludes.h>
// Include Files for MEX Generation
#include <matrix.h>
#include <mex.h>
// Namespace for using pylon objects.
using namespace Pylon;
// We are lazy and use Basler USB namespace
using namespace Basler_UsbCameraParams;
// Standard namespace
using namespace std;
// Define Variables Globally to be remembered between each call
// Filenames for CamConfig
const String_t filenames[] = { "NodeMapCam1.pfs","NodeMapCam2.pfs" };
// Limits the amount of cameras used for grabbing.
static const size_t camerasToUse = 2;
// Create an array of instant cameras for the found devices and
// avoid exceeding a maximum number of devices.
CBaslerUsbInstantCameraArray cameras(camerasToUse);
void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
{
// Automagically call PylonInitialize and PylonTerminate to ensure the pylon runtime system.
// is initialized during the lifetime of this object
PylonAutoInitTerm autoInitTerm;
try
{
// Get the transport layer factory
CTlFactory& tlFactory = CTlFactory::GetInstance();
// Get all attached devices and exit application if no device or USB Port is found.
DeviceInfoList_t devices;
ITransportLayer *pTL = dynamic_cast<ITransportLayer*>(tlFactory.CreateTl(BaslerUsbDeviceClass));
if (pTL == NULL)
{
throw RUNTIME_EXCEPTION("No USB transport layer available.");
}
if (pTL->EnumerateDevices(devices) == 0)
{
throw RUNTIME_EXCEPTION("No camera present.");
}
// Create and attach all Pylon Devices. Load Configuration
for (size_t i = 0; i < cameras.GetSize(); ++i)
{
cameras[i].Attach(tlFactory.CreateDevice(devices[i]));
}
// Open all cameras.
cameras.Open();
// Load Configuration and execute Trigger
for (size_t i = 0; i < cameras.GetSize(); ++i)
{
CFeaturePersistence::Load(filenames[i], &cameras[i].GetNodeMap());
}
if (cameras[0].IsOpen() && cameras[1].IsOpen())
{
mexPrintf("\nCameras are fired up and configuration is applied\n");
// HERE I WOULD LIKE TO GRAB PICTURES AND SEND THEM
// PERIODICALLY TO MATLAB.
}
}
catch (GenICam::GenericException &e)
{
// Error handling
mexPrintf("\nAn exception occured:\n");
mexPrintf(e.GetDescription());
}
return;
}

Answer 1

You could loop and send images back to MATLAB periodically, but how do you want it to be in the workspace (multiple 2D images, a huge 3D/4D array, cell, etc.)? I think the solution you are looking for is a stateful MEX file, which can be launched with an 'init' or 'new' command, and then called again repeatedly with 'capture' commands for an already initialized camera.
There is an example of how to do this in my GitHub. Start with class_wrapper_template.cpp and modify it for your commands (new, capture, delete, etc.). Here is a rough and untested example of how the core of it might look (also mirrored on Gist.GitHub):
// pylon_mex_camera_interface.cpp
#include "mex.h"
#include <vector>
#include <map>
#include <algorithm>
#include <memory>
#include <string>
#include <sstream>
//////////////////////// BEGIN Step 1: Configuration ////////////////////////
// Include your class declarations (and PYLON API).
#include <pylon/PylonIncludes.h>
#include <pylon/usb/PylonUsbIncludes.h>
#include <pylon/usb/BaslerUsbInstantCamera.h>
#include <pylon/PylonUtilityIncludes.h>
// Define class_type for your class
typedef CBaslerUsbInstantCameraArray class_type;
// List actions
enum class Action
{
// create/destroy instance - REQUIRED
New,
Delete,
// user-specified class functionality
Capture
};
// Map string (first input argument to mexFunction) to an Action
const std::map<std::string, Action> actionTypeMap =
{
{ "new", Action::New },
{ "delete", Action::Delete },
{ "capture", Action::Capture }
}; // if no initializer list available, put declaration and inserts into mexFunction
using namespace Pylon;
using namespace Basler_UsbCameraParams;
const String_t filenames[] = { "NodeMapCam1.pfs","NodeMapCam2.pfs" };
static const size_t camerasToUse = 2;
///////////////////////// END Step 1: Configuration /////////////////////////
// boilerplate until Step 2 below
typedef unsigned int handle_type;
typedef std::pair<handle_type, std::shared_ptr<class_type>> indPtrPair_type; // or boost::shared_ptr
typedef std::map<indPtrPair_type::first_type, indPtrPair_type::second_type> instanceMap_type;
typedef indPtrPair_type::second_type instPtr_t;
// getHandle pulls the integer handle out of prhs[1]
handle_type getHandle(int nrhs, const mxArray *prhs[]);
// checkHandle gets the position in the instance table
instanceMap_type::const_iterator checkHandle(const instanceMap_type&, handle_type);
void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[]) {
// static storage duration object for table mapping handles to instances
static instanceMap_type instanceTab;
if (nrhs < 1 || !mxIsChar(prhs[0]))
mexErrMsgTxt("First input must be an action string ('new', 'delete', or a method name).");
char *actionCstr = mxArrayToString(prhs[0]); // convert char16_t to char
std::string actionStr(actionCstr); mxFree(actionCstr);
for (auto & c : actionStr) c = ::tolower(c); // remove this for case sensitivity
if (actionTypeMap.count(actionStr) == 0)
mexErrMsgTxt(("Unrecognized action (not in actionTypeMap): " + actionStr).c_str());
// If action is not 'new' or 'delete' try to locate an existing instance based on input handle
instPtr_t instance;
if (actionTypeMap.at(actionStr) != Action::New && actionTypeMap.at(actionStr) != Action::Delete) {
handle_type h = getHandle(nrhs, prhs);
instanceMap_type::const_iterator instIt = checkHandle(instanceTab, h);
instance = instIt->second;
}
//////// Step 2: customize each action in the switch in mexFuction ////////
switch (actionTypeMap.at(actionStr))
{
case Action::New:
{
if (nrhs > 1 && mxGetNumberOfElements(prhs[1]) != 1)
mexErrMsgTxt("Second argument (optional) must be a scalar, N.");
handle_type newHandle = instanceTab.size() ? (instanceTab.rbegin())->first + 1 : 1;
// Store a new CBaslerUsbInstantCameraArray in the instance map
std::pair<instanceMap_type::iterator, bool> insResult =
instanceTab.insert(indPtrPair_type(newHandle, std::make_shared<class_type>(camerasToUse)));
if (!insResult.second) // sanity check
mexPrintf("Oh, bad news. Tried to add an existing handle."); // shouldn't ever happen
else
mexLock(); // add to the lock count
// return the handle
plhs[0] = mxCreateDoubleScalar(insResult.first->first); // == newHandle
// Get all attached devices and exit application if no device or USB Port is found.
CTlFactory& tlFactory = CTlFactory::GetInstance();
// Check if cameras are attached
ITransportLayer *pTL = dynamic_cast<ITransportLayer*>(tlFactory.CreateTl(BaslerUsbDeviceClass));
// todo: some checking here... (pTL == NULL || pTL->EnumerateDevices(devices) == 0)
// Create and attach all Pylon Devices. Load Configuration
CBaslerUsbInstantCameraArray &cameras = *instance;
DeviceInfoList_t devices;
for (size_t i = 0; i < cameras.GetSize(); ++i) {
cameras[i].Attach(tlFactory.CreateDevice(devices[i]));
}
// Open all cameras.
cameras.Open();
// Load Configuration and execute Trigger
for (size_t i = 0; i < cameras.GetSize(); ++i) {
CFeaturePersistence::Load(filenames[i], &cameras[i].GetNodeMap());
}
if (cameras[0].IsOpen() && cameras[1].IsOpen()) {
mexPrintf("\nCameras are fired up and configuration is applied\n");
break;
}
case Action::Delete:
{
instanceMap_type::const_iterator instIt = checkHandle(instanceTab, getHandle(nrhs, prhs));
(instIt->second).close(); // may be unnecessary if d'tor does it
instanceTab.erase(instIt);
mexUnlock();
plhs[0] = mxCreateLogicalScalar(instanceTab.empty()); // just info
break;
}
case Action::Capture:
{
CBaslerUsbInstantCameraArray &cameras = *instance; // alias for the instance
// TODO: create output array and capture a frame(s) into it
plhs[0] = mxCreateNumericArray(...);
pixel_type* data = (pixel_type*) mxGetData(plhs[0]);
cameras[0].GrabOne(...,data,...);
// also for cameras[1]?
}
}
default:
mexErrMsgTxt(("Unhandled action: " + actionStr).c_str());
break;
}
//////////////////////////////// DONE! ////////////////////////////////
}
// See github for getHandle and checkHandle
The idea is that you would call it once to init:
>> h = pylon_mex_camera_interface('new');
Then you would call it in a MATLAB loop to get frames:
>> newFrame{i} = pylon_mex_camera_interface('capture', h);
When you are done:
>> pylon_mex_camera_interface('delete', h)
You should wrap this with a MATLAB class. Derive from cppclass.m to do this easily. For a derived class example see pqheap.m.

Answer 2

Instead of sending data to MATLAB you should make your mex file store camera related settings so that it does not initialize in each call. One way to do this is to use two modes of calls for your mex file. An 'init' call and a call to get data. Pseudo code in MATLAB would be
cameraDataPtr = myMex('init');
while ~done
data = myMex('data', cameraDataPtr);
end
In your mex file, you should store the camera settings in a memory which is persistent across calls. One way to do this is using 'new' in c++. You should return this memory pointer as an int64 type to MATLAB which is shown as cameraDataPtr in the above code. When 'data' is asked for you should take cameraDataPtr as input and cast back to your camera settings. Say in C++, you have a CameraSettings object which stores all data related to camera then, a rough pseudo code in c++ would be
if prhs[0] == 'init' { // Use mxArray api to check this
cameraDataPtr = new CameraSettings; // Initialize and setup camera
plhs[0] = createMxArray(cameraDataPtr); // Use mxArray API to create int64 from pointer
return;
} else {
// Need data
cameraDataPtr = getCameraDataPtr(plhs[1]);
// Use cameraDataPtr after checking validity to get next frame
}
This works because mex files stay in memory once loaded until you clear them. You should use mexAtExit function to release camera resource when the mex file is unloaded from memory. You could also use 'static' to store your camera settings in c++ if this is the only place your mex file is going to be used. This will avoid writing some mxArray handling code for returning your c++ pointer.
If you wrap the call to this mex file inside a MATLAB object you can control the initialization and run-time process more easily and present a better API to your users.

Answer 3

I ran into the same problem and wanted to use a Basler camera with the mex API in Matlab. The contributions and hints here definitely helped me to come up with some ideas. However, there is a much simpler solution than the previously proposed one. It's not necessary to return the camera pointer to Matlab back, because objects will stay in memory across multiple mex calls. Here is a working code which I programmed with the new mex C++ API. Have fun with it.
Here is the C++ File which can be compiled with mex:
#include <opencv2/core/core.hpp>
#include <opencv2/opencv.hpp>
#include <pylon/PylonIncludes.h>
#include <pylon/usb/PylonUsbIncludes.h>
#include <pylon/usb/BaslerUsbInstantCamera.h>
#include <pylon/PylonUtilityIncludes.h>
#include "mex.hpp"
#include "mexAdapter.hpp"
#include <chrono>
#include <string>
using namespace matlab::data;
using namespace std;
using namespace Pylon;
using namespace Basler_UsbCameraParams;
using namespace GenApi;
using namespace cv;
using matlab::mex::ArgumentList;
class MexFunction : public matlab::mex::Function{
matlab::data::ArrayFactory factory;
double Number = 0;
std::shared_ptr<matlab::engine::MATLABEngine> matlabPtr = getEngine();
std::ostringstream stream;
Pylon::CInstantCamera* camera;
INodeMap* nodemap;
double systemTime;
double cameraTime;
public:
MexFunction(){}
void operator()(ArgumentList outputs, ArgumentList inputs) {
try {
Number = Number + 1;
if(!inputs.empty()){
matlab::data::CharArray InputKey = inputs[0];
stream << "You called: " << InputKey.toAscii() << std::endl;
displayOnMATLAB(stream);
// If "Init" is the input value
if(InputKey.toUTF16() == factory.createCharArray("Init").toUTF16()){
// Important: Has to be closed
PylonInitialize();
IPylonDevice* pDevice = CTlFactory::GetInstance().CreateFirstDevice();
camera = new CInstantCamera(pDevice);
nodemap = &camera->GetNodeMap();
camera->Open();
camera->RegisterConfiguration( new CSoftwareTriggerConfiguration, RegistrationMode_ReplaceAll, Cleanup_Delete);
CharArray DeviceInfo = factory.createCharArray(camera -> GetDeviceInfo().GetModelName().c_str());
stream << "Message: Used Camera is " << DeviceInfo.toAscii() << std::endl;
displayOnMATLAB(stream);
}
// If "Grab" is called
if(InputKey.toUTF16() == factory.createCharArray("Grab").toUTF16()){
static const uint32_t c_countOfImagesToGrab = 1;
camera -> StartGrabbing(c_countOfImagesToGrab);
CGrabResultPtr ptrGrabResult;
Mat openCvImage;
CImageFormatConverter formatConverter;
CPylonImage pylonImage;
while (camera -> IsGrabbing()) {
camera -> RetrieveResult(5000, ptrGrabResult, TimeoutHandling_ThrowException);
if (ptrGrabResult->GrabSucceeded()) {
formatConverter.Convert(pylonImage, ptrGrabResult);
Mat openCvImage = cv::Mat(ptrGrabResult->GetHeight(), ptrGrabResult->GetWidth(), CV_8UC1,(uint8_t *)pylonImage.GetBuffer(), Mat::AUTO_STEP);
const size_t rows = openCvImage.rows;
const size_t cols = openCvImage.cols;
matlab::data::TypedArray<uint8_t> Yp = factory.createArray<uint8_t>({ rows, cols });
for(int i = 0 ;i < openCvImage.rows; ++i){
for(int j = 0; j < openCvImage.cols; ++j){
Yp[i][j] = openCvImage.at<uint8_t>(i,j);
}
}
outputs[0] = Yp;
}
}
}
// if "Delete"
if(InputKey.toUTF16() == factory.createCharArray("Delete").toUTF16()){
camera->Close();
PylonTerminate();
stream << "Camera instance removed" << std::endl;
displayOnMATLAB(stream);
Number = 0;
//mexUnlock();
}
}
// ----------------------------------------------------------------
stream << "Anzahl der Aufrufe bisher: " << Number << std::endl;
displayOnMATLAB(stream);
// ----------------------------------------------------------------
}
catch (const GenericException & ex) {
matlabPtr->feval(u"disp", 0, std::vector<Array>({factory.createCharArray(ex.GetDescription()) }));
}
}
void displayOnMATLAB(std::ostringstream& stream) {
// Pass stream content to MATLAB fprintf function
matlabPtr->feval(u"fprintf", 0,
std::vector<Array>({ factory.createScalar(stream.str()) }));
// Clear stream buffer
stream.str("");
}
};
This mex File can be called from Matlab with the following commands:
% Initializes the camera. The camera parameters can also be loaded here.
NameOfMexFile('Init');
% Camera image is captured and sent back to Matlab
[Image] = NameOfMexFile('Grab');
% The camera connection has to be closed.
NameOfMexFile('Delete');
Optimization and improvements of this code are welcome. There are still problems with the efficiency of the code. An image acquisition takes about 0.6 seconds. This is mainly due to the cast from a cv::mat image to a TypedArray which is necessary to return it back to Matlab. See this line in the two loops: Yp[i][j] = openCvImage.at<uint8_t>(i,j);
I have not figured out how to make this more efficient yet. Furthermore, the code cannot be used to return multiple images back to Matlab.
Maybe someone has an idea or hint to make the conversion from cv::mat to a Matlab array type faster. I already mentioned the problem in another post. See here: How to Return a Opencv image cv::mat to Matlab with the Mex C++ API