Detection of head nod pose from pitch angle - c++

Question

I am making an android application that will detect driver's fatigue from head nod pose.
What I did:
I have detected 68 facial landmarks from image using dlib library.
Used solvePnP to find out rotation matrix and from that matrix I have got roll, pitch and yaw angles.
Now, I have to detect head nod from pitch angle.
My Problem:
How to set a threshold value so that an angle below or above the threshold can be said as a head nod?
It results some negative angles. What does a negative angle mean in 3 dimensional axis?
My code:
void getEulerAngles(Mat &rotCamerMatrix,Vec3d &eulerAngles)
{
Mat cameraMatrix,rotMatrix,transVect,rotMatrixX,rotMatrixY,rotMatrixZ;
double* _r = rotCamerMatrix.ptr<double>();
double projMatrix[12] = {_r[0],_r[1],_r[2],0,
_r[3],_r[4],_r[5],0,
_r[6],_r[7],_r[8],0};
decomposeProjectionMatrix( Mat(3,4,CV_64FC1,projMatrix),
cameraMatrix,
rotMatrix,
transVect,
rotMatrixX,
rotMatrixY,
rotMatrixZ,
eulerAngles);
}
int renderToMat(std::vector<full_object_detection>& dets, Mat& dst)
{
Scalar color;
std::vector<cv::Point2d> image_points;
std::vector<cv::Point3d> model_points;
string disp;
int sz = 3,l;
color = Scalar(0,255,0);
double p1,p2,p3,leftear,rightear,ear=0,yawn=0.00,yaw=0.00,pitch=0.00,roll=0.00;
l=dets.size();
//I am calculating only for one face.. so assuming l=1
for(unsigned long idx = 0; idx < l; idx++)
{
image_points.push_back(
Point2d(dets[idx].part(30).x(),dets[idx].part(30).x() ) );
image_points.push_back(Point2d(dets[idx].part(8).x(),dets[idx].part(8).x() ) );
image_points.push_back(Point2d(dets[idx].part(36).x(),dets[idx].part(36).x() ) );
image_points.push_back(Point2d(dets[idx].part(45).x(),dets[idx].part(45).x() ) );
image_points.push_back(Point2d(dets[idx].part(48).x(),dets[idx].part(48).x() ) );
image_points.push_back(Point2d(dets[idx].part(54).x(),dets[idx].part(54).x() ) );
}
double focal_length = dst.cols;
Point2d center = cv::Point2d(dst.cols/2.00,dst.rows/2.00);
cv::Mat camera_matrix = (cv::Mat_<double>(3.00,3.00) << focal_length, 0, center.x, 0, focal_length, center.y, 0,
0, 1);
cv::Mat dist_coeffs = cv::Mat::zeros(4,1,cv::DataType<double>::type);
cv::Mat rotation_vector; //s Rotation in axis-angle form
cv::Mat translation_vector;
cv::Mat rotCamerMatrix1;
if(l!=0)
{
model_points.push_back(cv::Point3d(0.0f, 0.0f, 0.0f));
model_points.push_back(cv::Point3d(0.0f, -330.0f, -65.0f));
model_points.push_back(cv::Point3d(-225.0f, 170.0f, -135.0f));
model_points.push_back(cv::Point3d(225.0f, 170.0f, -135.0f));
model_points.push_back(cv::Point3d(-150.0f, -150.0f, -125.0f));
model_points.push_back(cv::Point3d(150.0f, -150.0f, -125.0f));
cv::solvePnP(model_points, image_points, camera_matrix, dist_coeffs,rotation_vector, translation_vector);
Rodrigues(rotation_vector,rotCamerMatrix1);
Vec3d eulerAngles;
getEulerAngles(rotCamerMatrix1,eulerAngles);
yaw = eulerAngles[1];
pitch = eulerAngles[0];
roll = eulerAngles[2];
/*My problem begins here. I don't know how to set a threshold value for pitch so that I can say a value below or
above the pitch is a head nod!*/
}
return 0;
}

Answer 1

First you have to understand how are the 3 angles used in a 3D context. Basically they rotate an object in the 3D space with respect to an origin (the origin can change depending on the context), but how are the 3 angles applied?.
This question can be express as: in which order do they rotate the object. If you apply yaw, then pitch and then roll it may give you a different orientation of the object that if you do it in the opposite order. Having said that, you must understand what are those values representing, to understand what to do with them.
Now, you ask what would be a good threshold, well it depends. On what? well, in the order they are applied. For instance if you apply pitch first with 45 degrees so it looks down and then you apply a roll 180 degrees, then it is looking up, so it is a little hard to define a threshold.
Since you have your model points, you can create a 3D rotation matrix and apply it to them with different pitch angles (the rest of the angles will be 0 so the order will not be important here) and visualize them and choose the one that you consider is nodding. This is a little bit subjective, so you should be the one doing it.
Now, to the second question. The answer again is, it depends. On what this time? you may ask. Simple, is your system left handed or right handed? in one this means that the rotation is apply clockwise and the other counter clockwise, the negative sign changes the direction. So, in a left handed system it will be clockwise and with the negative sign will be counterclockwise. The right handed system will be counterclockwise and the negative sign will make it clockwise.
As a suggestion, you can make a vector (0,0,-1) assuming your z axis looks towards the back. Then apply the rotation to it and proyect it to a 2D plane parallel to the z axis, here take the tip of your vector and see what is the angle here. This way you are sure what are you getting.