I am using aruco markers to get the location of a robot. After getting the pose from the estimatePoseSingleMarkers i obtain the rvecs and tvecs for a given marker. From this how could i obtain the rotated angle of the marker about each axis.
i used the code below to detect and draw the aruco markers along with its axis
while(true)
{
vector< vector<Point2f>> corners; //All the Marker corners
vector<int> ids;
cap >> frame;
cvtColor(frame, gray, CV_BGR2GRAY);
aruco::detectMarkers(gray, dictionary, corners, ids);
aruco::drawDetectedMarkers(frame,corners,ids);
aruco::estimatePoseSingleMarkers(corners, arucoMarkerLength, cameraMatrix, distanceCoefficients, rvecs, tvecs);
for(int i = 0; i < ids.size(); i++)
{
aruco::drawAxis(frame, cameraMatrix, distanceCoefficients, rvecs[i], tvecs[i], 0.1f);
}
imshow("Markers", frame);
int key = waitKey(10);
if((char)key == 'q')
break;
}
The rotation of the marker with respect to the camera was obtained by first taking the rotation matrix from the rotation vector(rvec) and then by taking the euler angle.
Converting Rotation matrix to Eurler angles are given here
Related
My Images;
Requirement:
I am not able to understand how axis is decided to make the image always horizontal.
Algorithm:
Read the image
Find external contour
Draw the contours
Use the external contour to detect minArearect (bounding box will not help for me)
get the rotation matrix and rotate the image
Extract the required patch from the rotated image
My code:
//read the image
img = imread("90.jpeg")
cv::Mat contourOutput = img.clone();
// detect external contour(images will have noise, although example images doesn't have)
std::vector<std::vector<cv::Point> > contours;
cv::findContours(contourOutput, contours, RETR_EXTERNAL, CHAIN_APPROX_SIMPLE);
int largest_area = 0;
int largest_contour_index = 0;
for (size_t i = 0; i < contours.size(); i++) {
double area = contourArea(contours[i]);
// copy the largest contour index
if (area > largest_area) {
largest_area = area;
largest_contour_index = i;
}
}
//draw contours
drawContours(img, contours, largest_contour_index, Scalar(255, 0, 0),
2);
// detect minimum area rect to get the angle and centre
cv::RotatedRect box = cv::minAreaRect(cv::Mat(contours[largest_contour_index]));
// take the box angle
double angle = box.angle;
if (angle < -45) {
box.angle += 90;
}
angle = box.angle;
// create rotation matrix
cv::Mat rot_mat = cv::getRotationMatrix2D(box.center, angle, 1);
// Apply the transformation
cv::Mat rotated;
cv::warpAffine(img, rotated, rot_mat, img.size(), cv::INTER_CUBIC);
cv::Size box_size = box.size;
if (box.angle < -45.)
std::swap(box_size.width, box_size.height);
// get the cropped image
cv::Mat cropped;
cv::getRectSubPix(rotated, box_size, box.center, cropped);
// Display the image
namedWindow("image2", WINDOW_NORMAL);
imshow("image2", cropped);
waitKey(0);
The idea is to compute the rotated bounding box angle using minAreaRect then deskew the image with getRotationMatrix2D and warpAffine. One final step is to rotate by 90 degrees if we are working with a vertical image. Here's the results with before (left) and after (right) and the angle of rotation:
-39.999351501464844
38.52387619018555
1.6167902946472168
1.9749339818954468
I implemented it in Python but you can adapt the same approach into C++
Code
import cv2
import numpy as np
# Load image, grayscale, and Otsu's threshold
image = cv2.imread('4.png')
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
thresh = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)[1]
# Compute rotated bounding box
coords = np.column_stack(np.where(thresh > 0))
angle = cv2.minAreaRect(coords)[-1]
# Determine rotation angle
if angle < -45:
angle = -(90 + angle)
else:
angle = -angle
print(angle)
# Rotate image to deskew
(h, w) = image.shape[:2]
center = (w // 2, h // 2)
M = cv2.getRotationMatrix2D(center, angle, 1.0)
rotated = cv2.warpAffine(image, M, (w, h), flags=cv2.INTER_CUBIC, borderMode=cv2.BORDER_REPLICATE)
# Vertical image so rotate to horizontal
h, w, _ = rotated.shape
if h > w:
rotated = cv2.rotate(rotated, cv2.ROTATE_90_CLOCKWISE)
cv2.imshow('rotated', rotated)
cv2.imwrite('rotated.png', rotated)
cv2.waitKey()
I'm developing a simple marker based augmented reality application with OpenCV Viz and ARUCO. I just want to visualize a 3D object (in PLY format) on a marker.
I can run marker detection and pose estimation (returning rotation and translation vectors) with ARUCO without a problem. And I can visualize any 3D object (PLY format) and camera frames in Viz window. However, I stuck in using rotation and translation vector outputs from ARUCO to localize 3D model on the marker.
I'm creating an affine transformation with rotation and translation vectors and applying it to 3D model. Is it correct? How should I make use of translation and rotation vectors?
Below is my code snippet.
// Camera calibration outputs
cv::Mat cameraMatrix, distCoeffs;
loadIntrinsicCameraParameters(cameraMatrix, distCoeffs);
// Marker dictionary
Ptr<cv::aruco::Dictionary> dictionary = cv::aruco::getPredefinedDictionary(cv::aruco::DICT_6X6_250);
viz::Viz3d myWindow("Coordinate Frame");
cv::Mat image;
// Webcam frame pose, without this frame is upside-down
Affine3f imagePose(Vec3f(3.14159,0,0), Vec3f(0,0,0));
// Viz viewer pose to see whole webcam frame
Vec3f cam_pos( 0.0f,0.0f,900.0f), cam_focal_point(0.0f,0.0f,0.0f), cam_y_dir(0.0f,0.0f,0.0f);
Affine3f viewerPose = viz::makeCameraPose(cam_pos, cam_focal_point, cam_y_dir);
// Video capture from source
VideoCapture cap(camID);
int frame_width = cap.get(CV_CAP_PROP_FRAME_WIDTH);
int frame_height = cap.get(CV_CAP_PROP_FRAME_HEIGHT);
cap >> image;
// Load mash data
viz::WMesh batman(viz::Mesh::load("../data/bat.ply"));
viz::WImage3D img(image, Size2d(frame_width, frame_height));
// Show camera frame, mesh and a coordinate widget (for debugging)
myWindow.showWidget("Image", img);
myWindow.showWidget("Batman", batman);
myWindow.showWidget("Coordinate Widget", viz::WCoordinateSystem(5.0));
myWindow.setFullScreen(true);
myWindow.setViewerPose(viewerPose);
// Rotation vector of 3D model
Mat rot_vec = Mat::zeros(1,3,CV_32F);
cv::Vec3d rvec, tvec;
// ARUCO outputs
float roll, pitch, yaw;
float x, y, z;
while (!myWindow.wasStopped()) {
if (cap.read(image)) {
cv::Mat image, imageCopy;
cap.retrieve(image);
image.copyTo(imageCopy);
// Marker detection
std::vector<int> ids;
std::vector<std::vector<cv::Point2f> > corners;
cv::aruco::detectMarkers(image, dictionary, corners, ids);
if (ids.size() > 0){
// Draw a green line around markers
cv::aruco::drawDetectedMarkers(imageCopy, corners, ids);
vector<Vec3d> rvecs, tvecs;
// Get rotation and translation vectors of each markers
cv::aruco::estimatePoseSingleMarkers(corners, 0.05, cameraMatrix, distCoeffs, rvecs, tvecs);
for(int i=0; i<ids.size(); i++){
cv::aruco::drawAxis(imageCopy, cameraMatrix, distCoeffs, rvecs[i], tvecs[i], 0.1);
// Take only the first marker's rotation and translation to visualize 3D model on this marker
rvec = rvecs[0];
tvec = tvecs[0];
roll = rvec[0];
pitch = rvec[1];
yaw = rvec[2];
x = tvec[0];
y = tvec[1];
z = tvec[2];
qDebug() << rvec[0] << "," << rvec[1] << "," << rvec[2] << "---" << tvec[0] << "," << tvec[1] << "," << tvec[2];
}
}
// Show camera frame in Viz window
img.setImage(imageCopy);
img.setPose(imagePose);
}
// Create affine pose from rotation and translation vectors
rot_vec.at<float>(0,0) = roll;
rot_vec.at<float>(0,1) = pitch;
rot_vec.at<float>(0,2) = yaw;
Mat rot_mat;
Rodrigues(rot_vec, rot_mat);
Affine3f pose(rot_mat, Vec3f(x, y, z));
// Set the pose of 3D model
batman.setPose(pose);
myWindow.spinOnce(1, true);
}
I think your problem is that rvec it is not the roll pitch and yaw angles but rodriguez vector. To find roll pitch and yaw angles you have to use Rodriguez function of Opencv to transform rotation vector (rvec) to the rotation matrix. Then use RQDecomp3x3 to decompose rotation matrix to euler angles (roll pitch and yaw )
I need explanation of the following loop for face detection in opencv
VideoCapture capture("DSC_0772.avi"); //-1, 0, 1 device id
Mat cap_img,gray_img;
vector<Rect> faces, eyes;
while(1)
{
capture >> cap_img;
waitKey(10);
cvtColor(cap_img, gray_img, CV_BGR2GRAY);
cv::equalizeHist(gray_img,gray_img);
face_cascade.detectMultiScale(gray_img, faces, 1.1, 5, CV_HAAR_SCALE_IMAGE | CV_HAAR_DO_CANNY_PRUNING, cvSize(0,0), cvSize(300,300));
for(int i=0; i < faces.size();i++)
{
Point pt1(faces[i].x+faces[i].width, faces[i].y+faces[i].height);
Point pt2(faces[i].x,faces[i].y);
rectangle(cap_img, pt1, pt2, cvScalar(0,255,0), 2, 8, 0);
}
I don't understand faces[i].x and the other for loop parameters
how they are selected for face detection
Thanks for help
faces is a std::vector of Rect. So the for loop is going through each Rect in the vector and it is creating two points. Rect stores not only an x and y(of the top left corner) but also the height and width of the rectangle. So faces[i].x+faces[i].width is taking the x coordinate of the rectangle plus its width and faces[i].y+faces[i].height is taking the y coordinate of the rectangle plus its height. This is getting the opposite corner of the rectangle. You are then feeding those points plus the image into the rectangle() function.
I have a set of 3d points in world coordinates and respective correspondences with 2d points in an image. I want to find a matrix that gives me the transformation between these set of points. How can I do this in OpenCV?
cv::solvePnP() is what you are looking for, it finds an object pose from 3D-2D point correspondences and results a rotation vector (rvec), that, together with translation vector (tvec), brings points from the model coordinate system to the camera coordinate system.
you can use solvePnP for this:
// camMatrix based on img size
int max_d = std::max(img.rows,img.cols);
Mat camMatrix = (Mat_<double>(3,3) <<
max_d, 0, img.cols/2.0,
0, max_d, img.rows/2.0,
0, 0, 1.0);
// 2d -> 3d correspondence
vector<Point2d> pts2d = ...
vector<Point3d> pts3d = ...
Mat rvec,tvec;
solvePnP(pts3d, pts2d, camMatrix, Mat(1,4,CV_64F,0.0), rvec, tvec, false, SOLVEPNP_EPNP);
// get 3d rot mat
Mat rotM(3, 3, CV_64F);
Rodrigues(rvec, rotM);
// push tvec to transposed Mat
Mat rotMT = rotM.t();
rotMT.push_back(tvec.reshape(1, 1));
// transpose back, and multiply
return camMatrix * rotMT.t();
I am currently working on face detection and thereafter eyes, mouth, nose and other facial features.For above detection I have used haarcascade( frontal face, eyes, right_ear, left_ear and mouth).Now, everything works perfectly, if the face is frontal and straight. But I am not getting good result if the face is in side view or it is rotated. For side view, I have used lbscascade_profile.xml( it works only for right side of face). But for rotated face, I am not able to detect face.Can anyone help me out in above context.I am adding my code here for better understanding.
P.S : Thanks in advance and Pardon me for childish question( it might be because I am very new to programming).
void detectAndDisplay( Mat frame)
{
// create a vector array to store the face found
std::vector<Rect> faces;
Mat frame_gray;
bool mirror_image = false;
// convert the frame image into gray image file
cvtColor( frame, frame_gray, CV_BGR2GRAY);
//equalize the gray image file
equalizeHist( frame_gray, frame_gray);
//find the frontal faces and store them in vector array
face_cascade1.detectMultiScale(frame_gray,
faces,
1.1, 2,
0|CV_HAAR_SCALE_IMAGE|CV_HAAR_FIND_BIGGEST_OBJECT,
Size(40, 40),
Size(200, 200));
// find the right side face and store that in the face vector
if(!(faces.size()))
{
profileface_cascade.detectMultiScale( frame_gray,
faces,
1.2, 3,
0|CV_HAAR_SCALE_IMAGE|CV_HAAR_FIND_BIGGEST_OBJECT,
Size(40, 40),
Size(200, 200));
}
// find whether left face exist or not by flipping the frame and checking through lbsprofile
if(!faces.size())
{
cv::flip(frame_gray, frame_gray, 1);
profileface_cascade.detectMultiScale( frame_gray,
faces,
1.2, 3,
0|CV_HAAR_SCALE_IMAGE|CV_HAAR_FIND_BIGGEST_OBJECT,
Size(40, 40),
Size(200, 200));
mirror_image = true;
}
// if the frame is not flipped then the it could be directly drawn into frame
if(mirror_image and faces.size())
{
// flip the frame
cv::flip(frame_gray, frame_gray, 1);
}
if(faces.size())
{
//draw rectangle for the faces detected
rectangle(frame, faces[0], cvScalar(0, 255, 0, 0), 1, 8, 0);
}
// check whether any face is present in frame or not
else
image_not_found++;
imshow("Face Detection", frame);
}
Flandmark will be your friend, then! I've been using it recently quite often, and it turned out to be a successful tool in head pose estimation hence particular in detecting "rotated" face. It works quite reasonable in range of angles: tilt (rotation around axis parallel to image's width) from -30 to +30 degrees, pan (rotation around axis parallel to image's height) from -45 to +45 degrees. Also it is a robust solution.