Using transform matrix from larger area on ROI - c++

Question

I've a problem that I don't really know how to solve. The situation is that I've got a picture, inside this picture I have one ROI (a large rect). Inside this rect I then have X number of smaller ROI:s.
I then use the getPerspectiveTransform() method on the larger of the ROI. What I then want to do is to apply this matrix on the smaller ROI:s in-order to warp them separately but using the matrix from the larger ROI. The reason I want to do this is basically because I want to keep the bounding rects I've got before the warp. If there is any way to warp the larger ROI and keep the bounding rects inside of this (i.e. keep the smaller ROI:s) that would be very helpful!
Anyway, this is what I've tried and the results it produces:
vector<Mat> warpSmallerRoi( vector<Rect> smallerRoi ) {
vector<Mat> quad;
for( int i = 0; i < smallerRoi.size(); ++ i ) {
Mat wholeQuad;
wholeQuad = Mat::zeros( originalImage(smallerRoi[i]).rows, originalImage(smallerRoi[i]).cols, CV_8UC1 );
vector<Point2f> largerRoiCorners;
vector<Point2f> quadPoints;
// Takes the corners from the first smaller ROI and the last smaller ROI, represents the bigger ROI
largerRoiCorners.push_back( Point2f( smallerRoi[0].tl().x, smallerRoi[0].tl().y ) ); // Top left
largerRoiCorners.push_back( Point2f( smallerRoi[ smallerRoi.size() - 1 ].br().x, smallerRoi[ smallerRoi.size() - 1 ].tl().y ) ); // Top right
largerRoiCorners.push_back( Point2f( smallerRoi[ smallerRoi.size() - 1 ].br().x, smallerRoi[ smallerRoi.size() - 1 ].br().y ) ); // Bottom right
largerRoiCorners.push_back( Point2f( smallerRoi[0].tl().x, smallerRoi[0].br().y ) ); // Bottom left
quadPoints.push_back( Point2f(0, 0) );
quadPoints.push_back( Point2f(wholeQuad.cols, 0) );
quadPoints.push_back( Point2f( wholeQuad.cols, wholeQuad.rows ) );
quadPoints.push_back( Point2f(0, wholeQuad.rows) );
// Transform matrix for the larger ROI, this warps into a perfect result.
Mat largerRoiTransformMatrix = getPerspectiveTransform( largerRoiCorners, quadPoints );
//Corners of the smaller ROI
vector<Point2f> corners;
corners.push_back( Point2f( smallerRoi[i].tl().x, smallerRoi[i].tl().y ) ); // Top left
corners.push_back( Point2f( smallerRoi[i].br().x, smallerRoi[i].tl().y ) ); // Top right
corners.push_back( Point2f( smallerRoi[i].br().x, smallerRoi[i].br().y )); // Bottom right
corners.push_back( Point2f( smallerRoi[i].tl().x, smallerRoi[i].br().y ) ); // Bottom left
Mat transformMatrix = getPerspectiveTransform( corners, quadPoints );
/* This part is just experimental and does not work all the time, it works well sometimes though
Uses the pars from the larger ROI transform matrix and applies it on the smaller ROI
then warps it.
*/
transformMatrix.at<double>(1,0) = largerRoiTransformMatrix.at<double>(1,0);
transformMatrix.at<double>(1,1) = largerRoiTransformMatrix.at<double>(1,1);
transformMatrix.at<double>(1,2) = largerRoiTransformMatrix.at<double>(1,2);
transformMatrix.at<double>(2,0) = largerRoiTransformMatrix.at<double>(2,0);
transformMatrix.at<double>(2,1) = largerRoiTransformMatrix.at<double>(2,1);
transformMatrix.at<double>(2,2) = largerRoiTransformMatrix.at<double>(2,2);
// Warps the
warpPerspective( plateRgb, wholeQuad, transformMatrix, wholeQuad.size() );
quad.push_back(wholeQuad);
}
return quad;
}
This function works sometimes, especially when the larger ROI is already pretty straight (I guess this is because my replacements are not that big difference from the original values then.
Eg:
From:
To:
But then when the larger ROI has much skew the result is not so good:
From:
To:
As you can see the right part of the "H" is here a bit outside of the image. How should I go forward to transform my ROI:s so that the "H" (and all others) fit into the image and are warped with the correct tranformation matrix?
Sorry if I missed out on any information, ask in that case! Thanks :)