I want to verify if one person is similar to another person. Therefore I want to get the similarity between two faces.
These are the input faces
Mindy Face
Madonna Face
Now I want to push them through the DNN and after that I want to get the Euklidian distance between the two resulting matrices.
I've used the following OpenFace model: https://storage.cmusatyalab.org/openface-models/nn4.small2.v1.t7
This is my code for calculating the distances:
cv::Mat madonna = cv::imread("/home/der/Madonna_Face.jpg");
cv::Mat mindy = cv::imread("/home/der/Mindy_Face.jpg");
cv::resize(madonna, madonna, cv::Size(96, 96));
cv::resize(mindy, mindy, cv::Size(96, 96));
cv::Mat madonnaBlob = cv::dnn::blobFromImage(madonna, 1.0 / 255, cv::Size(96, 96), cv::Scalar{0,0,0}, true, false);
cv::Mat mindyBlob = cv::dnn::blobFromImage(mindy, 1.0 / 255, cv::Size{96, 96}, cv::Scalar{0,0,0}, true, false);
cv::dnn::Net _net = cv::dnn::readNetFromTorch("/home/der/nn4.small2.v1.t7");
_net.setInput(madonnaBlob);
cv::Mat madonnaMat = _net.forward();
_net.setInput(mindyBlob);
cv::Mat mindyMat = _net.forward();
std::cout << cv::norm(madonnaMat, mindyMat);
And if I'm doing so the result from cv::norm is 0.
The representations are exactly the same:
std::vector<double> master = madonnaMat;
std::vector<double> slave = mindyMat;
for(int i; i < 128; i++) {
std::cout << master[i] << " # " << slave[i] << std::endl;
}
Output:
> -0.0865457 # -0.0865457
> 0.133816 # 0.133816
> -0.105774 # -0.105774
> 0.05389 # 0.05389
> -0.00391233 # -0.00391233
> ...
Results:
Madonna Representation: [-0.060358506, 0.14156586, -0.10181303, 0.060315549, 0.0016125928, 0.066964693, -0.044892643, -0.043857966, 0.088271223, 0.047121659, 0.078663647, 0.025775915, 0.062051967, 0.034234334, -0.049976062, 0.045926169, 0.084343202, 0.046965379, -0.092582494, 0.13601208, -0.003582818, -0.15382886, 0.075037867, 0.19894752, -0.041007876, -0.12050319, -0.056161541, -0.018724455, 0.024790274, 0.0092850979, 0.095108159, 0.067354925, 0.06044127, 0.041365273, -0.12024247, 0.18279234, 0.027767293, 0.09874554, -0.16951905, 0.062370241, -0.014530737, 0.015518869, -0.0056175897, -0.066358574, -0.02390888, -0.07608442, 0.13011196, 0.031423025, -0.010443882, 0.12755248, -0.010195011, 0.0051672528, -0.10453289, -0.013270194, 0.096139617, 0.10375636, -0.047089621, 0.050923191, 0.066422582, -0.046726897, -0.1845296, 0.031028474, 0.086226918, -0.27064508, 0.055891197, -0.0053421594, 0.035870265, -0.026942547, -0.17279817, 0.13772435, 0.0071162563, 0.075375959, -0.046405111, 0.12658595, 0.11093359, 0.0030428318, 0.070016958, 0.1725318, -0.056130294, -0.14420295, -0.12438529, 0.056423288, -0.080888703, -0.052004829, -0.06481386, 0.14163122, -0.059617694, -0.026075639, 0.052098148, -0.0055074869, -0.014869845, -0.11943244, 0.068051606, -0.096071519, 0.19727865, -0.016027609, -0.05776047, 0.069935486, -0.020494614, 0.013407955, -0.06065508, -0.056143567, -0.04608072, 0.072748154, -0.035580911, 0.15261506, -0.074352823, -0.081481896, 0.020475708, -0.021581693, -0.16350025, 0.12794609, 0.082243897, 0.015881324, 0.011330541, -0.026391003, 0.086644463, -0.10490314, 0.088207267, 0.17892174, 0.025871141, 0.012454472, 0.010682535, 0.1253885, -0.12909022, 0.082067415, -0.035789803, 0.032903988]
Madonna Size: 1 x 128
Madonna Dims: 2
Mindy Representation: [-0.082645342, 0.14463238, -0.10716592, 0.065654278, 0.0045089996, 0.064019054, -0.047334831, -0.056190431, 0.099919245, 0.048234992, 0.068906084, 0.028518379, 0.057044145, 0.046223734, -0.056203742, 0.033566523, 0.082230642, 0.055683684, -0.080982864, 0.12431844, -0.00075431512, -0.14511517, 0.045022864, 0.20965824, -0.030178605, -0.11852413, -0.066858761, -0.01461118, 0.032898057, 0.02857255, 0.1088237, 0.07066118, 0.044605579, 0.022743503, -0.10785796, 0.20373915, 0.010565795, 0.063950166, -0.18701579, 0.062780239, -0.0042907735, 0.031276166, -0.006556896, -0.038440779, -0.01419229, -0.072688736, 0.13676986, 0.040385362, 0.010314438, 0.095734902, -0.0080824783, 0.011763249, -0.098884396, -0.040797569, 0.10534941, 0.12088351, -0.07317061, 0.063644305, 0.0830286, -0.050620016, -0.18088549, 0.03330183, 0.090282671, -0.25393733, 0.056058947, -0.020288708, 0.049997903, -0.044997148, -0.15860014, 0.15251927, 0.015151619, 0.088731326, -0.028061632, 0.11127418, 0.090425298, 0.0052096732, 0.053858042, 0.18543676, -0.066999368, -0.15851147, -0.11389373, 0.088093147, -0.08713299, -0.048095752, -0.063261949, 0.12453313, -0.051213119, -0.023759408, 0.048403475, -0.012721839, -0.021282939, -0.098075315, 0.066707589, -0.11601795, 0.20438787, -0.015739718, -0.052848384, 0.057336167, -0.01592578, 0.014057826, -0.058749981, -0.043632519, -0.031006066, 0.046038814, -0.065755703, 0.15442967, -0.082077362, -0.099808514, 0.016168201, 0.0046916353, -0.14556217, 0.11152669, 0.062443323, -0.00032889194, 0.0020548289, -0.026999777, 0.096809812, -0.11947374, 0.085579365, 0.16317753, 0.028130196, 0.014577032, 0.0079531483, 0.11340163, -0.15006165, 0.094127603, -0.0440454, 0.033095147]
Mindy Size: 1 x 128
Mindy Dims: 2
Any ideas what I'm doing wrong? Thanks.
I've experienced this several times. I couldn't find this explicitly mentioned in the OpenCV documentation, but the cv::dnn::Net::forward function returns a cv::Mat blob with the data member pointing always to the same zone of memory. Therefore on the second forward pass, that zone of memory is overwritten and both madonnaBlob and mindyBlob point there.
As #Christoph Rackwitz pointed out, you need to clone the cv::Mat before running the second inference:
_net.setInput(madonnaBlob);
cv::Mat madonnaMat = _net.forward();
madonnaMat = madonnaMat.clone(); // Copy memory
_net.setInput(mindyBlob);
cv::Mat mindyMat = _net.forward();
I'm writing C++ code using CAFFE to predict a single (for now) image. The image has already been preprocessed and is in .png format. I have created a Net object and read in the trained model. Now, I need to use the .png image as an input layer and call net.Forward() - but can someone help me figure out how to set the input layer?
I found a few examples on the web, but none of them work, and almost all of them use deprecated functionality. According to: Berkeley's Net API, using "ForwardPrefilled" is deprecated, and using "Forward(vector, float*)" is deprecated. API indicates that one should "set input blobs, then use Forward() instead". That makes sense, but the "set input blobs" part is not expanded on, and I can't find a good C++ example on how to do that.
I'm not sure if using a caffe::Datum is the right way to go or not, but I've been playing with this:
float lossVal = 0.0;
caffe::Datum datum;
caffe::ReadImageToDatum("myImg.png", 1, imgDims[0], imgDims[1], &datum);
caffe::Blob< float > *imgBlob = new caffe::Blob< float >(1, datum.channels(), datum.height(), datum.width());
//How to get the image data into the blob, and the blob into the net as input layer???
const vector< caffe::Blob< float >* > &result = caffeNet.Forward(&lossVal);
Again, I'd like to follow the API's direction of setting the input blobs and then using the (non-deprecated) caffeNet.Forward(&lossVal) to get the result as opposed to making use of the deprecated stuff.
EDIT:
Based on an answer below, I updated to include this:
caffe::MemoryDataLayer<unsigned char> *memory_data_layer = (caffe::MemoryDataLayer<unsigned char> *)caffeNet.layer_by_name("input").get();
vector< caffe::Datum > datumVec;
datumVec.push_back(datum);
memory_data_layer->AddDatumVector(datumVec);
but now the call to AddDatumVector is seg faulting.. I wonder if this is related to my prototxt format? here's the top of my prototxt:
name: "deploy"
input: "data"
input_shape {
dim: 1
dim: 3
dim: 100
dim: 100
}
layer {
name: "conv1"
type: "Convolution"
bottom: "data"
top: "conv1"
I base this part of the question on this discussion about a "source" field being important in the prototxt...
caffe::Datum datum;
caffe::ReadImageToDatum("myImg.png", 1, imgDims[0], imgDims[1], &datum);
MemoryDataLayer<float> *memory_data_layer = (MemoryDataLayer<float> *)caffeNet->layer_by_name("data").get();
memory_data_layer->AddDatumVector(datum);
const vector< caffe::Blob< float >* > &result = caffeNet.Forward(&lossVal);
Something like this could be useful. Here you will have to use MemoryData layer as the input layer. I am expecting the layer name to be named data.
The way of using datum variable may not be correct. If my memory is correct, I guess, you have to use a vector of datum data.
I think this should get you started.
Happy brewing. :D
Here is an excerpt from my code located here where I used Caffe in my C++ code. I hope this helps.
Net<float> caffe_test_net("models/sudoku/deploy.prototxt", caffe::TEST);
caffe_test_net.CopyTrainedLayersFrom("models/sudoku/sudoku_iter_10000.caffemodel");
// Get datum
Datum datum;
if (!ReadImageToDatum("examples/sudoku/cell.jpg", 1, 28, 28, false, &datum)) {
LOG(ERROR) << "Error during file reading";
}
// Get the blob
Blob<float>* blob = new Blob<float>(1, datum.channels(), datum.height(), datum.width());
// Get the blobproto
BlobProto blob_proto;
blob_proto.set_num(1);
blob_proto.set_channels(datum.channels());
blob_proto.set_height(datum.height());
blob_proto.set_width(datum.width());
int size_in_datum = std::max<int>(datum.data().size(),
datum.float_data_size());
for (int ii = 0; ii < size_in_datum; ++ii) {
blob_proto.add_data(0.);
}
const string& data = datum.data();
if (data.size() != 0) {
for (int ii = 0; ii < size_in_datum; ++ii) {
blob_proto.set_data(ii, blob_proto.data(ii) + (uint8_t)data[ii]);
}
}
// Set data into blob
blob->FromProto(blob_proto);
// Fill the vector
vector<Blob<float>*> bottom;
bottom.push_back(blob);
float type = 0.0;
const vector<Blob<float>*>& result = caffe_test_net.Forward(bottom, &type);
What about:
Caffe::set_mode(Caffe::CPU);
caffe_net.reset(new caffe::Net<float>("your_arch.prototxt", caffe::TEST));
caffe_net->CopyTrainedLayersFrom("your_model.caffemodel");
Blob<float> *your_blob = caffe_net->input_blobs()[0];
your_blob->set_cpu_data(your_image_data_as_pointer_to_float);
caffe_net->Forward();