How to get not normalized MNIST dataset PyTorch C++ - c++

I'm trying to follow this C++ PyTorch example but I need to load the MNIST dataset with its standard values, between 0 and 255. I removed the application of the Normalize() method, but I continue getting value between 0 and 1. What am I doing wrong?
My code is:
int main(int argc, char* argv[]) {
const int64_t batch_size = 1;
// MNIST Dataset
auto train_dataset = torch::data::datasets::MNIST("./mnist")
.map(torch::data::transforms::Stack<>());
// Number of samples in the training set
auto num_train_samples = train_dataset.size().value();
cout << "Number of training samples: " << num_train_samples << endl;
// Data loaders
auto train_loader = torch::data::make_data_loader<torch::data::samplers::RandomSampler>(
std::move(train_dataset), batch_size);
for (auto& batch : *train_loader) {
auto data = batch.data.view({batch_size, -1}).to(device);
auto record = data[0].clone();
cout << "Max value: " << max(record) << endl;
cout << "Min value: " << max(record) << endl;
break;
}
}
The MNIST dataset I downloaded is the original one, from the site.
Thank you in advance for your help.

I have looked at the source file and it appears that pytorch mnist dataset class performs the division by 255 to return only tensors within the [0,1] range. So you will have to multiply the batches by 255 yourself.
The normalize transform was not the culprit. It is used to change the mean and variance of your data

Related

Storing output data (numbers) into XML/json file format - to develop graph

I am working in Computer Vision domain, coding completely in C++ using OpenCV API, I have found results and printing the values in cmd prompt. I wanna save these results(basically integers and floating point numbers) in XML file and then develop a graph (bar charts or line graphs)- basically a web dashboard(GUI). Now I am in a stage, such that I use, ofstream and save the data in a csv/xml file. But it just prints the value same as such in cmd prompt.
Can someone kindly help me with a technique to store the values in XML tree structure? so that I will create a web dashboard(bar graphs) with that xml data
I have also came across msxml6,tinyxml,libxml++ but have not got any fruitful results.
Thanks in advance - please provide link to the other question if this is a duplicate.
The code sample :
#include<opencv2/core/core.hpp>
#include<iostream>
#include<fstream>
int main ()
{
cv::VideoCapture capVideo;
capVideo.open("video.mp4");
cv::Mat imgFrame1;
cv::Mat imgFrame2;
double fps = capVideo.get(CV_CAP_PROP_FPS);
std::cout << "FPS = " << fps <<std::endl;
double fc = capVideo.get(CV_CAP_PROP_FRAME_COUNT);
std::cout << "Total Framecount = " << fc <<std::endl;
std::ofstream outfile;
outfile.open("theBigDataSheet.xml");
capVideo.read(imgFrame1);
int frameCount = 1;
while (true)
{
int divisor = fps*15;
if (frameCount%divisor == 0 || frameCount==fc-1)
{
if (frameCount<fc-1)
{
outfile << frameCount/fps << std::endl;
outfile << frameCount << std::endl;
}
else{
outfile << frameCount/fps << std::endl;
outfile << frameCount << std::endl;
}
}
if ((capVideo.get(CV_CAP_PROP_POS_FRAMES) + 1) <
capVideo.get(CV_CAP_PROP_FRAME_COUNT))
{
capVideo.read(imgFrame2);
frameCount++;
}
else {
std::cout << "end of video\n";
break;
}
cv::waitKey(33);
}
outfile.close();
return(0);
}
see the code : at every 15th second, it will print the framecount, and the seconds - finally it will print the final number of frames and seconds - i need to plot this as a graph (which will be a linear line)

TIFF files garbled by ArrayFire (C++)

I notice that this simple ArrayFire program is causing loaded TIFF images to be heavily distorted:
#include <iostream>
#include <arrayfire.h>
int main( int argc, char** argv ) {
af::array img = af::loadImage( argv[1] );
double mn, mx;
unsigned idxn, idxx;
af::min( &mn, &idxn, img );
af::max( &mx, &idxx, img );
std::cout << "Image size = " << img.dims()[0] << ", " << img.dims()[1] << '\n';
std::cout << "Data type = " << img.type() << '\n';
std::cout << "Min = " << mn << " (at " << idxn << ")\n";
std::cout << "Max = " << mx << " (at " << idxx << ")\n";
af::saveImage( argv[2], img );
return 0;
}
I then compile and run on a simple (monochrome) image:
./a.out orig.tif out.tif
with the following output:
Image size = 256, 256
Data type = 0
Min = 0 (at 65535)
Max = 81.5025 (at 31356)
When I visualize these images I get the following result:
which of course is not what ArrayFire is expected to do; I would expect it to dump the exact same image out since I didn't make any changes to it. Unfortunately I don't know enough about the TIFF image format or the graphics backend of ArrayFire to understand what is going on. Am I doing something wrong while loading the image? (I followed the ArrayFire documentation for loadImage and saveImage).
I also tried using loadImageNative and saveImageNative alternatively, but the latter returns a 4-layer TIFF image while the original image is only a 1-layer TIFF.
Any help at all from ArrayFire experts would be great.
Thanks!

Magick++ get pixel color. pixelpacket doesnt seem to exist anymore

I'm trying to get the color of a pixel in a jpg image and all the tutorials I can find online say to use pixelpacket which doesnt exist anymore. searching and searching through google i've found that getvirtualpixels might be the new way but that doesnt seem to exist in the newest version either. so how do I get the color of pixels from an image with an image cache? I do not need to set any pixels btw.
Using windows10 and visual studio 2015
PixelPacket structures have been replaced with a pointer to Quantum data type. From Porting to ImageMagick Version 7, the key thing to understand is that a pixel can now have dynamic parts (or "traits" previously called as "channels").
ImageMagick version 7 supports any number of channels from 1 to 32 (and beyond) and simplifies access with a single method that returns an array of pixel channels of type Quantum.
You are now responsible for authoring code that handles deterministic (is that the right term?) pixel channels. For example...
Magick::Image rose("rose:");
const Magick::Quantum * q = rose.getConstPixels(5, 5, 1, 1);
size_t channel_length = rose.channels();
for (size_t k = 0; k < channel_length; k++ ) {
std::cout << "Channel " << k << " = ";
std::cout << q[k] << " of " << QuantumRange << std::endl;
}
/** Outputs
-------
Channel 0 = 13107 of 65535
Channel 1 = 12079 of 65535
Channel 2 = 11308 of 65535
*/
Without seeing your code, or understanding what your finial goal would be, I would guess that you really just want RGB info. If that's the case, the Magick::Image.getColor will work,
Magick::Image rose("rose:");
Magick::Color pixelColor = rose.pixelColor(5, 5);
std::cout << "Channel 0 = ";
std::cout << pixelColor.quantumRed() << " of " << QuantumRange << std::endl;
std::cout << "Channel 1 = ";
std::cout << pixelColor.quantumGreen() << " of " << QuantumRange << std::endl;
std::cout << "Channel 2 = ";
std::cout << pixelColor.quantumBlue() << " of " << QuantumRange << std::endl;
/** Outputs
-------
Channel 0 = 13107 of 65535
Channel 1 = 12079 of 65535
Channel 2 = 11308 of 65535
*/
But! Sometimes it's just easier to export the image data into a format you're already working with. For example, I might only need Green values as single-point float list.
Magick::Image rose("rose:");
Magick::PixelData pixelBlob(rose, "G", Magick::FloatPixel);
float * buffer = (float *)pixelBlob.data();
// ...

Opencv - RTrees algorithm : adding weight to class

I am using OpenCV's implementation of Random Forest algorithm (i.e. RTrees) and am facing a little problem when setting parameters.
I have 5 classes and 3 variables and I want to add weight to classes because the samples sizes for each classes vary a lot.
I took a look at the documentation here and here and it seems that the priors array is the solution, but when I try to give it 5 weights (for my 5 classes) it gives me the following error :
OpenCV Error: One of arguments' values is out of range (Every class weight should be positive) in CvDTreeTrainData::set_data, file /home/sguinard/dev/opencv-2.4.13/modules/ml/src/tree.cpp, line 644
terminate called after throwing an instance of 'cv::Exception'
what(): /home/sguinard/dev/opencv-2.4.13/modules/ml/src/tree.cpp:644: error: (-211) Every class weight should be positive in function CvDTreeTrainData::set_data
If I understand well, it's due to the fact that the priors array have 5 elements. And when I try to give it only 3 elements (as my number of variables) everything works.
According to the documentation, this array should be used to add weight to classes but it actually seems that it is used to add weight to variables...
So, does anyone knows how to add weight to classes on OpenCV's RTrees algorithm ? (I'm working with OpenCV 2.4.13 in c++)
Thanks in advance !
Here is my code :
cv::Mat RandomForest(cv::Mat train_data, cv::Mat response_data, cv::Mat sample_data, int size, int size_predict, float weights[5])
{
#undef CV_TERMCRIT_ITER
#define CV_TERMCRIT_ITER 10
#define ATTRIBUTES_PER_SAMPLE 3
cv::RandomTrees RFTree;
float priors[] = {1,1,1};
CvRTParams RFParams = CvRTParams(25, // max depth
500, // min sample count
0, // regression accuracy: N/A here
false, // compute surrogate split, no missing data
5, // max number of categories (use sub-optimal algorithm for larger numbers)
//priors
weights, // the array of priors (use weights or priors)
true,//false, // calculate variable importance
2, // number of variables randomly selected at node and used to find the best split(s).
100, // max number of trees in the forest
0.01f, // forrest accuracy
CV_TERMCRIT_ITER | CV_TERMCRIT_EPS // termination cirteria
);
cv::Mat varIdx = cv::Mat();
cv::Mat vartype( train_data.cols + 1, 1, CV_8U );
vartype.setTo(cv::Scalar::all(CV_VAR_NUMERICAL));
vartype.at<uchar>(ATTRIBUTES_PER_SAMPLE, 0) = CV_VAR_CATEGORICAL;
cv::Mat sampleIdx = cv::Mat();
cv::Mat missingdatamask = cv::Mat();
for (int i=0; i!=train_data.rows; ++i)
{
for (int j=0; j!=train_data.cols; ++j)
{
if(train_data.at<float>(i,j)<0
|| train_data.at<float>(i,j)>10000
|| !float(train_data.at<float>(i,j)))
{train_data.at<float>(i,j)=0;}
}
}
// Training
std::cout << "Training ....." << std::flush;
bool train = RFTree.train(train_data,
CV_ROW_SAMPLE,//tflag,
response_data,//responses,
varIdx,
sampleIdx,
vartype,
missingdatamask,
RFParams);
if (train){std::cout << " Done" << std::endl;}
else{std::cout << " Failed" << std::endl;return cv::Mat();}
std::cout << "Variable Importance : " << std::endl;
cv::Mat VI = RFTree.getVarImportance();
for (int i=0; i!=VI.cols; ++i){std::cout << VI.at<float>(i) << " - " << std::flush;}
std::cout << std::endl;
std::cout << "Predicting ....." << std::flush;
cv::Mat predict(1,sample_data.rows,CV_32F);
float max = 0;
for (int i=0; i!=sample_data.rows; ++i)
{
predict.at<float>(i) = RFTree.predict(sample_data.row(i));
if (predict.at<float>(i)>max){max=predict.at<float>(i);/*std::cout << predict.at<float>(i) << "-"<< std::flush;*/}
}
// Personnal test due to an error I got (everyone sent to 0)
if (max==0){std::cout << " Failed ... Max value = 0" << std::endl;return cv::Mat();}
std::cout << " Done ... Max value = " << max << std::endl;
return predict;
}

FANN examples give wrong results although training seems successful

Using FANN I can't succeed to run copy&pasted code from FANN's website. I am using FANN version 2.2.0 on Windows 7 and MS Visual Studio 2008. My code for the training program of the XOR example looks like:
#include "floatfann.h"
#include "fann_cpp.h"
#include <ios>
#include <iostream>
#include <iomanip>
#include <string>
using std::cout;
using std::cerr;
using std::endl;
using std::setw;
using std::left;
using std::right;
using std::showpos;
using std::noshowpos;
// Callback function that simply prints the information to cout
int print_callback(FANN::neural_net &net, FANN::training_data &train,
unsigned int max_epochs, unsigned int epochs_between_reports,
float desired_error, unsigned int epochs, void *user_data)
{
cout << "Epochs " << setw(8) << epochs << ". "
<< "Current Error: " << left << net.get_MSE() << right << endl;
return 0;
}
// Test function that demonstrates usage of the fann C++ wrapper
void xor_test()
{
cout << endl << "XOR test started." << endl;
const float learning_rate = 0.7f;
const unsigned int num_layers = 3;
const unsigned int num_input = 2;
const unsigned int num_hidden = 3;
const unsigned int num_output = 1;
const float desired_error = 0.00001f;
const unsigned int max_iterations = 300000;
const unsigned int iterations_between_reports = 1000;
cout << endl << "Creating network." << endl;
FANN::neural_net net;
net.create_standard(num_layers, num_input, num_hidden, num_output);
net.set_learning_rate(learning_rate);
//net.set_activation_steepness_hidden(0.5);
//net.set_activation_steepness_output(0.5);
net.set_activation_function_hidden(FANN::SIGMOID_SYMMETRIC_STEPWISE);
net.set_activation_function_output(FANN::SIGMOID_SYMMETRIC_STEPWISE);
// Set additional properties such as the training algorithm
//net.set_training_algorithm(FANN::TRAIN_QUICKPROP);
// Output network type and parameters
cout << endl << "Network Type : ";
switch (net.get_network_type())
{
case FANN::LAYER:
cout << "LAYER" << endl;
break;
case FANN::SHORTCUT:
cout << "SHORTCUT" << endl;
break;
default:
cout << "UNKNOWN" << endl;
break;
}
net.print_parameters();
cout << endl << "Training network." << endl;
FANN::training_data data;
if (data.read_train_from_file("xor.data"))
{
// ***** MY INPUT
std::string fn;
fn = "xor_read.data";
data.save_train(fn);
fann_type **train_dat;
fann_type **out_dat;
train_dat = data.get_input();
out_dat = data.get_output();
printf("*****************\n");
printf("Printing read data (%d):\n", data.num_input_train_data());
for(unsigned int i = 0; i < data.num_input_train_data(); i++)
{
printf("XOR test (%f,%f) -> %f\n", train_dat[i][0], train_dat[i][1], out_dat[i][0]);
}
printf("*****************\n");
// END: MY INPUT **************
// Initialize and train the network with the data
net.init_weights(data);
cout << "Max Epochs " << setw(8) << max_iterations << ". "
<< "Desired Error: " << left << desired_error << right << endl;
net.set_callback(print_callback, NULL);
net.train_on_data(data, max_iterations,
iterations_between_reports, desired_error);
cout << endl << "Testing network." << endl;
for (unsigned int i = 0; i < data.length_train_data(); ++i)
{
// Run the network on the test data
fann_type *calc_out = net.run(data.get_input()[i]);
cout << "XOR test (" << showpos << data.get_input()[i][0] << ", "
<< data.get_input()[i][2] << ") -> " << *calc_out
<< ", should be " << data.get_output()[i][0] << ", "
<< "difference = " << noshowpos
<< fann_abs(*calc_out - data.get_output()[i][0]) << endl;
}
cout << endl << "Saving network." << endl;
// Save the network in floating point and fixed point
net.save("xor_float.net");
unsigned int decimal_point = net.save_to_fixed("xor_fixed.net");
data.save_train_to_fixed("xor_fixed.data", decimal_point);
cout << endl << "XOR test completed." << endl;
}
}
/* Startup function. Syncronizes C and C++ output, calls the test function
and reports any exceptions */
int main(int argc, char **argv)
{
try
{
std::ios::sync_with_stdio(); // Syncronize cout and printf output
xor_test();
}
catch (...)
{
cerr << endl << "Abnormal exception." << endl;
}
return 0;
}
I commented out :
//net.set_activation_steepness_hidden(0.5);
//net.set_activation_steepness_output(0.5);
otherwise it crashes. The file xor.data :
4 2 1
1 1
-1
-1 -1
-1
-1 1
1
1 -1
1
The output looks odd to me:
XOR test started.
Creating network.
Network Type : LAYER
Input layer : 2 neurons, 1 bias
Hidden layer : 3 neurons, 1 bias
Output layer : 1 neurons
Total neurons and biases : 8
Total connections : 13
Connection rate : 1.000
Network type : FANN_NETTYPE_LAYER
Training algorithm : FANN_TRAIN_RPROP
Training error function : FANN_ERRORFUNC_TANH
Training stop function : FANN_STOPFUNC_MSE
Bit fail limit : 0.350
Learning rate : 0.700
Learning momentum : 0.000
Quickprop decay : -0.000100
Quickprop mu : 1.750
RPROP increase factor : 1.200
RPROP decrease factor : 0.500
RPROP delta min : 0.000
RPROP delta max : 50.000
Cascade output change fraction : 0.010000
Cascade candidate change fraction : 0.010000
Cascade output stagnation epochs : 12
Cascade candidate stagnation epochs : 12
Cascade max output epochs : 150
Cascade min output epochs : 50
Cascade max candidate epochs : 150
Cascade min candidate epochs : 50
Cascade weight multiplier : 0.400
Cascade candidate limit :1000.000
Cascade activation functions[0] : FANN_SIGMOID
Cascade activation functions[1] : FANN_SIGMOID_SYMMETRIC
Cascade activation functions[2] : FANN_GAUSSIAN
Cascade activation functions[3] : FANN_GAUSSIAN_SYMMETRIC
Cascade activation functions[4] : FANN_ELLIOT
Cascade activation functions[5] : FANN_ELLIOT_SYMMETRIC
Cascade activation functions[6] : FANN_SIN_SYMMETRIC
Cascade activation functions[7] : FANN_COS_SYMMETRIC
Cascade activation functions[8] : FANN_SIN
Cascade activation functions[9] : FANN_COS
Cascade activation steepnesses[0] : 0.250
Cascade activation steepnesses[1] : 0.500
Cascade activation steepnesses[2] : 0.750
Cascade activation steepnesses[3] : 1.000
Cascade candidate groups : 2
Cascade no. of candidates : 80
Training network.
*****************
Printing read data (2):
XOR test (0.000000,1.875000) -> 0.000000
XOR test (0.000000,-1.875000) -> 0.000000
*****************
Max Epochs 300000. Desired Error: 1e-005
Epochs 1. Current Error: 0.260461
Epochs 36. Current Error: 7.15071e-006
Testing network.
XOR test (+0, +1.875) -> +5.295e-035, should be +0, difference = 5.295e-035
XOR test (+0, -1.875) -> +0, should be +0, difference = -0
XOR test (+0, -1.875) -> +0, should be +0, difference = -0
XOR test (+0, +1.875) -> +0, should be +0, difference = -0
Saving network.
XOR test completed.
The output after Testing network. looks like :
the training data as well as the test data are interpreted to be (0, +/- 1.875), as you can see in directly below the lines after Printing read data (2) and Testing network..
The (2) after Printing read data is taken from data.num_input_train_data() and my expectation would be to get a (4) since I have four sets of training data.
The "target" seems to be always "0" (see output), although the training data is never zero, but always +/- 1.
A different question has the same odd output hinting towards the training data being interpreted as (0,+/-1.875)->0.0. With this example training (like in my XOR example) also seemed to be successful, but the execution of the ANN (even on the data used for training) returned seemingly random numbers.
I found the answer in FANN - I get incorrect results (near 0) at simply task. It says when including "doublefann.h" one should also link the doublefann lib. This obviously holds for "floatfann.h" and the floatfann lib as well.