I get the wrong result using IBK in weka - weka

I have two data set, one for training data set and another for test data set.
I want to find the first nearest neighbor of the first instance of test data set among training data set, I wrote the following code:
data.setClassIndex(data.numAttributes() - 1);
data1.setClassIndex(data1.numAttributes() - 1);
IBk knn = new IBk();
String[] options = new String[2];
options[0]= "-E";
options[1]= "-I";
knn.setOptions(options);
knn.setKNN(1);
knn.setCrossValidate(true);
knn.buildClassifier(data);
int d = knn.getKNN();
Double c = knn.classifyInstance(data1.instance(0));
System.out.println(d);
System.out.println(c);
I don't know why I get the wrong result. Because the instance that I am looking for the nearest neighbor for, is in training set as well and the result should be the same value as the test instance , but it is not!
My Training dataset:
#relation MyRelation
#attribute Fname {Tina,Alex,Poul,Johan,Sarah}
#attribute Lname {Sansed,Erikson,Nadi,Raj,Maad}
#attribute Status {Single,Maried}
#attribute sex {Fmale,male}
#attribute 'worked years' numeric
#attribute age numeric
#attribute Adults numeric
#attribute tax numeric
#attribute salary numeric
#data
Tina,Sansed,Single,Fmale,14,35,5,362.79,1332.5
Alex,Erikson,Maried,male,14,40,6,0,1245.3
Poul,Nadi,Maried,male,6,35,6,207.32,1150.8
Johan,Raj,Maried,male,29,48,5,0,959
Sarah,Maad,Single,Fmale,16,42,2,0,667.1
Now in Test data set I want to determine the nearest neighbor for instance:
Johan,Raj,Maried,male,29,48,5,0,959
The Result of running the code is:
1
1332.5
It gives me the first instance of the training set every time that I try!

Related

Train and test set are not compatible

I am trying to use Weka to make decisions using arff files, however when i try to classify i recieve the error "Problem evaluating classifier: Train and test set are not compatible".
This is my weather.arff file
#relation weather
#attribute outlook {sunny, overcast, rainy}
#attribute temperature real
#attribute humidity real
#attribute windy {TRUE, FALSE}
#attribute play {yes, no}
#data
sunny,85,85,FALSE,no
sunny,80,90,TRUE,no
overcast,83,86,FALSE,yes
rainy,70,96,FALSE,yes
rainy,68,80,FALSE,yes
rainy,65,70,TRUE,no
overcast,64,65,TRUE,yes
sunny,72,95,FALSE,no
sunny,69,70,FALSE,yes
rainy,75,80,FALSE,yes
sunny,75,70,TRUE,yes
overcast,72,90,TRUE,yes
overcast,81,75,FALSE,yes
rainy,71,91,TRUE,no
and this is my weather.nominal.classify.arff
#relation weather.symbolic
#attribute outlook {sunny, overcast, rainy}
#attribute temperature {hot, mild, cool}
#attribute humidity {high, normal}
#attribute windy {TRUE, FALSE}
#attribute play {yes, no}
#data
overcast,mild,high,TRUE,?
Your attributes are different. In your first file you have
#attribute temperature real
and in your second file you have
#attribute temperature {hot, mild, cool}
Same with humidity. you need to have the attribute definitions completely identical.

WEKA - Compute an average number of non-missing attributes per class attribute

I'm working with a dataset with a lot of missing values. For analytical purposes, I want to be able to compute the average number of non-missing attributes assigned to each class label in the following manner.
Having data like
#relation class
#attribute one {1,2}
#attribute two {1,2}
#attribute three {1,2}
#attribute class {human, animal}
#data
1,Na,Na,human
1,1,Na,human
Na,Na,2,animal
I want to be able to obtain a result like this.
Average non-missing attributes per class label:
- human = 1.5
- animal = 1
Is there any way of doing this in the WEKA explorer?

How to get attributes per classes from Weka

I have 11 classes/categories in my data set. And for every class there are some instances assigned to it. I need to know the attributes/words that Weka extracted per category and the numeric value assigned to each attribute. Is there a way to do that?
This is a sample of arff file format for Weka TRAINING and CLASSIFY:
In this case I'm talking about a portion of file used in Semeval 2014 competition for the Spanish...
#relation Task10EnglishS2014
#attribute PathLenAlign numeric
#attribute ResAlign numeric
#attribute LcAlign numeric
#attribute WupAlign numeric
#attribute Res numeric
#attribute Lc numeric
#attribute DiceSimilarityAttribute numeric
#attribute NumericEvaluation numeric
#data
1,9.5852985,3.637587,1,8.0142254,3.637587,0.75,5.000
1,9.20881283333333,3.637587,1,8.3916004,3.637587,1,5.000
0.625,2.812914,2.754695,0.761905,2.812914,2.754695,0.5,0.292893218813452,0.300
...
piecemeal:
#relation Task10EnglishS2014
#relation + name of the set or experiment
#attribute LcAlign numeric
#attribute + name of attribute + type of attribute
#data
from here to start the instances or vector of values ​​for each input.
This is the training set and this is used for training a model for classify new instances.
In Weka explorer we need to load this file in the Preprocess tab. Them in Classify tab you need to choose a classifier and set Cross-validation with 10 Folds and click to Start button. This will generate a model trained.
The arff classification file must have the following structure:
#relation Task10EnglishS2014
#attribute PathLenAlign numeric
#attribute ResAlign numeric
#attribute LcAlign numeric
#attribute WupAlign numeric
#attribute Res numeric
#attribute Lc numeric
#attribute DiceSimilarityAttribute numeric
#attribute NumericEvaluation numeric
#data
1,9.5852985,3.637587,1,8.0142254,3.637587,0.75,?
1,9.20881283333333,3.637587,1,8.3916004,3.637587,1,?
0.625,2.812914,2.754695,0.761905,2.812914,2.754695,0.5,0.292893218813452,?
...
The ? simbol mean that this is the value to classify.
Them you must select the option: "Supplied test set" and select the file to classify and in "Mode options..." select "Output predictions" and them right click in the model and select "Re-evaluate model on current test set".
In the right panel the results were visualized.

Weka greyed out classifers

I am sorry once again to bring another question up but why is Weka is graying out some classifiers for the data which I bring about. And a sample glipse of the data file is attached as follows:
#relation whatever
#attribute ClearanceFactor numeric
#attribute CrestFactor numeric
#attribute HistogramLB numeric
#attribute HistogramUB numeric
#attribute ImpulseFactor numeric
#attribute KurtVal numeric
#attribute PeakVal numeric
#attribute RMSVal numeric
#attribute Status { Normal }
#data
1 , 0.944758327 , 0.818823375 , 0.835884533 , 0.973802319 , 0.922274575 , 0.836712854 , 0.830582178 , Normal
1 , 0.922118042 , 0.737125289 , 0.762040973 , 0.963101929 , 0.889826729 , 0.762426651 , 0.753675509 , Normal
1 , 0.975667525 , 0.916722849 , 0.924607883 , 0.988490457 , 0.962805959 , 0.925217603 , 0.922378149 , Normal
I got it working by a simple trick. In the last column, I added all of the other possible solution sets as follows and it worked! :D
#relation whatever
#attribute ClearanceFactor numeric
#attribute CrestFactor numeric
#attribute HistogramLB numeric
#attribute HistogramUB numeric
#attribute ImpulseFactor numeric
#attribute KurtVal numeric
#attribute PeakVal numeric
#attribute RMSVal numeric
#attribute Status { Normal, faulty0, faulty1}

Weka : How to prepare test set in weka

I have been using SVM classifier with the following data
#relation whatever
#attribute mfe numeric
#attribute GB numeric
#attribute GTB numeric
#attribute Seeds numeric
#attribute ABP numeric
#attribute AU_Seed numeric
#attribute GC_Seed numeric
#attribute GU_Seed numeric
#attribute UP numeric
#attribute AU numeric
#attribute GC numeric
#attribute GU numeric
#attribute A-U_L numeric
#attribute G-C_L numeric
#attribute G-U_L numeric
#attribute (G+C) numeric
#attribute MFEi1 numeric
#attribute MFEi2 numeric
#attribute MFEi3 numeric
#attribute MFEi4 numeric
#attribute dG numeric
#attribute dP numeric
#attribute dQ numeric
#attribute dD numeric
#attribute Outcome {Yes,No}
#data
-24.3,1,18,2,9,4,3,0.5,8,10,7,1,0.454545455,0.318181818,0.045454545,7,-0.157792208,-0.050206612,-1.104545455,-1.35,-1.104545455,0,0,0,Yes
-24.8,2,15,2,7.5,2,3,1,7,5,8,2,0.208333333,0.333333333,0.083333333,8,-0.129166667,-0.043055556,-0.516666667,-1.653333333,-1.033333333,0,0,0,No
-24.4,1,16,3,5.333333333,1.666666667,2.666666667,1,4,5,8,3,0.217391304,0.347826087,0.130434783,8,-0.132608696,-0.046124764,-1.060869565,-1.525,-1.060869565,0,0,0,Yes
-24.2,1,18,2,9,2,2.5,1,10,5,11,2,0.227272727,0.5,0.090909091,11,-0.1,-0.05,-1.1,-1.344444444,-1.1,0,0,0,Yes
-24.5,3,17,2,8.5,2,3,1,5,6,9,2,0.272727273,0.409090909,0.090909091,9,-0.123737374,-0.050619835,-0.371212121,-1.441176471,-1.113636364,-0.12244898,0,0,Yes
This is my training set . And in this its defined whether my data is yes class or no class. My question is my test data is from unknown source and i dont have idea to what class it belongs. so how to prepare my test set. without the outcome attribute weka is giving the "ereor: Data mismatch " . How to prepare the test set? to separate my variable as Yes and nO class using SVM.
Steps to prepare the test set:
Create a training set in CSV format.
Also create the test set in CSV format with same no. of attributes and same type.
Copy the test set and paste at the end of the training set and save as new CSV file.
Import the saved CSV file in step 3 using Weka>>Explorer>>Preprocess.
In Filter Option Choose filters>>unsupervised>>instances>>Remove Range.
Click the feed which says RemoveRange-R first-last.
Specify the range you want to remove say the training data had 100 values, then select first-100 and Apply the filter.
Save as Arff file and this can be used as a test set.
Then Apply this set. If you still have any errors, write as a reply to this post.
If you don't want to go through hassles, then you can prepare your test set with exact names, data types and data range as in your training set and of course with attribute values. The class attribute will be present but the value should be a question mark (?). For instance, to convert your given training set to a test set the following change can be done`#relation whatever
#relation whatever-TEST
#attribute mfe numeric
#attribute GB numeric
#attribute GTB numeric
#attribute Seeds numeric
#attribute ABP numeric
#attribute AU_Seed numeric
#attribute GC_Seed numeric
#attribute GU_Seed numeric
#attribute UP numeric
#attribute AU numeric
#attribute GC numeric
#attribute GU numeric
#attribute A-U_L numeric
#attribute G-C_L numeric
#attribute G-U_L numeric
#attribute (G+C) numeric
#attribute MFEi1 numeric
#attribute MFEi2 numeric
#attribute MFEi3 numeric
#attribute MFEi4 numeric
#attribute dG numeric
#attribute dP numeric
#attribute dQ numeric
#attribute dD numeric
#attribute Outcome {Yes,No}
#data
-24.3,1,18,2,9,4,3,0.5,8,10,7,1,0.454545455,0.318181818,0.045454545,7,-0.157792208,-0.050206612,-1.104545455,-1.35,-1.104545455,0,0,0,?
-24.8,2,15,2,7.5,2,3,1,7,5,8,2,0.208333333,0.333333333,0.083333333,8,-0.129166667,-0.043055556,-0.516666667,-1.653333333,-1.033333333,0,0,0,?
-24.4,1,16,3,5.333333333,1.666666667,2.666666667,1,4,5,8,3,0.217391304,0.347826087,0.130434783,8,-0.132608696,-0.046124764,-1.060869565,-1.525,-1.060869565,0,0,0,?
-24.2,1,18,2,9,2,2.5,1,10,5,11,2,0.227272727,0.5,0.090909091,11,-0.1,-0.05,-1.1,-1.344444444,-1.1,0,0,0,?
-24.5,3,17,2,8.5,2,3,1,5,6,9,2,0.272727273,0.409090909,0.090909091,9,-0.123737374,-0.050619835,-0.371212121,-1.441176471,-1.113636364,-0.12244898,0,0,?
`
Do we need to replace the values of last attribute with question mark in test data?
I am confused
I did test my data by two methods
removing the values of last attribute and putting ? As a replacement.
I used the test data as it is ( not reming the class attribute)
Whether you are evaluating a trained model on a dataset or trying to make predictions with a trained model, the dataset has to have the exact same structure as the training data (attribute names, attribute types, order of nominal labels). This includes the class attribute.
If you want to test your model, then you need ground truth values to compare the predictions against. Otherwise you cannot generate statistics.
If you want to make predictions, then the class values should be all missing.
For removing the class values, you can either do that manually, or you can use the missing-values-imputation Weka package. Use the weka.filters.unsupervised.attribute.MissingValuesInjection filter in conjunction with the ClassOnly injection scheme.