Train and Test Set are not compatible: comparing 2 csv files - weka
I am now facing a common problem of Weka, "Train and Test Set are not compatible".
I had a large data sets in csv format. As I got such error I reduce my data set(for practice purpose) to only 2 data in train set and 1 data in test set. I am using j48 classifier.
I have checked that same data format appears in csv and the same column title has also appeared.
The train and test sets are given below:
Train Set:
Stop Words/Punctuations1(L),Stop Words/Punctuations/Stemming1(L),Stop Words/Punctuations/Lammetizing1(L),Stop Words/Punctuations1(NC),Stop Words/Punctuations/Stemming1(NC),Stop Words/Punctuations/Lammetizing1(NC),Stop Words/Punctuations2(L),Stop Words/Punctuations/Stemming2(L),Stop Words/Punctuations/Lammetizing2(L),Stop Words/Punctuations2(NC),Stop Words/Punctuations/Stemming2(NC),Stop Words/Punctuations/Lammetizing2(NC),Stop Words/Punctuations3(L),Stop Words/Punctuations/Stemming3(L),Stop Words/Punctuations/Lammetizing3(L),Stop Words/Punctuations3(NC),Stop Words/Punctuations/Stemming3(NC),Stop Words/Punctuations/Lammetizing3(NC),Stop Words/Punctuations4(L),Stop Words/Punctuations/Stemming4(L),Stop Words/Punctuations/Lammetizing4(L),Stop Words/Punctuations4(NC),Stop Words/Punctuations/Stemming4(NC),Stop Words/Punctuations/Lammetizing4(NC),Stop Words/Punctuations5(L),Stop Words/Punctuations/Stemming5(L),Stop Words/Punctuations/Lammetizing5(L),Stop Words/Punctuations5(NC),Stop Words/Punctuations/Stemming5(NC),Stop Words/Punctuations/Lammetizing5(NC),(No Stop Words/Punctuations)1(L),(No Stop Words/Punctuations)/Stemming1(L),(No Stop Words/Punctuations)/Lammetizing1(L),(No Stop Words/Punctuations)1(NC),(No Stop Words/Punctuations)/Stemming1(NC),(No Stop Words/Punctuations)/Lammetizing1(NC),(No Stop Words/Punctuations)2(L),(No Stop Words/Punctuations)/Stemming2(L),(No Stop Words/Punctuations)/Lammetizing2(L),(No Stop Words/Punctuations)2(NC),(No Stop Words/Punctuations)/Stemming2(NC),(No Stop Words/Punctuations)/Lammetizing2(NC),(No Stop Words/Punctuations)3(L),(No Stop Words/Punctuations)/Stemming3(L),(No Stop Words/Punctuations)/Lammetizing3(L),(No Stop Words/Punctuations)3(NC),(No Stop Words/Punctuations)/Stemming3(NC),(No Stop Words/Punctuations)/Lammetizing3(NC),(No Stop Words/Punctuations)4(L),(No Stop Words/Punctuations)/Stemming4(L),(No Stop Words/Punctuations)/Lammetizing4(L),(No Stop Words/Punctuations)4(NC),(No Stop Words/Punctuations)/Stemming4(NC),(No Stop Words/Punctuations)/Lammetizing4(NC),(No Stop Words/Punctuations)5(L),(No Stop Words/Punctuations)/Stemming5(L),(No Stop Words/Punctuations)/Lammetizing5(L),(No Stop Words/Punctuation)5(NC),(No Stop Words/Punctuations)/Stemming5(NC),(No Stop Words/Punctuations)/Lammetizing5(NC),No Stop Words1(L),No Stop Words/Stemming1(L),No Stop Words/Lammetizing1(L),No Stop Words1(NC),No Stop Words/Stemming1(NC),No Stop Words/Lammetizing1(NC),No Stop Words2(L),No Stop Words/Stemming2(L),No Stop Words/Lammetizing2(L),No Stop Words2(NC),No Stop Words/Stemming2(NC),No Stop Words/Lammetizing2(NC),No Stop Words3(L),No Stop Words/Stemming3(L),No Stop Words/Lammetizing3(L),No Stop Words3(NC),No Stop Words/Stemming3(NC),No Stop Words/Lammetizing3(NC),No Stop Words4(L),No Stop Words/Stemming4(L),No Stop Words/Lammetizing4(L),No Stop Words4(NC),No Stop Words/Stemming4(NC),No Stop Words/Lammetizing4(NC),No Stop Words5(L),No Stop Words/Stemming5(L),No Stop Words/Lammetizing5(L),No Stop Words5(NC),No Stop Words/Stemming5(NC),No Stop Words/Lammetizing5(NC),No Punctuations1(L),No Punctuations/Stemming1(L),No Punctuations/Lammetizing1(L),No Punctuations1(NC),No Punctuations/Stemming1(NC),No Punctuations/Lammetizing1(NC),No Punctuations2(L),No Punctuations/Stemming2(L),No Punctuations/Lammetizing2(L),No Punctuations2(NC),No Punctuations/Stemming2(NC),No Punctuations/Lammetizing2(NC),No Punctuations3(L),No Punctuations/Stemming3(L),No Punctuations/Lammetizing3(L),No Punctuations3(NC),No Punctuations/Stemming3(NC),No Punctuations/Lammetizing3(NC),No Punctuations4(L),No Punctuations/Stemming4(L),No Punctuations/Lammetizing4(L),No Punctuations4(NC),No Punctuations/Stemming4(NC),No Punctuations/Lammetizing4(NC),No Punctuations5(L),No Punctuations/Stemming5(L),No Punctuations/Lammetizing5(L),No Punctuations5(NC),No Punctuations/Stemming5(NC),No Punctuations/Lammetizing5(NC),Class
1.00,2.00,3.00,2.00,4.00,4.20,4.80,5.40,6.00,6.60,7.20,7.80,8.40,9.00,9.60,10.20,10.80,11.40,12.00,12.60,13.20,13.80,14.40,15.00,15.60,16.20,16.80,17.40,18.00,18.60,19.20,19.80,20.40,21.00,21.60,22.20,22.80,23.40,24.00,24.60,25.20,25.80,26.40,27.00,27.60,28.20,28.80,29.40,30.00,30.60,31.20,31.80,32.40,33.00,33.60,34.20,34.80,35.40,36.00,36.60,37.20,37.80,38.40,39.00,39.60,40.20,40.80,41.40,42.00,42.60,43.20,43.80,44.40,45.00,45.60,46.20,46.80,47.40,48.00,48.60,49.20,49.80,50.40,51.00,51.60,52.20,52.80,53.40,54.00,54.60,55.20,55.80,56.40,57.00,57.60,58.20,58.80,59.40,60.00,60.60,61.20,61.80,62.40,63.00,63.60,64.20,64.80,65.40,66.00,66.60,67.20,67.80,68.40,69.00,69.60,70.20,70.80,71.40,72.00,72.60,HR
1.00,2.00,3.00,2.00,4.00,4.20,4.80,5.40,6.00,6.60,7.20,7.80,8.40,9.00,9.60,10.20,10.80,11.40,12.00,12.60,13.20,13.80,14.40,15.00,15.60,16.20,16.80,17.40,18.00,18.60,19.20,19.80,20.40,21.00,21.60,22.20,22.80,23.40,24.00,24.60,25.20,25.80,26.40,27.00,27.60,28.20,28.80,29.40,30.00,30.60,31.20,31.80,32.40,33.00,33.60,34.20,34.80,35.40,36.00,36.60,37.20,37.80,38.40,39.00,39.60,40.20,40.80,41.40,42.00,42.60,43.20,43.80,44.40,45.00,45.60,46.20,46.80,47.40,48.00,48.60,49.20,49.80,50.40,51.00,51.60,52.20,52.80,53.40,54.00,54.60,55.20,55.80,56.40,57.00,57.60,58.20,58.80,59.40,60.00,60.60,61.20,61.80,62.40,63.00,63.60,64.20,64.80,65.40,66.00,66.60,67.20,67.80,68.40,69.00,69.60,70.20,70.80,71.40,72.00,72.60,LR
Test Set:
Stop Words/Punctuations1(L),Stop Words/Punctuations/Stemming1(L),Stop Words/Punctuations/Lammetizing1(L),Stop Words/Punctuations1(NC),Stop Words/Punctuations/Stemming1(NC),Stop Words/Punctuations/Lammetizing1(NC),Stop Words/Punctuations2(L),Stop Words/Punctuations/Stemming2(L),Stop Words/Punctuations/Lammetizing2(L),Stop Words/Punctuations2(NC),Stop Words/Punctuations/Stemming2(NC),Stop Words/Punctuations/Lammetizing2(NC),Stop Words/Punctuations3(L),Stop Words/Punctuations/Stemming3(L),Stop Words/Punctuations/Lammetizing3(L),Stop Words/Punctuations3(NC),Stop Words/Punctuations/Stemming3(NC),Stop Words/Punctuations/Lammetizing3(NC),Stop Words/Punctuations4(L),Stop Words/Punctuations/Stemming4(L),Stop Words/Punctuations/Lammetizing4(L),Stop Words/Punctuations4(NC),Stop Words/Punctuations/Stemming4(NC),Stop Words/Punctuations/Lammetizing4(NC),Stop Words/Punctuations5(L),Stop Words/Punctuations/Stemming5(L),Stop Words/Punctuations/Lammetizing5(L),Stop Words/Punctuations5(NC),Stop Words/Punctuations/Stemming5(NC),Stop Words/Punctuations/Lammetizing5(NC),(No Stop Words/Punctuations)1(L),(No Stop Words/Punctuations)/Stemming1(L),(No Stop Words/Punctuations)/Lammetizing1(L),(No Stop Words/Punctuations)1(NC),(No Stop Words/Punctuations)/Stemming1(NC),(No Stop Words/Punctuations)/Lammetizing1(NC),(No Stop Words/Punctuations)2(L),(No Stop Words/Punctuations)/Stemming2(L),(No Stop Words/Punctuations)/Lammetizing2(L),(No Stop Words/Punctuations)2(NC),(No Stop Words/Punctuations)/Stemming2(NC),(No Stop Words/Punctuations)/Lammetizing2(NC),(No Stop Words/Punctuations)3(L),(No Stop Words/Punctuations)/Stemming3(L),(No Stop Words/Punctuations)/Lammetizing3(L),(No Stop Words/Punctuations)3(NC),(No Stop Words/Punctuations)/Stemming3(NC),(No Stop Words/Punctuations)/Lammetizing3(NC),(No Stop Words/Punctuations)4(L),(No Stop Words/Punctuations)/Stemming4(L),(No Stop Words/Punctuations)/Lammetizing4(L),(No Stop Words/Punctuations)4(NC),(No Stop Words/Punctuations)/Stemming4(NC),(No Stop Words/Punctuations)/Lammetizing4(NC),(No Stop Words/Punctuations)5(L),(No Stop Words/Punctuations)/Stemming5(L),(No Stop Words/Punctuations)/Lammetizing5(L),(No Stop Words/Punctuation)5(NC),(No Stop Words/Punctuations)/Stemming5(NC),(No Stop Words/Punctuations)/Lammetizing5(NC),No Stop Words1(L),No Stop Words/Stemming1(L),No Stop Words/Lammetizing1(L),No Stop Words1(NC),No Stop Words/Stemming1(NC),No Stop Words/Lammetizing1(NC),No Stop Words2(L),No Stop Words/Stemming2(L),No Stop Words/Lammetizing2(L),No Stop Words2(NC),No Stop Words/Stemming2(NC),No Stop Words/Lammetizing2(NC),No Stop Words3(L),No Stop Words/Stemming3(L),No Stop Words/Lammetizing3(L),No Stop Words3(NC),No Stop Words/Stemming3(NC),No Stop Words/Lammetizing3(NC),No Stop Words4(L),No Stop Words/Stemming4(L),No Stop Words/Lammetizing4(L),No Stop Words4(NC),No Stop Words/Stemming4(NC),No Stop Words/Lammetizing4(NC),No Stop Words5(L),No Stop Words/Stemming5(L),No Stop Words/Lammetizing5(L),No Stop Words5(NC),No Stop Words/Stemming5(NC),No Stop Words/Lammetizing5(NC),No Punctuations1(L),No Punctuations/Stemming1(L),No Punctuations/Lammetizing1(L),No Punctuations1(NC),No Punctuations/Stemming1(NC),No Punctuations/Lammetizing1(NC),No Punctuations2(L),No Punctuations/Stemming2(L),No Punctuations/Lammetizing2(L),No Punctuations2(NC),No Punctuations/Stemming2(NC),No Punctuations/Lammetizing2(NC),No Punctuations3(L),No Punctuations/Stemming3(L),No Punctuations/Lammetizing3(L),No Punctuations3(NC),No Punctuations/Stemming3(NC),No Punctuations/Lammetizing3(NC),No Punctuations4(L),No Punctuations/Stemming4(L),No Punctuations/Lammetizing4(L),No Punctuations4(NC),No Punctuations/Stemming4(NC),No Punctuations/Lammetizing4(NC),No Punctuations5(L),No Punctuations/Stemming5(L),No Punctuations/Lammetizing5(L),No Punctuations5(NC),No Punctuations/Stemming5(NC),No Punctuations/Lammetizing5(NC),Class
25.41,25.41,24.86,0.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,HR
Can anyone advise me regarding this.
Thank You,
Well, I am not sure about your whole data set, but for sample you have posted I think it would be because in you test set is only one class label. I just copied the line in test set and add second label (LR) and everything works. I guess that you have to have same class labels in train and test set to evaluate classifier. Good luck.
Related
ESP32/FreeRTOS, how to stop currently running task when the new one was triggered (avoiding overlap)
I'm working on code to control 2 module relay regarding door access. I'm looking for the way to stop the currently running tasks, before running the new one (the same task). All I want is to avoid overlap.
void TaskOpenManRoom(void *parameter){
Serial.println("Opening men room");
digitalWrite(manRelay, LOW);
vTaskDelay(6000 / portTICK_PERIOD_MS);
digitalWrite(manRelay, HIGH);
Serial.println("Closing men room");
vTaskDelete(NULL);
};
xTaskCreate(
TaskOpenManRoom,
"TaskOpenManRoom",
1000,
(void *) &man,
1,
&TaskMen
);
My goal is to extend the time when the door should be opened. So basically when the first task was triggered and then after some while the second one, the door should stay opened another 6000ms.
In mu current code, when the second task is called like in the middle of the first one, the door get closed because of first task calling digitalWrite(manRelay, HIGH);
I would appreciate the hint how I can kill the first task when the second been triggered.
Tasks are meant to be long-running, because they are relatively heavyweight. Don't start and end tasks for each user activity and don't delay them for extended periods of time.
You don't need any task at all for your functionality, you just need a timer to perform the closing activity after 6000 ms. Then you can reset it whenever you need.
TimerHandle_t closeManRoomTimer;
void OpenManRoom() {
xTimerReset(closeManRoomTimer, 100); // <------ (re)arm the close timer
Serial.println("Opening men room");
digitalWrite(manRelay, LOW);
};
void CloseManRoom(TimerHandle_t) {
Serial.println("Closing men room");
digitalWrite(manRelay, HIGH);
};
// during program startup, setup a one-shot close timer
closeManRoomTimer = xTimerCreate("closeManRoomTimer", pdMS_TO_TICKS(6000), pdFALSE, 0, &CloseManRoom);
I would not kill the first task when the second starts.
If you use a task at all, I'd rewrite the task to something along this general line:
cast parameter to pointer to uint32
atomic increment open count, and if it was zero {
open the door
repeat {
sleep six seconds
} atomic decrement count, and exit loop if it was 1
close the door
}
exit the task
...and when you create the task, pass a pointer to a uint32_t for it to use to store the open count.
So the task starts by atomically incrementing the open count, which returns the value that was previously in the open count. If that was zero, it means the door is currently closed. In that case, we open it and got to sleep.
If the task runs again while it's sleeping, the open count will now be one. We immediately increment that, but when we check the previous value, it was 1, so we don't try to open the door again--we just skip all the stuff in the if statement, and exit the task.
When the first instance of the task wakes up, it decrements the count, and it it was 1, it exits the loop, closes the door, and exits the task. But if the task ran again while it was sleeping, the count will still be greater than 1, so it will stay in the loop and sleep some more.
This is open to a little bit of optimization. As it stands right now, it sleeps a fixed period of time (six seconds) even if the current open count is greater than 1. If the task as expensive enough to justify a little extra work, we could do an atomic exchange, to retrieve the current open count and set the open count to 0, multiply the retrieved value by 6000, then sleep for that long. That adds quite a bit of extra complexity though, and in this case, the benefit would be much too small to justify it.
This does depend on our not running the task more than 4 billion times while the door is open. If we did, our atomic increment would overflow, and the code would misbehave. For the case at hand (and most others) this is unlikely to be a problem. In the rare situation where it might be, the obvious fix is a 64-bit variable (and 64-bit atomic increment and decrement). Incrementing the variable until a 64-bit variable overflows is generally not a realistic possibility (e.g., if you incremented at 1 GHz, it would take centuries).
Many ways: use vTaskDelay which puts the task in the not running state (it is not blocking Wait for, mutex semaphore, queue or task notification from another task. I would appreciate the hint how I can kill the first task when the second been triggered. It will kill current task: vTaskDelete(NULL);
lauterbach scripting: checking if program stops at breakpoint
I have to check by way of a script if a program stops at a breakpoint: example: break.set func1 /Program Go IF (program stops at breakpoint) ( do smth ) I am new to this language and I cannot seem to find relevant information on google. Thank you
If you don't know whether the program stops at the program you need to wait a certain (defined by you) amount of time and then check the run-state of the processor, and if stopped then check whether the PC is at your symbol. ; set breakpoint Break.Set func1 /Program ; start processor Go ; wait until processor stopped or 5 seconds elasped WAIT !STATE.RUN() 5.0s IF STATE.RUN() ( PRINT %ERROR "Processor still running!" ENDDO ) ; check PC IF R(PC)!=sYmbol.BEGIN(func1) ( PRINT %ERROR "Processor stopped but wrong PC!" ENDDO ) PRINT "Test passed!" ENDDO
If you actual goal is to execute a script when a breakpoint gets hit, I recommend to used the option /CMD of the Break.Set. E.g.: Break.Set func1 /Program /CMD "DO smth.cmm"
Correct way to catch SIGINT and cleanup?
Writing a cli tool that on startup turns on the OS X web proxy and on shutdown I'd like to turn it off again. What's the correct way to catch SIGINT and perform app cleanup? Tried the following and it traces the message but does not run the system command or exit:
Signal::INT.trap do
puts "trap"
fork do
system "networksetup -setwebproxystate Wi-Fi off"
end
exit
end
This code does exit but gives an 'Invalid memory access' error
at_exit do
fork do
system "networksetup -setwebproxystate Wi-Fi off"
end
end
LibC.signal Signal::INT.value, ->(s : Int32) { exit }
Invalid memory access (signal 10) at address 0x10d3a8e00
[0x10d029b4b] *CallStack::print_backtrace:Int32 +107
[0x10d0100d5] __crystal_sigfault_handler +181
[0x7fff6c5b3b3d] _sigtramp +29
UPDATE
Here's the complete 'app' using Signal::INT.trap, for me running that will correctly turn on and off the OS X proxy settings but the loop will continue to run after the interrupt signal.
fork do
system "networksetup -setwebproxy Wi-Fi 127.0.0.1 4242"
end
Signal::INT.trap do
puts "trap"
fork do
system "networksetup -setwebproxystate Wi-Fi off"
end
exit
end
loop do
sleep 1
puts "foo"
end
You can use a Fibers? spawn do system "networksetup -setwebproxy Wi-Fi 127.0.0.1 4242" end sleep 0.1 Signal::INT.trap do puts "trap" spawn do system "networksetup -setwebproxystate Wi-Fi off" end sleep 0.1 exit end loop do sleep 1 puts "foo" end
IMHO, the trouble is from crystal-lang's fork, which has some strange semantic meaning. When you tried to start a working process to run system call, crystal duplicated the loop too... And when exit is executed, the first loop exited, not the forked one. To verify this, you can write some sleep into the fork and INT.trap block like this: fork do system "echo \"start\"" end Signal::INT.trap do puts "trap" fork do system "echo \"off\"" sleep 15 end sleep 20 exit end loop do sleep 1 puts "foo" end Then try to watch the result of ps command continuously. Alternative approach has been answered by #Sergey Fedorov, using fiber. Further reading: Process.fork has dangerous semantics
Abort execution during OMNET simulation
I have 4 nodes in my simulation, in the code i check on a flag.
If it is become true during the run of any node then >> i want to stop the execution for all the nodes and go to execute finish method for each node.
I try to do as following but it doesn't call finish method!
if(flag == true)
{ finish();
abort();
}
How i can do it?
Thanks,
Have you tried this command? endSimulation(); It ends the simulation, however I am not sure if it runs finish method or not..
What can be adjusted in this simple code to make signal change in fsm
Well i have process a in my main component and process b in my other sub component(inmplemented in the main one). both process a and b have only the clock in their sensitivity list: process a control eneable signal called ready which if 1 process b can work , 0 process b will do nothing. Problem is in process a , when process a changes value of enable signal to 0 , it has to take to the next clock cycle to change so process b ends up and run an extra clock cycle. a:process(clk) begin if(rising_edge(clk)) then if(output/=old_output) then enable<='0'; end if; end if; end process; b:process(clk) begin if(rising_edge(clk)) then if(enable='1') then --do anything end if; end if; end process;
The reason is that the value is latched/sampled at the exact rising_edge of the clock. At that time, 'enable' is still equal to one. In that simulation delta, enabled will get the value zero, but it won't be available until AFTER the first delta. This is also true for when enable BECOMES one (given that it is also generated on a rising clock edge), the process will latch the value exactly when clock rises, and in the simulator, enabled will look high for a whole clock period, even though "--do anything" will not happen. You can think of this as real electrical circuits instead of a programming language. Consider that the evaluation of "output/=old_output" will consume time, and that you as a designer want that to be DONE before the next rising clock edge. Hope this helps, but this is how the language works. I could give you a better answer if both the setting and resetting of the enable.