Editing a text file in python? - python-2.7

I have this in a text file :
Rubble HM3 80 HM2 90 HM4 92
Bunny HM2 92 HM5 70 HM1 98
Duck HM1 86 HM3 100 HM2 93 HM4 94
Chipmunk HM4 96 HM1 86
Simpson HM3 70 HM1 90 Test1 90
and i want to write a code that changes it to this :
Name | HM1 | HM2 | HM3 | HM4 | Avg. |
________________________________________________
Bunny | 98 | 92 | 0 | 0 | 47.50 |
Chipmunk | 86 | 0 | 0 | 96 | 45.50 |
Duck | 86 | 93 | 100 | 94 | 93.25 |
Rubble | 0 | 90 | 80 | 92 | 65.50 |
Simpson | 90 | 0 | 70 | 0 | 40.00 |
so far :
my_file=open("C:/python27/tools/student_grades.txt", "r+")
my_file_pointer=my_file.read()
for lines in my_file_pointer:
x=my_file_pointer.replace("HM2","|")
print x
Go Easy first time programmer . :)
and if i use the replace function how can i print it all at once and then sort it under every subject "HM1" ?

Related

Sort by two or more variables

I'm trying to sort by ID and then by Date.
What I have:
| ID | Date |
| ----------------------|
| 112 | 2013-01-01 |
| 112 | 2013-01-15 |
| 113 | 2012-01-01 |
| 112 | 2014-02-13 |
| 112 | 2013-01-02 |
| 113 | 2011-01-11 |
What I need:
| ID | Date |
| ----------------------|
| 112 | 2013-01-01 |
| 112 | 2013-01-02 |
| 112 | 2013-01-15 |
| 112 | 2014-02-13 |
| 113 | 2011-01-11 |
| 113 | 2012-01-01 |
My problem is that I only know how to sort by ID or Date.

More generally:
clear
input id foo
1 56
1 34
2 13
1 67
1 22
2 89
2 61
2 76
end
sort id (foo)
list, sepby(id)
+----------+
| id foo |
|----------|
1. | 1 22 |
2. | 1 34 |
3. | 1 56 |
4. | 1 67 |
|----------|
5. | 2 13 |
6. | 2 61 |
7. | 2 76 |
8. | 2 89 |
+----------+
In a more advanced programming context you can use the same syntax with bysort.

Merging two variables

I have the following data:
I want to turn the data in the upper panel into the data in the lower one.
For each origin group, I want to add one line with destination value -1, and var value from varnew.
I tried to find if there is a command which adds one row so that I can do something like:
bysort origin: addrow
However, it seems there isn't any such thing.

Using your toy example data:
clear
input destination origin var varnew
0 111 124 .
111 111 671 168
0 222 623 .
222 222 768 865
end
list, abbreviate(15)
+-------------------------------------+
| destination origin var varnew |
|-------------------------------------|
1. | 0 111 124 . |
2. | 111 111 671 168 |
3. | 0 222 623 . |
4. | 222 222 768 865 |
+-------------------------------------+
expand 2 if varnew != .
sort origin destination
list, abbreviate(15)
+-------------------------------------+
| destination origin var varnew |
|-------------------------------------|
1. | 0 111 124 . |
2. | 111 111 671 168 |
3. | 111 111 671 168 |
4. | 0 222 623 . |
5. | 222 222 768 865 |
|-------------------------------------|
6. | 222 222 768 865 |
+-------------------------------------+
The following works for me:
bysort origin: replace destination = -1 if destination[_n] == destination[_n+1] & !missing(varnew)
bysort origin: replace var = varnew if var[_n] == var[_n+1] & !missing(varnew)
list destination origin var, abbreviate(15)
+----------------------------+
| destination origin var |
|----------------------------|
1. | 0 111 124 |
2. | -1 111 168 |
3. | 111 111 671 |
4. | 0 222 623 |
5. | -1 222 865 |
|----------------------------|
6. | 222 222 768 |
+----------------------------+

Conditionally create new observations

I have data in the following format (there are a lot more variables):
year ID Dummy
1495 65 1
1496 65 1
1501 65 1
1502 65 1
1520 65 0
1522 65 0
What I am trying to achieve is conditionally create new observations that fills in the data between two points in time conditional on a dummy. If the dummy is equal to 1, the data is supposed to be filled in. If the variable is equal to 0 then it shall not be filled in.
For example:
year ID Dummy
1495 65 1
1496 65 1
1497 65 1
1498 65 1
.
.
1501 65 1
1502 65 1
1503 65 1
1504 65 1
.
.
.
1520 65 0
1522 65 0

Here's one way to do this:
clear
input year id dummy
1495 65 1
1496 65 1
1501 65 1
1502 65 1
1520 65 0
1522 65 0
end
generate tag = year[_n] != year[_n+1] & dummy == 1
generate delta = year[_n] - year[_n+1] if tag
replace delta = . if abs(delta) == 1
expand abs(delta) if tag & delta != .
sort year
bysort year: egen seq = seq() if delta != .
replace seq = seq - 1
replace seq = 0 if seq == .
replace year = year + seq if year != .
drop tag delta seq
The above code snippet will produce:
list
+-------------------+
| year id dummy |
|-------------------|
1. | 1495 65 1 |
2. | 1496 65 1 |
3. | 1497 65 1 |
4. | 1498 65 1 |
5. | 1499 65 1 |
|-------------------|
6. | 1500 65 1 |
7. | 1501 65 1 |
8. | 1502 65 1 |
9. | 1503 65 1 |
10. | 1504 65 1 |
|-------------------|
11. | 1505 65 1 |
12. | 1506 65 1 |
13. | 1507 65 1 |
14. | 1508 65 1 |
15. | 1509 65 1 |
|-------------------|
16. | 1510 65 1 |
17. | 1511 65 1 |
18. | 1512 65 1 |
19. | 1513 65 1 |
20. | 1514 65 1 |
|-------------------|
21. | 1515 65 1 |
22. | 1516 65 1 |
23. | 1517 65 1 |
24. | 1518 65 1 |
25. | 1519 65 1 |
|-------------------|
26. | 1520 65 0 |
27. | 1522 65 0 |
+-------------------+

Train our own classifier

Now I am training my own classifier.So for that I am using traincascade.But when I am giving this command 'opencv_traincascade -data facedet -vec vecfile.vec -bg negative.txt -npos 2650 -nneg 581 -nstages 20 -w 20 -h 20' it shows error like this.
PARAMETERS:
cascadeDirName: facedet
vecFileName: vecfile.vec
bgFileName: negative.txt
numPos: 2000
numNeg: 1000
numStages: 20
precalcValBufSize[Mb] : 256
precalcIdxBufSize[Mb] : 256
stageType: BOOST
featureType: HAAR
sampleWidth: 20
sampleHeight: 20
boostType: GAB
minHitRate: 0.995
maxFalseAlarmRate: 0.5
weightTrimRate: 0.95
maxDepth: 1
maxWeakCount: 100
mode: BASIC
===== TRAINING 0-stage =====
<BEGIN
POS count : consumed 2000 : 2000
NEG count : acceptanceRatio 1000 : 1
Precalculation time: 3
+----+---------+---------+
| N | HR | FA |
+----+---------+---------+
| 1| 1| 1|
+----+---------+---------+
| 2| 1| 1|
+----+---------+---------+
| 3| 1| 1|
+----+---------+---------+
| 4| 1| 1|
+----+---------+---------+
| 5| 1| 1|
+----+---------+---------+
| 6| 0.9955| 0.391|
+----+---------+---------+
END>
Parameters can not be written, because file facedet/params.xml can not be opened.
What is this error.I don't understand.Any one help me to solve this.
Positive samples:
/home/arya/myown/Positive/images18413.jpeg 1 1 1 113 33
/home/arya/myown/Positive/images1392.jpeg 1 113 33 107 133
/home/arya/myown/Positive/face841.jpeg 1 185 93 35 73
/home/arya/myown/Positive/images866.jpeg 2 121 26 64 68 121 26 88 123
/home/arya/myown/Positive/images83.jpeg 1 102 13 107 136
/home/arya/myown/Positive/images355.jpeg 2 92 16 224 25 92 16 117 130
/home/arya/myown/Positive/images888.jpeg 1 108 29 116 71
/home/arya/myown/Positive/images2535.jpeg 1 108 29 111 129
/home/arya/myown/Positive/images18221.jpeg 1 110 34 109 124
/home/arya/myown/Positive/images1127.jpeg 1 110 34 92 104
/home/arya/myown/Positive/images18357.jpeg 1 103 27 142 133
/home/arya/myown/Positive/images889.jpeg 1 86 25 134 124
Negative samples:
./Negative/face150.jpeg
./Negative/face1051.jpeg
./Negative/Pictures174.jpeg
./Negative/Pictures160.jpeg
./Negative/Pictures34.jpeg
./Negative/face130.jpeg
./Negative/face1.jpeg
./Negative/Pictures319.jpeg
./Negative/face1120.jpeg
./Negative/Pictures317.jpeg
./Negative/face1077.jpeg
./Negative/Pictures93.jpeg
./Negative/Pictures145.jpeg
./Negative/face1094.jpeg
./Negative/Pictures7.jpeg

Please be sure that you have already created the folder "facedet" before training your classifier as it does not create it by itself.
It needs this folder to create "params.xml" file in inside it.

memcpy used in DLL and loaded in another application, but the memory is changing itself, did i do something wrong?

This problem is fixed, thanks for pointing out the mistake in declaration of the typedef.
in which i am missing _stdcall to it. After digging some materials it is something about calling conventions, somehow it treat the stack in different way. I am not really sure, but it seems fixed it. Thanks
Greetings and Hello everyone,
I am currently working on an DLL which provide Serial Port Communication and then write to the buffer the result.
Lets say the DLL function part is DLL,
and Application part be EXE.
After certain process, I will be able to create a string in DLL function.
And then i used memcpy to copy the buffer in DLL to the char* pointer the EXE provide
However, it is fine sometime with all bytes copied ,
and the EXE can dump the result just the same as the DLL has written,
but sometimes it has shown the correct values in DLL function part,
but in EXE the memory section has become gibberish.
I want to know if i am missing some obvious part, as i am not really experienced in memory and how Extern Modules works with it.
I will now post DLL part and EXE part code for your reference.
DLL Part: compiled in VC++ 2012
dumpstr : dump the first 20 bytes from the address given.
**mp_get_var is exported via a .def file and exported to a DLL**
void dumpstr(char* charC, char* desc) {
printf("\n= S T A R T ===\nDLL Dumping the %s Array...\n", desc);
int i;
for(i=0; i<20; i++) {
printf("%p:%3d | ", charC+i, *(charC+i));
if((i+1)%4==0) printf("\n");
}
printf("\n==E N D====\n");
}
int _stdcall mp_get_var(char* inSI, char* cParam, int paramIndex, char *outSI) {
//std::string returnValue = MyConvert::get_fieldvalue2(inSI,cParam, paramIndex);
printf("ON DLL START, INSI ADDR (%p)\n", inSI);
printf("ON DLL START, OUTSI ADDR (%p)\n", outSI);
dumpstr(outSI, "PREV OUTSI");
//printf("DLL received: %s\n", inSI);
//dumpstr(inSI, "THIS > INSI");
std::string returnValue = "EPTATROPICAL";
memcpy(outSI,returnValue.c_str(),returnValue.length());
//*(outSI + returnValue.length()) = '\0';
//printf("DLL copied outSI: %s\n", outSI);
dumpstr(outSI, "THIS COPIED OUTSI");
printf("ON DLL END, INSI ADDR (%p)\n", inSI);
printf("ON DLL END, OUTSI ADDR (%p)\n", outSI);
return returnValue.length();
}
EXE Part: compiled in Dev-C++
I import the library using the following statements:
and i do two request to the DLL function which respectively using Result2 and Result3 as output buffer.
typedef int (*mp_get_var)(char * , char* , int , char *);
int main() {
HINSTANCE hLib = LoadLibrary("MPComsDLL.dll");
MP_GET_VAR = (mp_get_var)GetProcAddress((HMODULE)hLib, "mp_get_var");
sprintf(cRequest, "EPAMTOH");
//using **Result2** char array for result buffer.
returnval = MP_GET_VAR(Result, cRequest, 1, Result2);
printf("ON EXE, OUTSI address : %p", Result2);
printf("%s return length: %d\n", cRequest, returnval);
dumpstr(Result2);
sprintf(cRequest, "MERCHANTID");
//using **Result3** char array for result buffer.
returnval = MP_GET_VAR(Result, cRequest, 1, Result3);
printf("ON EXE, OUTSI address : %p", Result3);
printf("%s return length: %d\n", cRequest, returnval);
dumpstr(Result3);
}
However, here is the screen dump:
For the first call
ON DLL START, INSI ADDR (0028F8B0)
ON DLL START, OUTSI ADDR (0028F680)
= S T A R T ===
DLL Dumping the PREV OUTSI Array...
0028F680: 0 | 0028F681: 0 | 0028F682: 0 | 0028F683: 0 |
0028F684: 0 | 0028F685: 0 | 0028F686: 0 | 0028F687: 0 |
0028F688: 0 | 0028F689: 0 | 0028F68A: 0 | 0028F68B: 0 |
0028F68C: 0 | 0028F68D: 0 | 0028F68E: 0 | 0028F68F: 0 |
0028F690: 0 | 0028F691: 0 | 0028F692: 0 | 0028F693: 0 |
==E N D====
= S T A R T ===
DLL Dumping the THIS COPIED OUTSI Array...
0028F680: 69 | 0028F681: 80 | 0028F682: 84 | 0028F683: 65 |
0028F684: 84 | 0028F685: 82 | 0028F686: 79 | 0028F687: 80 |
0028F688: 73 | 0028F689: 67 | 0028F68A: 65 | 0028F68B: 76 |
0028F68C: 0 | 0028F68D: 0 | 0028F68E: 0 | 0028F68F: 0 |
0028F690: 0 | 0028F691: 0 | 0028F692: 0 | 0028F693: 0 |
==E N D====
ON DLL END, INSI ADDR (0028F8B0)
ON DLL END, OUTSI ADDR (0028F680)
ON EXE, OUTSI address : 0028F680佩 return length: 12
DLL side ended for first call
Here is the dumping in EXE part for the same address.
----
Now Dumping the Char Array...
0028F680: 69 | 0028F681: 80 | 0028F682: 84 | 0028F683: 65 |
0028F684: 84 | 0028F685: 82 | 0028F686: 79 | 0028F687: 80 |
0028F688: 73 | 0028F689: 67 | 0028F68A: 65 | 0028F68B: 76 |
0028F68C: 0 | 0028F68D: 0 | 0028F68E: 0 | 0028F68F: 0 |
0028F690: 0 | 0028F691: 0 | 0028F692: 0 | 0028F693: 0 |
Dumping finish in EXE part, you can see it is the same as DLL part dump, so I assume it is working normally.
Then here it comes the next call, which use Result3 as buffer. I see it has address from 0028F450
ON DLL START, INSI ADDR (0028F8B0)
ON DLL START, OUTSI ADDR (0028F450)
= S T A R T ===
DLL Dumping the PREV OUTSI Array...
0028F450: 0 | 0028F451: 0 | 0028F452: 0 | 0028F453: 0 |
0028F454: 0 | 0028F455: 0 | 0028F456: 0 | 0028F457: 0 |
0028F458: 0 | 0028F459: 0 | 0028F45A: 0 | 0028F45B: 0 |
0028F45C: 0 | 0028F45D: 0 | 0028F45E: 0 | 0028F45F: 0 |
0028F460: 0 | 0028F461: 0 | 0028F462: 0 | 0028F463: 0 |
==E N D====
= S T A R T ===
DLL Dumping the THIS COPIED OUTSI Array...
0028F450: 69 | 0028F451: 80 | 0028F452: 84 | 0028F453: 65 |
0028F454: 84 | 0028F455: 82 | 0028F456: 79 | 0028F457: 80 |
0028F458: 73 | 0028F459: 67 | 0028F45A: 65 | 0028F45B: 76 |
0028F45C: 0 | 0028F45D: 0 | 0028F45E: 0 | 0028F45F: 0 |
0028F460: 0 | 0028F461: 0 | 0028F462: 0 | 0028F463: 0 |
==E N D====
ON DLL END, INSI ADDR (0028F8B0)
ON DLL END, OUTSI ADDR (0028F450)
ON EXE, OUTSI address : 0028F450MERCHANTID return length: 12
Strange started from here, I see in 0028F450 in DLL, it should be
69,80,84,65,84,82,79,80,73,67,65,76
However... The dump in exe become like this,
But it becomes
0,0,68,0,83,-12,40,0,0,0,0,0
The dump is here still at address 0028F450
----
Now Dumping the Char Array...
0028F450: 0 | 0028F451: 0 | 0028F452: 68 | 0028F453: 0 |
0028F454: 83 | 0028F455:-12 | 0028F456: 40 | 0028F457: 0 |
0028F458: 0 | 0028F459: 0 | 0028F45A: 0 | 0028F45B: 0 |
0028F45C: 0 | 0028F45D: 0 | 0028F45E: 0 | 0028F45F: 0 |
0028F460: 0 | 0028F461: 0 | 0028F462: 0 | 0028F463: 0 |
Just for your reference, here is the third call, with Result2 again as the buffer:
However, this hasnt been changed to gibberish mysteriously.
ON DLL START, INSI ADDR (0028F8B0)
ON DLL START, OUTSI ADDR (0028F680)
= S T A R T ===
DLL Dumping the PREV OUTSI Array...
0028F680: 69 | 0028F681: 80 | 0028F682: 84 | 0028F683: 65 |
0028F684: 84 | 0028F685: 82 | 0028F686: 79 | 0028F687: 80 |
0028F688: 73 | 0028F689: 67 | 0028F68A: 65 | 0028F68B: 76 |
0028F68C: 0 | 0028F68D: 0 | 0028F68E: 0 | 0028F68F: 0 |
0028F690: 0 | 0028F691: 0 | 0028F692: 0 | 0028F693: 0 |
==E N D====
= S T A R T ===
DLL Dumping the THIS COPIED OUTSI Array...
0028F680: 69 | 0028F681: 80 | 0028F682: 84 | 0028F683: 65 |
0028F684: 84 | 0028F685: 82 | 0028F686: 79 | 0028F687: 80 |
0028F688: 73 | 0028F689: 67 | 0028F68A: 65 | 0028F68B: 76 |
0028F68C: 0 | 0028F68D: 0 | 0028F68E: 0 | 0028F68F: 0 |
0028F690: 0 | 0028F691: 0 | 0028F692: 0 | 0028F693: 0 |
==E N D====
ON DLL END, INSI ADDR (0028F8B0)
ON DLL END, OUTSI ADDR (0028F680)
TERMINALID return length: 12
----
Now Dumping the Char Array...
0028F680: 69 | 0028F681: 80 | 0028F682: 84 | 0028F683: 65 |
0028F684: 84 | 0028F685: 82 | 0028F686: 79 | 0028F687: 80 |
0028F688: 73 | 0028F689: 67 | 0028F68A: 65 | 0028F68B: 76 |
0028F68C: 0 | 0028F68D: 0 | 0028F68E: 0 | 0028F68F: 0 |
0028F690: 0 | 0028F691: 0 | 0028F692: 0 | 0028F693: 0 |
I am confusing now and wasted a whole day on it, any further guidance is much appreciated :)
If i needed to supply more information, i would be glad to do so, thanks for your help.