An automatic grader in c++ - c++

Question

I am trying to write an automatic grading program for a C++ course I am teaching. Usually, automatic graders use (input,output) pairs: the submitted program reads from standard input and writes to standart output, and the grader compares it with the expected output. But I want my students to practice specific C++ constructs (and not e.g. write a program in C), so the tests are written in C++. In a simple example, I give them a main program such as:
#include "func.hpp"
...
int main() {
test(func(1)==2);
test(func(2)==33);
...
/* some 100 tests, including some randomized tests */
...
cout << "Grade: " << grade << endl;
}
The students have to submit the files func.hpp and func.cpp (for more complex assignments, some more files are required).
There is a bash script that compiles main.cpp with func.cpp, runs the program and reads the grade from the last line (It is run within a docker container, to prevent inadvertent damage to the host computer).
The problem is that a student can, from within func, print "Grade: 100" and exit.
Is there a simple way to make the grader more reliable?

Answer 1

I don't want to go crazy with this, but if you want to make it safe, you should put the student work in a "clean room" so to speak. This is a tedious exercise, but once done, it will be very safe. You said you run in a docker container, so I would run the student's work in a sub-process with IO redirected (to a pipe, file, or to /dev/null, depending on whether some of your assignments involve console output). That way a smart-ass student can print whatever they want, but they will only be piping it to you, not printing to the console. I'm not going to go into the code for this - I saw lots of examples I liked here on Stack Overflow by searching for "fork with redirected stdout" and it's a classic forumula you probably know.
In pseudo code, it would look something like:
main() {
<tedious setup for stdin/stdout/stderr redirection>;
int ch = fork();
if (ch == 0) {
test(func(1)==2);
test(func(2)==33);
.
.
// test clearly generates "grade" so use it as an exit code as an
// easy way to return the information you want
exit(grade);
} else {
for (;;) {
<wait on and read stdout and stderr and do whatever you want to it - it won't go to the console, so no dirty tricks.>;
<you could analyze the student's output as part of grading, or pass it to the console with a highlight, so you know it's not your grade output>
<wait on ch to exit, get the exit code, display it as the grade and break>;
<generate some input if you want to have an assignment where the students read from stdin>;
<add a time-out on waiting for the child to exit. That way evil student can't just hang or busy-loop, whether on purpose by accident.>;
<if it seems to be hanging, you have to process id and can kill it.>;
}
}
}
Obviously the above is sketchy. When you loop on managing a child process, you have to check on everything simultaneously with select or something similar, but you sound like a pretty senior engineer. You get the basic idea. The students can't get away with anything if you keep their run-time environment in a child process that you manage. Plus it gives you you a platform to pull the strings of their code anyway you see fit.

Answer 2

Easy. When you do the analysis, just use a different main than what you gave the students as a reference program. Your version prints your secret key.
int main() {
test(func(1)==2);
test(func(2)==33);
...
/* some 100 tests, including some randomized tests */
...
cout << "The student doesn't know my version prints this next line" << endl;
cout << "Secret validation code: NCC-1701" << endl;
cout << "Grade: " << grade << endl;
}
A clever student who does what you suggests (prints 100 and exits), won't know about the secret message that your version prints and script validates for.
True story. When I was teaching a networking class, there was a homework assignment to implement a "stop and wait" protocol with UDP sockets. My code implemented the "client". The student's version implemented the "server". One student didn't have the networking calls working, but was clever enough to just print the expected output with sleep calls in between print statements. He would have gotten away with it if I hadn't of stopped my client program first and noticed his was still printing our lines of "incoming" data being received.