Automatic grading of test-cases Last year I taught a programming course, and used an automatic grading system for grading the students' homework (in addition to other evaluation measuares). It worked quite well; the bottleneck was the need to write test-cases for each assignment.

This year, I would like to have the students themselves write the test-cases - both to save time, and to have them practice this important skill.

I thought of the following scheme:


In each of the 10 assignments, 10% of the students will be the "test set" and the other 90% will be the "train set".
The "test set" students will write test-cases - each student will write e.g. 10 test-cases.
The "train set" students will solve the assignment; their solutions will be tested using the test-cases of the test-set.


The problem with this scheme is that, if the solution of student A from the train-set fails on the test-case of student B from the test-set, it is not clear whether it's because an error in student A's solution or an error in student B's test-case. Of course I can look and see who is correct, but this requires a lot of work and undermines the idea of automatic grading.

How can I improve this scheme to allow automatic or almost-automatic grading of both homework assignments and test-cases?
Have you thought about having a solution that you create and test the students test cases against your solution? Then you have a stable platform for running the student defined tests.

Also run test cases which you prepare against the student's code. Now you have graded the two parts independently of one another.

Just out of curiosity, what did you use for auto-grading?