Using autograding (Github Classroom) - pitfalls and best practices? I'm currently as a TA developing a series of programming assignments for a bachelor course on machine learning. Since we expect the number of students taking this course to grow in the future, we're looking at autograding to see if it's useful for us. In particular, we're looking at using Github Classroom. I have a number of related questions about this. Feel free to answer all or only some of them.

Autograding leans heavily on unit tests to grade, and these unit tests are in the students' repositories. This means students can work on their code until it satisfies all the tests, which would result in a perfect grade. What kind of assignment setup/grading setups are possible that aren't so all or nothing?

Can you stop students from altering files in the repository that they're not supposed to alter? For example, if their starter code contains a file assignment.py and tests.py, can you enforce that they only change the assignment file?

Is it possible to apply hidden unit tests, that aren't visible in the students' starter code repository? For example, we might want them to implement a ML algorithm, then test it on a dataset they haven't seen yet, to see if it generalizes well. Also, it would enable us to have a tamper-proof unit test (see point 2).

What are general issues to watch out for when using autograding for (some of) the assignments in a course? Or Github Classroom-specific problems?

What are in general best practices for using autograding in a course?
In the auto grader I use, the following process is followed:

The students files are copied into a pristine work area
The &quot;provided&quot; file(s) are removed from the work area
If I intend to use MOSS, the remaining files are copied for later use
The &quot;provided&quot; file(s) are copied into the work directory
The result is built (if necessary)
Tests are executed and graded

Note that the &quot;provided&quot; file(s) copied in at step 4 may be a superset of the file(s) provided to the student in the assignment and typically contain other/additional test cases.
If the build fails because a student has modified a &quot;provided&quot; file, the students gets no points. If Java assignments, the provided file is often a interface, sometimes an abstract class. In C/C++ assignments, the provided file is often a header file.
If some test cases are provided and reused verbatim in the final run, those test cases only account for 60% of the final grade. Thus, the student can not receive a passing grade by just making those test cases pass.