How can we determine which students are a good fit for an educational program?
tl;dr Approximate the "real world" practice/environment as the assessment.

Pre-setup



To begin with, I'm going to presume that there is some form or pre-screening, and/or qualification process. I.e.: having taken such &amp; such course(s), being actively enrolled in some, possibly specific, institution, passed through an interview, etc. The process below most likely would be quite unwieldy with a couple hundred applicants, but should be reasonable with a few dozen. Especially if administered in batches.

In addition, as the program involves working remotely, I would expect that there is some form of real-time communication channel that will be utilized in the day-to-day working setup. I do not mean the telephone either. I could be as simple as using some form of I/M (AOL, Yahoo!, Messenger, etc.) or it could be an in-house developed and employed enterprise solution. Issues and comments on GitHub are good, but not sufficient when dealing with a mentor-ship program. Ideally, the solution (using different tools if necessary) needs to include a "private" channel between the mentor and mentee, and a "team" channel. A third channel for all the mentors and mentees to network, outside of the project development would also be beneficial.

Old tools, which I would recommend migrating from if in use, are Skype and TeamSpeak, though either would be better than nothing (provided you want to pay for TeamSpeak anyway). Two current tools I would suggest investigating are Keybase and Discord. Both are available for install on Windows, macOS, Linux, iPhone, and Android, as well as being usable form within a web browser. Both are also free, though Discord does have enhanced content available for a membership fee. Discord has voice channel capabilities, while Keybase does not. Keybase offers end-to-end encryption and has the ability to share, under encryption, files as well as having the ability to integrate with git and create an encrypted repository outside of GitHub. As a user of both, Keybase would be my choice for a professional development environment.

Assessment setup



I would suggest having the assessment as a collection of small projects. Adapted to the project and environment they will be working on, of course. Each assessment can target one or two skills rather than having a full-scale project to assess everything at once.

Projects could include:


Create a routine to perform this task given parameters of X, Y, and Z and returning t and v.
Given this pre-existing routine, extend it to handle this.
This method, given (one, two, 15, false) should return expected_result, instead it returns bad_answer. Why? Fix-it.


In some of the assessments you can intentionally include ambiguous information, incomplete data, or other conditions which can only be handled by asking you (hence the need for real-time communication). Provided you've made it clear that questions are not only allowed, but encouraged and expected, they should be able to contact you with the questions. This also allows for checking how they deal with time management, if at some point you are "unavailable" and they don't get an answer. Do they plow ahead, developing based on a guess? Do they stop, waiting for an answer? Or, do they "table" that problem and work on another one?

If the work environment will include TDD or Agile, then the assessment should have the elements of that in some of the problems. As the work involves working as part of a team of 3, it would also be ideal if you could have the first few assessments deal with their own work, then have an assessment, or more, where three of the candidates, who may not even know each other, work on one project together. You could also have two candidates, and the third is a "ringer" who already works for the company.

As remote workers you will have no control over what resources they employ, including looking up how to write Java on Google. My opinion is that knowing how to find the answer is more important than knowing the answer. With those two points in mind, you could explain that "Google is your friend" or include links to the resources, including style guides, which most closely match your current standards, and encourage them to use those links in developing the "answers" to their assignments. This allows you to include ever harder issues, or more advanced concepts, and not only assess their knowledge, but their ability to expand that knowledge.

GitHub



You can also incorporate git and GitHub into the assessments, as early as the first one if you wish, by having them create their own GitHub account, provide an archive of the starting code if needed, and have each assessment be its own repository in their account. You can then use the issues and comments there to give feedback, even forking their repository, creating a new branch and making a pull request in their repository. One thing I would suggest not doing on GitHub is having a repository, or collection of repositories, from which the assessments, and code, are distributed. You could create a collection of separate accounts for yourself which are used with exactly one candidate at a time, and "reset" between as needed. With one account and all the repositories used in it, all the candidates would be able to see all the assessments, including ones they have not been assigned yet, and all the work of other, or prior, candidates.