Applying for a graduate-level Research Assistantship as an undergrad
Why would a professor ask for research assistant applicants for an incredibly complex project worthy of a PhD at an institution that does not offer one?


I think you’re answering your own question. The professor really wants to work on project X; no PhD programs are available, so he’s taking in research assistants. 


  As an undergraduate student, would it be a worthwhile investment of my time to develop the proof-of-concept for essentially an interview boost if the advising professor is certain that the project would be far too complex for an undergraduate student?


I’m fairly certain nothing magical happens to students once they start their graduate studies that makes them suddenly capable of working on complex problems. If you’ve done well in your undergraduate studies and are curious/handle failure well, you can do graduate work. What would raise red flags for me is that other RAs worked on this problem and failed. I would ask the professor for more details and perhaps see if the problem can be broken down a bit so you can successfully manage it.

Showing good results in an RAship is much more than an interview boost. It’s an excellent way of checking whether you like research work, getting a very warm reference letter and maybe even a decent publication or two! Good RAships can get you into top grad school even if your undergraduate degree wasn’t amazing.

Good luck!