Is it possible to perform neuroevolution without a fitness function?
You do not always need an explictly coded fitness function to perform genetic algorithm searches. The more general need is for a selection process that favours individuals that perform better at the core tasks in an environment (i.e. that are &quot;more fit&quot;). One way of assessing performance is to award a numerical score, but other approaches are possible, including:

Tournament selection where two or more individuals compete in a game, and the winner is selected.

Opportunity-based selection, where agents in a shared environment - typically with limited resources and chances to compete - may reproduce as one of the available actions, provided they meet some criteria such as having collected enough of some resource. I was not able to find a canonical name for this form of selection, but it is commonly implemented in artificial life projects.


A key distinction between A-life projects and GA optimisation projects is that in A-life projects there is no goal behaviour or target performance. Typically A-life projects are simulations with an open ended result and the developer runs a genetic algorithm to &quot;see what happens&quot; as opposed to &quot;make the best game-player&quot;. If your project is like this then you are most likely looking for the second option here.
To discover more details about this kind of approach, you could try searching &quot;artifical life genetic algorithms&quot; as there are quite a few projects of this type published online, some of which use NEAT.
Technically, you could view either of the methods listed above as ways of sampling comparisons between individuals against an unknown fitness function. Whether or not a true fitness function could apply is then partly a matter of philosophy. More importantly for you as the developer, is that you do not have to write one. Instead you can approximately measure fitness using various methods of individual selection.

So far I've implemented most of the basics in amethyst (a parallel game engine written in rust), but I'm worried that the learning will happen very slowly. Should I approach this problem differently?

It is difficult to say whether you should approach the problem differently. However, the biggest bottlenecks against successful GA approaches are:

Time/CPU resources needed to assess agents.

Size of search space for genomes.


Both of these can become real blockers for ambitious a-life projects. It is common to heavily simplify agents and environments in attempts address these issues.