Can we use the Tierra approach to optimize machine code? Thomas Ray's Tierra is a computer program which simulates life.

In the linked paper, he argues how this simulation may have real-world applications, showing how his digital organisms (computer programs) evolve in an interesting way: they develop novel ways of replicating themselves and become faster at it (he argues that the evolved organisms employ an algorithm which is 5 times faster than the original one he wrote).

Tierra's approach is different from standard GAs:


While in GAs usually there is a set of genomes manipulated, copied and mutated by the program, in Tierra everything is done by the programs themselves: they self-replicate.
There is no explicit fitness function: instead, digital organisms compete for energy resources (CPU time) and space resources (memory). 
Organisms which take a long time to replicate reproduce less frequently, and organisms who create many errors are penalized (they die out faster).
Tierran machine language is extremely small: operands included, it only has 32 instructions. Oftentimes, so called RISC instruction sets have a limited set of opcodes, but if you consider the operands, you get billions of possible instructions.
Consequentially, Tierran code is less brittle, and you can mutate it without breaking the code. In contrast, usually, if you mutate randomly some machine code, you get a broken program.


I was wondering if we could use this approach to optimize machine code. For instance, let's assume we have some assembly-like program which computes a certain function $f$. We could link reproduction time with efficiently computing $f$, and life-span with correctly computing it. This could motivate programs to find novel and faster ways to compute $f$.

Has anything similar ever been tried? Could it work? Where should I look into?
Yes it has been tried. In fact there is a whole field, dubbed Genetic Programming.

There is an annual competition to obtain "Human-Competitive" algorithms, and many instances of those have been found over the years.