What is the field that deals with observing an individual's fitness within a system called?
The study of physiological fitness is simply evolutionary biology. The study of animal behavior and how it evolved is ethology. John Maynard Smith was one of the first to apply mathematical modeling, especially game theory, to the evolution of species. It is now a well-established field (see for example the wikipedia page for evolutionary game theory). But other mathematical frameworks, such as dynamical systems, have been applied to the question.
https://en.wikipedia.org/wiki/Evolutionary_game_theory

The precise algorithm you describe is called an evolutionary algorithm and is a well-known, well-studied subclass of optimization algorithms that are loosely inspired by biological systems (e.g. genetic algorithm, particle swarm algorithm...). These algorithms are useful when trying to optimize a high-dimensional and/or highly non-linear function.
https://en.wikipedia.org/wiki/Evolutionary_algorithm

As for your side note I think you didn't think this through very carefully. Firstly you are now talking about behavior, not physiology. It is far less clear how behaviors are shaped by evolution. Then you describe how an observer would attribute value to an environment, while fitness is precisely the opposite. Finally there is no notion of "value" in evolution. You might be fit in one environment, but not in another. It doesn't make you any better or worse.