What does the typesetting process for a math/science journal using LaTeX look like?
This may differ quite a bit from journal to journal. In the best of worlds (from the author's point of view), the LaTeX source is used in some robust automated process to generate the final product, e.g. using latex or pdflatex, possibly with a slightly different documentclass and different fonts.

However, often the paper is typeset using a completely different "professional" typesetting system, for which there may be an automatic conversion from LaTeX. It may well be the case that many equations have to be corrected or completely typeset by humans. In this case, the person in charge will most likely have no training in mathematics, but they just manipulate symbols in what is to them effectively a foreign language. However, they should be able to understand LaTeX.

If the proofs are hopelessly messed up, then don't even try to correct them, but send them back demanding that the paper be properly typeset according to your LaTeX source. Point out roughly what the errors are and how many. It is the responsibility of the typesetter to set your paper properly and according to the LaTeX source.