firmware testing methods
it is so messy that no one dares to touch it because testing it would take months

Same boat. Don't touch it unless you have to for a hotfix.

I want to change this sticky situation and organise the code by introducing hardware abstraction layers

You will run into the problem that 8051 is not designed to run C, it has very little of everything and the cost of abstraction is too high, especially in a mature product. You probably can't get a chip with more memory since you're at architecture limit.
You're probably also running into a lot of optimization hacks that destroy any last readability the C had.
What you can do however, is extract to inline functions and macros which you can then run in a mockup scenario. You emulate all the requisites for it to compile and then you can test your functions. This is a lot of work to do retroactively.
Some functions you probably can't test due to the side effects. This is a design flaw. You can't &quot;fix&quot; that cheaply.
Tech debt is real, and it's expensive. Don't make a mess again.

Note that depending on the compiler 8051 is often used in big-ending mode, this could make testing difficult depending on the micro-optimizations used in the code.

On your edit:
Sound like you're still looking to test only the software, but you don't want to take the hardware out of the loop. So you're going to mock the chip environment instead of the chip itself. Result can be equal, just a little bit more work.
Even the act of modifying the code to compile in vanilla gnu C can uncover bugs.