Crumbling bricks behind fireplace
To answer the comment about brick wall thickness in a Victorian era house, it will at least be an 8 or 9" wall thickness. The style of brickwork shows that, with the ends of brick showing (header courses). 4" thick walls with header courses did not occur that I am aware of until much later.

It also appears that the firebrick that lined the firebox, may have been a later addition, since firebrick may have been a later development. This I am not sure of, but a search online could confirm the timeline of masonry material and availability. The main reason I bring this up it the charring of the brick, even low on the brick where the heat and smoke is normally not so heavy since it all rises with the air currents going up the chimney. I know it still gets really hot there but not like the rest of the back wall where the brick degradation is really bad. And since the box was getting in bad shape, and firebrick became available, it was rebuilt, covering the old brick. Otherwise the old brick would have survived much better.

So with all that said the remedy for me would be to make sure all the remaining brick are intact, no more crumbling, can be loose, but not crumbling for those can be reset with the proper mortar either type "N"  or type "K" which will be compatible with the softer old brick. After everything is made "tight", parge (skim with a heavy layer of mortar) over everything to finish off the surface to seal it all up. This may take more that one application to do, filling the deepest spots first, then coming back the next day and leveling everything up. Does not have to be pretty, although it can be made to look pretty good with lots of practice, but it really just needs to be there to do its job.