Is it normal for subfloor to extend under walls? If so, how can I replace it?
Utterly standard - the subfloor goes on, then the walls go up. Doing it any other way is rather fraught with difficulties to no benefit.
You've got the general idea, though it is not really critical on the cross-joist direction (as pictured) and often the parallel to the joist direction is solved by putting blocking between the exposed joist and the hidden under the wall joist. The floor can span between joists (which is the argument in favor of not bothering) but can be more prone to move at an unsupported edge (see another recent question, which you asked, Why is plywood with tongue and groove preferred for flooring?) But there is an approach that's valid (if more of a pain to do) where you make continuous support all around the edges.
If the couple of inches of flooring you have not removed at the wall is solid, you can screw through it and into a piece of wood below it that the new floor can also be screwed to in order to make the edge supported, in a somewhat easier fashion. It's crudely equivalent to having a T&amp;G joint in that the floor acts as a single piece.