Is it possible for people to go into math or physics fields from a degree in biology?
"Math" means many things - just at the start, that can mean something like applied math vs. pure math, for which this answer has very different tracks to it.

I'd say it breaks into two potential tracks:

Track One: "I want to be a mathematician" - you want to move in this direction, and either do math for its own sake, or math as it's applied to a wide number of different things. Here, in my experience, the answer is "it's possible, but hard". There will be expectations about your coursework background that you may not have been able to do without effectively double-majoring, but it's not impossible.

Track Two: "I want to be a very mathy biologist". As you've noted, the "in vogue" way to do this is to move from math to biology, but the other way is certainly possible. There are a number of labs that would fall under the heading of "Mathematical Biology" that accept biology majors into their ranks, and if you're willing to put in the work, there's no reason you couldn't build a very strong background in math in the process, while keeping the value of having subject-matter expertise (something that is undervalued, IMO).

I took the latter track, and have ended up moving from an undergraduate background in laboratory microbiology to something that very much fits the description of "mathematical biology".