Is probabilistic machine learning just the mathematical background of machine learning?
Knowledge in Probability and Statistics is absolutely important. It might feel you can do quite a bit without that and can get things work using the high level ML libraries that we have today. However if you are a serious practitioner you will figure out that beyond a point your knowledge on how these techniques work and are tuned is very hollow and that will automatically lead you into the more mathematically inclined books that give you the foundation knowledge.