PhD or physics as a hobby?
One possible way is to do both, in a way.

To help explain, I am drawing on my own current experience -  I earned my PhD in Physics back in 2014, but I still work in a different field. However, I continue research in Physics (and am still getting published) as an Adjunct researcher to the university where I completed the PhD. (Of course, you should check with the university if it is possible to be an Adjunct).

Completing a PhD can provide a foundation for you to continue research in an area of interest as an Adjunct (especially if you published while completing the PhD). Analytical and research skills developed in the PhD could potentially greatly benefit your work in your job (as I have found in my situation).

As in your case, my job allows a considerable amount of spare time, where I have found plenty of time to continue to do research, exercise and engage in my hobbies in my time.

Depending on your time constraints, there is no reason to give up either.