Is it correct to call oneself "Doctor/Scientist/Mathemathician" simply because one has a PhD in that field?
It's not the individual word that matters. Calling oneself a scientist is not unethical, no more than it is unethical to call oneself a musician without being in the Top100 charts. What is unethical is deliberately representing yourself in a way that causes people to assign you an authority that you do not have.

This is an incredibly gray area because it is entirely dependent on context. As an example, answering "I'm a doctor" to the question "so, what do you do?" would be misrepresenting yourself if you are in fact a mathematics PhD who now works in insurance. However, answering "I'm a doctor" to the question "how well-educated are you?" is perfectly reasonable.

The difference in understanding of a topic between someone who left a field a few years ago and someone who is currently still researching in it is probably not too huge, when it comes to science popularisation efforts. I'd also take issue with your characterization of "just" a PhD - PhDs are the frontline researchers of science, the further along you get in your career the less time you have to actually put your hands on the science. Professor is mostly a management job.

That said, you shouldn't be so impressed with someone that you defer uncritically to their opinion based on just a title. Every field has its marginal cranks, even actively currently practicing ones.