Changing major after BS and MS in CS and double major in Physics for PhD in Mathematics
What are my chances if I apply for PhD in Number Theory?

Every PhD program has different requirements and the admission's committee evaluates different aspects of a candidates application. So there is no real answer to your "chances" of be admitted if you apply for a PhD program in Number Theory. However, with that said, if you can show rudimentary course work and experience in Number Theory, that can work in your favor. I do have close friends who majored in Business, Political Science, etc. who took the minimum requirements for medical school admissions (basic bio, chem, physics) and were admitted because they were able to distinguish themselves from the rest of the applicants. The same situation applies here. You may not have an extensive background in Number Theory, but that's not to say you won't be admitted to a PhD program and be great at it! 

Do you recommend changing the field? Can you share other people's experiences in similar cases?

You need to follow your passions for where you want to take your career. I have had friends who pursued what they thought was their "dream" career, only to recognize that it was not at all what they had imagined - and wasted all that time thinking that it would change, that if they put in more effort than it would be exactly what they imagined it. On the other hand, I've also had friends who took a job that wasn't what they had wanted/dreamed and it turned into a great career and they absolutely love it! It all depends on what you want your future to be.