How to check what is wrong with my mobile connection?
Tracing networking problems in mobile networks is pretty complicated. 

Therefore from your perspective there is only one way to find out if it is your phone or not: Use a different phone (different model preferred) using your SIM card at the same places. 

If you encounter the same performance drops you know that it is not your phone. If not it may be your phone but that is not for sure.