Grout clogging up drain in bathtub
I have used a Powerful Wet/Dry Vac for loose pieces before, if that is not the case, and the Grout is a solid piece, Go to the hardware shop and buy a small bottle of Hydrochloric acid, it’s also called Muriatic acid in some places. 

Dilute the acid with water (2 parts water, 1 part muriatic acid) and pour it down your drain. You may need the repeat the process a couple of times still the grout is dissolved and flushed away. 
Make sure you wear eye, skin and breathing protection. Turn the bathroom fan on and open the windows. When you pour it down the drain leave the room straight away and close the door, come back in about 10 minutes and turn the water on the flush the drain. 
If you are not confident doing that get someone else who has had experience with this type of acid. Hydrochloric acid is used a lot on building sites the clean and remove grout. 
What I would try first is get the contractor back to your place and have him do it for you. That’s the least he can do for you. If he won’t come back I’m afraid you will have to do the job.