Is it weird to give your new PhD supervisor a "thank you" gift for supporting your application?
I guess this matter is related to the nationality of your supervisor and to the country you are going to work in.

In Europe, especially France where I'm doing my PhD it's somehow weird to give gifts to your supervisor, since they consider the help they are offering is a part of their job. Yet they understand that the people coming to work in Europe are of different cultural backgrounds. 

An important note to add, the people like the cultural exchange, so if you are giving him a part of your culture it will be appreciated, I got used to getting  them sweets from my home country and everyone like it. 

My advice if you want to do that, try to give it to him when you are alone, doing that in front of others is weird. 

And as we Lebanese say the act of kindness need no questioning.