So when should I use 旅舍，旅社，宾馆，酒店, or 旅馆?
酒店 usually means restaurant only, but it refers to the hotel with restaurant advertised or the larger hotel too.

宾馆/旅馆/旅店 is hotel, same exactly meaning, including the hotel with restaurant. But you never call a restaurant only as 宾馆. And you don't call a very good hotel as 旅馆/旅店.

旅社/招待所 is often used by mid aged people in China, when they say 旅社 they meant those hotels that are nation sponsored (国营的) in the planned economics time. Mostly it is equivalent to 宾馆/旅馆, but 旅社 refers to the smaller cheap hotel on the street nowadays, 招待所 is still for the nation or organisation sponsored hotels.

旅舍 means the accommodation for travellers, mostly in country side, specifically  means the small and cheaper hotel. It refers the building rather than the function of the hotel.

YHA can be translated as 青年旅馆 or 青年旅舍, rarely with other names. Bagpacker's can be translated as 背包旅馆 or 背包旅舍, nobody calls it as 背包宾馆. Hilton hotel can be translated as 希尔顿酒店 or 希尔顿宾馆, nobody calls it 希尔顿旅馆. Novotel like business hotel can be translated as 商务旅馆, 商务酒店, 商务宾馆.