Can I withdraw from the poster session?
This is very, very common. At most conferences, there are simply not enough time slots for everyone to give an oral presentation. Conferences still tend to be way too long most of the time with way too many "less important" presentations between "interesting" ones anyway. 
Posters can be a very efficient way of communicating your results and connecting with other researchers important to you. You are not "wasting" your paper by presenting a poster. And you will attend other's presentations and can engage in the discussions. So I don't really agree with your conclusion.

It's understandable that you are disappointed. However, while it is totally up to you, your decision on whether to participate should not depend on a potential resentment of being "rejected" from an oral presentation. Base your decision on whether to go on who you think you might get to know, and who might be there who could benefit from your results. Prepare a great poster. Have handouts ready. Approach people directly, start to network actively. 

Whether or not you might receive any money back highly depends on the specific event and the local organizing committee. You can ask. But at this stage I would say it's unlikely.