What do you do when you cannot make progress on the problem you have been working on?
Is this a common situation that a researchers notices that her idea is
  not going to work after considerable amount of work?


Yes. But as you get more experienced, you're able to "fail fast" - learn how to test the idea quickly to see if it passes a 'smell test'. 


  What do you do when you realized that an approach you had in mind is
  not going to work and the problem seems too difficult to solve?


It depends. Sometimes the best thing to do is put the problem away for a while and work on something else. Sometimes the failure suggests a different question. 


  What advice would you give to a student in my situation?