Single view to display related nodes I am trying to display related nodes block using a single view for all the taxonomy terms. So that if user click on any node title he/she will see the related nodes based on the node's taxonomy term.

I tried adding contextual filter with taxonomy term by selecting all the vocabularies, but doing this will display nodes from other terms as well.

The reason I am trying to use single view is, I have tabs (Quick tabs) structure on a node page, with related nodes as a Tab. So I cannot create different views for nodes based on terms, and cannot specify different views in quick tabs settings.
You need to create a block view with all content. Then filter content for your specific term by adding a filter-> content type. 

Then add a Content: Has taxonomy term ID. On the config of the later. When the form value is not provided in the URL choose provide default value -> Taxonomy term ID from URL. 

Enable load default filter from page and load default filter from node. On the latter enable your vocabulary. 

Afterward, config the case that the ID is present on the URL. Specify validation criteria by choosing taxonomy term and again chose your vocabulary. 

It should work!
Check the whole tutorial on http://www.drupalgardens.com/videos/410746