Display only the terms in the filter set using Views
I would suggest doing the following. 

Instead of displaying all the tags in one line display it in different lines. That is in the multiple filed settings you would choose it to display it on different rows. The result would be something like this

node 3 | red
node 5 | white
node 5 | green 

Without the filtered nodes. 

Now you would group it with the title field so that it would be displayed together. 

To do the grouping you would do the following.

Click on the settings link towards the right of Format :



This would bring up a screen as shown below. you would select the field you want to group from the dropdown Grouping field Nr.1