Contextual filter in taxonomy- Drupal 7
This is how I have gotten this working:


First thing you need to do is install the References module
Update your Place content type:


Rather than add a Destination term reference field, you should create a Destination node reference field.

Configure your view.


Under Format > Settings, select Grouping Field Nr. 1 as Content: Places (or whatever you called your places term reference field).
Assuming you have no relationships added, create a Relationship on Content: destination (field_destination) (or whatever you called your node reference field).
Save your relationship.
Assuming you have no contextual filters added, create a Contextual Filter on Content: Nid.
Select the relationship on field_destination (or whatever you called your node reference field).
Select Provide default value with a Type of Content ID from URL.
Save your filter.
Save your view.

Configure View Block


Go to Structure > Blocks
Configure your View Block
Add to region and set Content-Type to Destination.
Save the block.



Now if you go to one of your destination pages, you will see a list of places grouped by place type.