Loop through related section's entries
Wasn't sure if I need to also setup a field type entries in GamePublishers to reference to Games section.


Not necessary if you already have a "GamePublishers" Entry field in the "Games" section.

Something like this should work:

{% set games = craft.entries.section('games').relatedTo(publisher) %}


That's assuming you've already got a variable named publisher defined that's referencing the current publisher the user is viewing.

I'd suggest reading through the relations docs as well.  Craft has a pretty powerful relations engine you can do all sorts of crazy stuff with.