Which settings to use in hook_field_formatter_view?
This wasn't really a problem. This was because the page was a Panels page, and the field was added into the panel page with its own formatter settings.

Solved by setting the correct field formatter settings on the panel page.

It's one of d'ohs days.