How to pass formatter settings to field theme function?
Your formatter_view implementation looks ok, very similar to image_field_formatter_view().

However, you're info hook seems to be missing default values for your settings, I'm not sure if they are required or not, but try to add them, again see image_field_formatter_info()

Does the field_formatter_settings form work correctly? Does it show correctly in the summary and keeps the values when you edit it again?

Without more information, the only advice I can give you is install devel.module if you haven't already and make extensive use of dpm() to see where you get which information. For example, contains $display the necessary information in your formatter_view() function?

If yes, it could simply be a outdated theme registry that does not yet know about your additional theme argument (if you added later on for example).