How can I create a custom form settings to my custom field formatter?
Whenever you need to figure out how something works, look at examples that already do what you want. In your case, that's the ImageFormatter plugin.

Specifically, you are looking for the methods defaultSettings(), settingsForm() and settingsSummary(). Implement them, define the form for you settings, specificy default settings and for improved user experience, implement the summary method to give users a short summary of the current settings.

That's all, in viewElements(); you can then access your settings with getSetting().