How can I create a custom form settings to my custom field formatter? I need to add form settings to my custom display format, can you provide me with an example?

To be clear I just want to add some fields to the format settings form.
Whenever you need to figure out how something works, look at examples that already do what you want. In your case, that's the ImageFormatter plugin.

Specifically, you are looking for the methods defaultSettings(), settingsForm() and settingsSummary(). Implement them, define the form for you settings, specificy default settings and for improved user experience, implement the summary method to give users a short summary of the current settings.

That's all, in viewElements(); you can then access your settings with getSetting().