How to remove default div wrappers from fields output via custom field formatter?
The div wrapper you are looking for is probably in field.html.twig. Drupal provides a theme name suggestion for a custom field type automatically, so you can override this template by placing field--myfieldtype.html.twig in the theme templates folder.

If you want you can deliver a default template with the custom field module, like the comment module does for the comment field:

core/modules/comment/templates/field--comment.html.twig


Then you need to define a theme hook in mymodule.module. 

Example from the comment module:

/**
 * Implements hook_theme().
 */
function comment_theme() {
  return [
    // ...
    'field__comment' =&gt; [
      'base hook' =&gt; 'field',
    ],
  ];
}