8.4 - Use entity browser in a module block from in same way as I would in a custom block
No, this is not possible. Block forms in a block plugin are configuration forms and here you can only use Form API fields.


  I (think I) understand that custom block types are actually nodes
  internally and that they are subject to different rules to blocks
  which I define in my custom modules, but I don't fully grasp what the
  differences here are.


Custom blocks are not exactly nodes internally, it's very similar, custom blocks and nodes are both content entities. This doesn't change the rest of what you found out correctly. You can use entity fields and their form elements, widget and formatters only for content entities.