What was the motivation for developing the plugin API? I am trying to wrap around my head as to what was the need for the Plugin API when the module system extends the Drupal core.
It is not modules vs. plugins. Plugins are competing against hooks, which they will eventually replace entirely, together with event subscribers and services. By the way, all are implemented by modules because you are not allowed to insert them in core directly.