How to open a node edit form in an overlay (modal window) in Drupal 8
You can do that by constructing special links with the help of Colorbox and Colorbox Inline. You'll probably have to create your own style for the "minimized" overlay. 

There are other libraries that do the same or similar thing as Colorbox, also integrated into Drupal through a module, so you might want to search and investigate others too (Shadowbox, for example). 

These modules are also worth checking out: Modal Configuration, Simple Modal Entity Form, Entity Overlay.