How to open a node form in popup after click on edit from front end?
If you want to go through custom module you need to follow instructions given here. If you are looking for contrib module colorbox or modal forms to edit your content on modal popup(using Ctool).