Is there a possibility to run a custom Notes server for iOS?
So it looks like this is impossible. 

Yes, Notes does support IMAP and this can be set up by adding an Email account to iOS (or macOS) and checking "Notes" there. 

However those kind of Notes are handicapped: you can't draw there with Apple Pencil, can't use any markdown like tables/bullets. Basically it is just Text and (image) attachments, making this app nearly useless (most of the notes I am making are made with Apple Pencil). 

What I like about internal Notes.app is that it is automatically launched upon an apple pencil tap on the locked screen - this is why I prefer it over 3rd party apps. But, when I tap on an IMAP-connected note, I am getting an Alert "Apple Pencil is supported only on iCloud notes". Damn.