How to create a node group with a custom node tree?
There's a reference implementation for a custom Node Group for CustomNodeTrees, used in Lukas Toenne's object_nodes branch. While part of it is compiled, the UI is implemented using bpy's PyNodes ( A CustomNodeTree, same as Sverchok / AnimationNodes ).

If you are comfortable with Python then having a reference implementation may be enough, the whole concept is a little complex to try to convey as a single answer because it involves describing many sub problems. In short, it is possible, but it's not trivial. you'll have to deal implement a few things: 


How does a node Group behave when copied/duplicated
Implement the code to generate a node Group from the selected nodes
Creating a NodeSocketInterface (What's that?!)


Most of these can be copied almost verbatim from the reference implementation but others will likely need some modifications to fit with your own CustomNodetree type.



Alternate implementation

In Sverchok we recently pushed a new group node implementation (codename Monads): see anything imported by /nodes/basic_data/monad.py. Warning, expect some complex dynamic programming.

It takes a slightly different approach, it is not final but includes a lot of the same functional Operators / UI behaviour: 


socket recast
delete socket
rename socket
make nodegroup from selected. 


I don't consider this a reference implementation, but it may be of some interest.