Is it possible to create a custom object type with Python API?
No, object types are hard coded in Blender's C code.

Typically developers of add-ons use Empty types and attach custom properties to them.