Using a custom field as a sub-field in another custom field
Basically, the question is how to use a custom field in another field as a property.
In an entity a custom field is added by using BaseFieldDefinition::create(). However a property of a field type is added by DataDefintion::create() or MapDefinition::create() and only basic types are defined (in DataDefintion class) to be used as DataDefintion. So It doesn't look like it is possible to have a custom field in another custom field.
References:

Fieldable field explains the problem of having a field inside a field
Entity API explains how each data type and field type are defined.
Same issue filed in d.o at #2874458