Custom field set for custom entity
to CRM_Core_BAO_CustomGroup::mapTableName
I had to add this:

case 'Lead':
    return 'civicrm_lead';


after that I'm able to add custom fields for my custom entity. 

so steps to add custom data for custom entity you need these things:
Add entity to this group cg_extend_objects
and then map type to table here: CRM_Core_BAO_CustomGroup::mapTableName