How to add tables relations into custom entities content type?
First, forget about tables. Drupal 8 mostly takes care of that for entities.

Think about data structures instead: entities and fields.

What do you want?


Two separate entities, both possibly extensible with additional fields, maybe different bundles (e.g. product and shipping line items). Then make two entity types, with a reference field from line items to the order. 
Simpler, non-extensible line items. Then you can design them as a custom field type and add that as a base field with unlimited cardinality. (In this case, they will be stored in a separate table, but you don't need to know about that for the most part).