How does PostGIS make its data types compatable with Postgres?
Is there a way using bytea or any other stuff that I can directly store my instances in Postgres.


Yes, sure you can store anything as bytea. However, don't expect the database to know how to work with it. PostGIS is an extensive system that creates types, operators, access methods, indexes, and a slew of other stuff. You'll have to recreate all of that stuff for your type.

Alternatively, you can create something that reads a PostGIS types and experts one of your own types. This would be a C function. you can then create a cast from PostGIS's type, to your type and call all of your stuff -- your library -- working only on your native implementation.

This allows you to use all that is in PostGIS, and then still access all of the functionality you've got at the cost of a cast.