Which serialization format the transactions use?
All of the data structures in Bitcoin use a custom Bitcoin specific serialization format. The standard that is followed is the Bitcoin defined standard, not any other standard. A description of the non-segwit format can be found here. For segwit transactions, the format is described in BIP 144.