Which serialization format the transactions use? I'm not very familiar with low-level structures, so I would like to know if bitcoin protocol follows a standard on how a bitcoin transaction data is serialized and sent over TCP.

Does it use any of these serialization format?
https://en.wikipedia.org/wiki/Comparison_of_data_serialization_formats
All of the data structures in Bitcoin use a custom Bitcoin specific serialization format. The standard that is followed is the Bitcoin defined standard, not any other standard. A description of the non-segwit format can be found here. For segwit transactions, the format is described in BIP 144.