Why raw Bitcoin transaction are in hex format? Why are Bitcoin raw transactions in hexadecimal format? I am looking for scientific articles / books on this topic and I can't find anything, so my question is: why are they in hexadecimal and can there be any article on this topic?
Because unlike addresses, hex raw transactions aren't meant to be seen or used by end-users. Hex is easy to encode and decode (every two character in hex represents one byte*, whereas in base64 one character is 3/4 byte and Base58 is log(58)/log(256) = 0.7322476244 bytes) so it is convenient if you want to debug your implementation.

*: which means if you can find an erroneous byte in the transaction that you can CTRL+F for that byte in the serialized hex transaction and change it since unlike Base58 or Base64 an encoded character depends on the part of only one byte and never two neighboring bytes. Hex is the highest base with that property. Binary (1 bit divides 8 bits), Quaternary (2 bits divide 8 bits), not octal (3 bits don't divide 8 bits), and hexadecimal (4 bits is the largest divisor of 8 that isn't 8) are other bases with that property, so hex is the most efficient one among them.

Or simply, it was created as an internal tool which didn't need the complexity of a new format that provides space efficiency or avoiding similar characters (or checksum, even though hex doesn't depend on checksum) that is user-friendly.

From Bitcoin QT 0.7.0 release notes:


  Added raw transaction API. See https://gist.github.com/2839617


Gavin Andresen, who created and submitted the hex transaction encoding code to Bitcoin Core/QT wrote:


  I considered parsing/writing BIP 10 format for raw transactions, but
  decided that reading/writing BIP 10 format should happen at a higher
  level and not in the low-level RPC calls. So 'raw transactions' are
  simply hex-encoded into JSON strings, and encoding/decoding them is
  just a couple of lines of already-written-and-debugged code.