Is there a transmission protocol library for SPI?
Each spi peripheral has its own protocol in the sense of commands and addresses if any and data.  Spi simply makes a common ground for chip select vs clocks and clocks vs data and separate miso and mosi data lines.

So a general purpose library doesnt make sense.

Also spi certainly is not limited to a byte, many peripherals and many spi controllers allow for long sequences of data, you are also not strictly limited to multiples of 8 bits either although that is the norm.

Your arduino spi master may have its limitations but that has nothing to do with spi in general.

Integrity testing is not part of the spi protocol, you get what you get.  Ideally all of the parts are soldered down on the same board or through known connectors and the margins have been designed in.  You would need to do something outside the spi protocol, for example if it is a spi flash then after you read a chunk of data you can do a checksum and compare it with a checksum you have embedded in the data (or crc).  If it is a spi ethernet part you may choose to leave the frame checksums in the frames and re-validate each frame yourself.  If you create your own peripheral (hardware design or cpld or fpga) then certainly you can add a crc or checksum to every transfer if you wish.