Why would the TRWP Nandroid backups of system and vendor partitions be different between backups?
Answer to your question is dm-verity:


  The dm-verity feature lets you look at a block device, the underlying storage layer of the file system, and determine if it matches its expected configuration. It does this using a cryptographic hash tree. For every block (typically 4k), there is a SHA256 hash.


So /system and /vendor partitions are always mounted read-only because mounting R/W causes changed hash values of the partition/filesystem (at least some inodes belonging to filesystem structure may change even if you don't touch a file), thus breaking the security feature. To make any changes to them, we need to disable dm-verity which must be your case.

TWRP creates backups in two ways. boot, recovery etc. are just dumped (dd) as a whole, so their hashes don't change because those aren't mountable filesystems. /system and /vendor are either dumped as images to retain dm-verity as discussed above. Or otherwise a tar archive is created to keep the backup size minimum. In second case even the filesystem is unchanged, a new archive file is created every time you do the backup. If the files aren't changed and are read/written by the archiving utility in the same order, using same options/parameters every time, then created archives should be identical. But even a minor factor e.g. compression may cause a different hash value every time.

So you must have dm-verity disabled, and therefore you will keep on getting changed hashes. But the contents don't change (if you don't change yourself or Magisk/GApps don't put some addon.d script), so a regular backup of both partitions isn't required. In fact only /data backup is enough, boot, /system, /vendor are already part of flashable zips.