NBitcoin: Get address out of raw block stream
my test transaction has a Base160 encoding, so according to my understanding it was sent to an address


I guess you mean ripemd160 or base 16 (hexadecimal) rather than Base160. Ripemd160 is just a hashing algorithm so the output is just bytes, probably encoded in hex (base 16). You can guess it is a ripemd hash based on its length, but not based on its encoding, because the entire transaction is usually displayed in hex.

But to convert from a pubkey hash to an address, you also have to add the checksum onto the end, rather than just the version byte (the 0x00). That is most likely why you are getting a different result, because you have no created a checksum before encoding it with base58.

See the steps outlined here for more info: https://en.bitcoin.it/wiki/Technical_background_of_version_1_Bitcoin_addresses#How_to_create_Bitcoin_Address