Suppose we have a function f that gets a complex (non-flat) object as an input, serializes it, calculates hash of serialization results (e.g. md5) and returns calculated hash.It may use some exiting serialization method, e.g. msgpack.But are there any known techniques that may be more efficient than full (reversible) serialization and preserve f's properties as a hashing function (e.g., collision probability)?
Efficient irreversible serialization?
hashing;serialization
Have a look at the notion of composable hash functions.  The idea is that you have some snippets and you want to do some of the hashing for each snippet and the combine the hash results.  So, some hashing API's are designed to allow you to start the hash, then add more into the hash, then finish.  This is useful in a number of situations, for one, when you have several snippets instead of a (serialized) whole to hash.  For another, when the snippets each have their own different preferred mechanism for hashing.