What is the smallest size of the hash output? Suppose we want to make the hash pre-image resistant to 40 bit security (success chance is $2^{−40}$). What would be the smallest size of the hash output?
To get a preimage resistance of 40 bits, you need a hash with at least a 40-bit digest. Finding a preimage for an $n$-bit hash typically requires performing $2^n$ attempts. A collision for an $n$-bit hash requires $2^{n/2}$. For a 40-bit hash, a preimage requires $2^{40}$ operations and a collision requires $2^{20}$.

Note that 40 bits of preimage resistance is typically not considered sufficient for most applications!