If the sender address is known is it possible to find the recipient address? Let's say I give someone an address to send me some bitcoins, when I eventually use these bitcoins to send to someone else can the person that has my address somehow know the address of the person I sent the bitcoins to?

I'm asking this because I'm using a new address whenever I receive bitcoins (as recommended), but I want to collect my bitcoins in one place so I would only have to save one private key, and I wanna know if my privacy will be compromised.
The person you gave your address to wouldn't know if the address you subsequently sent the funds to was actually your cold storage or a merchant, or friend, etc. However if you did business with this person again in the future, and he sees that you again sent it to the same address you sent it to before, then he could assume it might be your address.    

With that said, there are BIP32 wallets that allow you to only store one extended public/private key pair that will allow to restore the millions of private keys and address that you may ever generate. This way you wouldn't have to keep all funds in a single address, instead just a single extended BIP32 key chain.