Why is a privacy problem to pay always to the same address? I know as a merchant it is not a good idea to receive all payments to the same address because then people could track our balance.    

But I don't understand why it could be a privacy problem if we receive payments from one customer to the same address (but every customer has a different address assigned).
Why will there be a privacy hole for the customer if we always receive their payments to the same address?
Assume you are using a unique common address for each customer and at the end of the day/week/month you make payments to your employees, suppliers or partners.  When making those payments, if those outgoing amounts exceed the amounts received from one customer, you will have to include inputs coming from other customers. If some of your customer addresses are well known (leaks or otherwise) then people can use those addresses to track others when you are batching multiple inputs to make payments. Once they know those addresses belong to you, they will track those to see how they are included in other inputs. So, a determined person can definitely make interconnections thus compromising privacy. 

If you are not re-using addresses, then even if some of them get leaked only few of your addresses gets compromised when you are including them in the transaction. However that will be one-off and snooping will end there. However, if you are persistently using the same unique addresses with the consumers, the problem is compounded as that address will be continuously monitored to compromise other addresses which are also being continuously used.

So now the attacker can create a map of all your customer specific addresses. If you reused the same address for all customer, the attacker could only know the volume of traffic and the revenue from those customers. But with unique address to each customer, the attacker can not only map the above but also now can know customer specific metrics like revenue/customer which is one level deeper and can create competitive issues.