Address reuse, one address per person I know about the privacy and security implications of address reuse. But I was thinking about a scenario where I have one address for every person I receive transactions from. This would still have the security problems, but not the privacy problems? Did I overlook something here?
but not the privacy problems? Did I overlook something here?


Yes, unfortunately.   When you later spend you often need to use coins originating from multiple transactions that previously paid you and these co-spends will link together your transactions.

For example.  Say Alice pays you 1 bitcoin to address A and then Bob pays you 1 bitcoin to addess B. You then spend 2 bitcoin and use those to coins, indicating to the world that the same party probably controls A and B.  Later Bob pays you 1 bitcoin again and Charlie pays you 1 bitcoin to address C.  You make another 2 BTC payment using the coins from Bob's second payment and Charlie's payment, now the world has reason to believe A, B, and C are all controlled by the same party.  Without the reuse people would know less.

This privacy loss doesn't just hurt you, but also A, B, and C and potentially the people they interact with so even if privacy isn't very important to you today it's still better to not gratuitously lose privacy.

Bitcoin Core has some logic to try to spend all payments to the same address at once (though in 0.17 it isn't enabled by default) but even that won't help you if the payments/spending are spread out over time. If you are forced to reuse addresses it's preferable to delay spending payments to a particular address until its unlikely to get any more and to then spend them all at once. 

Using one address per paying party is the bare minimum needed to even know who paid you but doesn't provide particularly great privacy.