PWM fans causing temp sensor erroneus values (interference?)
In case anyone has the same problem, I solved it. Turns out is was a bad PSU. Buying a new, higher quality one made everything work. So the issue was high ripple I suspect, or at least high enough when applying some load (which is why I noticed that things got worse after the fans started spinning at greater than 20% duty cycle) for the temperature sensor to flip out.