Using a 16MHz crystal with an atmega328, it works only if I remove the two capacitors connected to the crystal
22pF is too much already, the ATmega328 datasheet, Section 13.3/13.4 specifies 12pF to 22pF on both XTAL1 and XTAL2 for both the 16MHz full swing and the low power oscillator options, and your layout may already provide a few pF.

If you use a breadboard, it has about 15pF between adjacent clamps.