_delay_ms() is much slower than expected (by a factor of 6) on TinyAVR 0/1 (ATTiny1604)
Your toolchain is not broken. In the ATTiny804/1604 datasheet, page 77, it shows the CPU clock is divided from the 20/16 MHz oscillator by a prescaler (the ATTiny1614 datasheet shows the same clock prescaler in page 80. So it's a feature found in both tinyAVR 0/1 microcontrollers).

Source: Microchip ATTiny804/1604 datasheet, fair use.
The datasheet says...

10.3.3 Main Clock After Reset After any Reset, CLK_MAIN is provided by the 16/20 MHz Oscillator (OSC20M) and with a prescaler division factor of 6. Since the actual frequency of the OSC20M is determined by the Frequency Select bits (FREQSEL) of the Oscillator Configuration fuse (FUSE.OSCCFG), these frequencies are possible after Reset:

16 MHz ---&gt; 2.66 MHz
20 MHz ---&gt; 3.3 MHz


By default, the division factor is 6. It means the CPU is actually running at 3.333 MHz when the OSCCFG fuse is programmed to 20 MHz.
Solution
Use the correct F_CPU
If a slower CPU is not a problem to you, you can simply report the correct CPU frequency to the compiler by change #define F_CPU 20000000UL to #define F_CPU 3333333UL.
Or disable frequency division
Alternatively, you can disable the frequency division and run the CPU at its 20 MHz full speed.
#define F_CPU 20000000UL

int main(void)
{
    /* Set CPU clock to 20 MHz */
    _PROTECTED_WRITE(CLKCTRL_MCLKCTRLB, 0);

    /* your code */
}

But don't overclock the CPU
Nevertheless, you should note that 20 MHz operation is only guaranteed by Atmel at 4.5 V and above. If you are running the microcontroller at a lower voltage, such as 3.3 V, you are overclocking the CPU by putting it outside the safe operating area designated by Atmel. See datasheet page 473.

Source: Microchip ATTiny804/1604 datasheet, fair use.
Although an overclocked microcontroller will probably work, and in a hobby project, you are unlikely to see any issue. But still, at this point there is no guarantee of stability from the manufacturer. For example, the microcontroller is officially rated at 105°C, but an overclocked microcontroller may start crashing. Increased power consumption is also likely. The worst possible problem could be mysterious malfunction under certain rare conditions. Thus, if reliability and stability is important, you should not overclock the microcontroller.
At 3.3 V, given a 20 MHz system clock, the CPU should only operate at 10 MHz by setting the prescaler to 2x. And at 1.8 V, at 5 MHz with a 4x prescaler.
/* Set CPU clock to 10 MHz */
_PROTECTED_WRITE(CLKCTRL_MCLKCTRLB, 
                 CLKCTRL_PDIV_2X_gc | CLKCTRL_PEN_bm);

/* Set CPU clock to 5 MHz */
_PROTECTED_WRITE(CLKCTRL_MCLKCTRLB, 
                 CLKCTRL_PDIV_4X_gc | CLKCTRL_PEN_bm);