Bathroom fan fluctuates in power when using countdown timer switch
There are actually two problems, and they are related:
Low Power Fan
First I found a customer review on the Lowes web site that said:

I gave this only three stars as no where on packaging or instructions is there any note indicating this will not work with high efficiency bathroom fans. I spent many hours troubleshooting my Broan XB110L only to find during internet searches that this timer causes erratic fan operation. Actually in my experience this timer results in a fan that never reaches full speed and labors erratically at very low speeds.

which matches your problem pretty closely. And then I found this comment from a Home Depot customer:

Note: This timer requires a load of at least 40 watts to operate properly.
and that matches your problem exactly. Your fan is rated at 30W. Not clear if that is Fan + Light or just Fan. But even if it is just the fan, the LEDs are 13W so that would be a total of 43W - and likely less because the fan motor is probably listed at the high end of its usage (e.g., startup).

which led me to the definitive answer in the installation instructions:
40 Watt Minimum


Do not use where total load current is less than 0.3 A 40 W or greater than 5 A


which confirms the problem.
Not LED Compatible
I was at first a bit surprised at this, as the device seems quite up-to-date and even uses LEDs in its own design. But both Lowes and Home Depot list it as not LED compatible. Since your fan has LED lights, that's a no-go right there.
Why is there a minimum load? There are three ways to power a smart-switch/timer/motion-detector/etc.:

Neutral

Neutral requires a neutral wire. Many older homes don't have neutrals in the switch box, so it is a marketing advantage to have a product that doesn't require neutral - fewer returns due to &quot;can't install it here&quot;. This timer does not use neutral.

Ground

Ground is a general &quot;no, you can't do that&quot; but it is allowed when properly designed but has a limited amount of current allowed for the &quot;smarts&quot; of the device. This timer does not use ground.

Leak through switched hot

This is the only option left! Works great with incandescent &amp; halogen but does not work well with LEDs. And I suspect they have some tricks that just don't work well unless there is a large load so that the timer's own usage is &quot;small&quot; relative to the fan/light load.
Bottom line: SEND IT BACK
Recommendations
Look for a timer that says:

LED Compatible - That indicates it doesn't &quot;leak current&quot;
Requires Neutral - That tells you how it gets the necessary current