Will a channel be automatically closed if there's a expiring HTLC that doesn't have its own output in the commitment transaction? When routed Lightning payments are smaller than some amount they don't get their own HTLC output in the commitment transaction. Instead the funds reserved to it are directed to transaction fees.

When there's an actual HTLC it makes sense for the sender to close the channel and claim it back when it's expired, but it makes no sense to do the same when there's not a real HTLC in there. The same applies to the receiver.

But still, will a channel close in these circumstances?
If it's a mutual close, the HTLC will be resolved prior to closing of the channel. If the close is unilateral, then yes the HTLC which is below the dust_limit_satoshi cannot be resolved. When you are adding HTLC below the dust_limit_satoshi there is some trust element involved and hence we deduct the HTLC amount from our "balance" as a fee in the commitment transaction.

The bigger question at play is asking yourself why you are closing the channel unilaterally. There may be a number of reasons like peer is unresponsive, peer has fallen behind, or worse peer is trying to cheat. In such cases, protecting your balance in the channel is much more important than a couple of satoshis that might have been added below the dust limit.

Moreover, the fees that you will pay for unilateral close will also be higher than a mutual close. If the commitment transaction had a bunch of HTLCs, the fees from the transaction size will be much higher than the couple of satoshis that you might have added in form of HTLC that you have forwarded. So, when you broadcast the commitment transaction for the unilateral close containing HTLCs below dust limit, you will lose that much amount if you have forwarded the HTLC. Your peer will lose the HTLC amount (below dust) if it was the one that had forwarded the HTLC.