How big of a table or how many data pages in an index is worth considering running index rebuild or index reorganize operations on?
Also "reducing fragmentation" is not per se a performance goal.  On many (most?) modern storage platforms there is little difference between sequential and random IO, which is a major historical reason for defragmenting.

I've worked on systems where the difference in throughput between sequential and random IO was 10x or more.  As SQL Server attempts to scan a fragmented index, the physical file locations of the linked-list of pages jumps around every few extents, reducing the IO size, randomizing the IO, preventing read-ahead IO, and eliminating the benefit of speculative reads into the SAN controller cache.

On modern systems, the storage either has lots of spindles, or is flash-based.  Both of these reduce the performance differential between small, random IOs and large, sequential IOs.  

Also if tables are heavily cached, the benefit of defragmenting their storage on disk diminishes.  And small rowstore tables will typically be cached.