Estimated number of rows is inaccurate
You could create additional statistics for the data. Include a WHERE clause so each new statistics object references only a subset of the table. By bucketing the rows appropriately (big, medium, small?) the filtered statistics will be less bad than the whole-table ones, leading to better plans.

The predicate must match that used in queries. Depending on your settings there may be more system activity for stats maintenance. Of course this assumes the table contains (or could change to contain) some combination of columns on which rows can be meaningfully aggregated.