Understanding why a query is slow
Without seeing the plan...

Updating all 92k rows


scans the table start to end
updates all rows (whether value is the same or not)
clustered index leaf = the actual data


= you are actually doing clustered index update

When you updating 3,200 rows means you are doing 96.5% less data changes work (3200 is 3.5% of 92000) so of course it will run faster

Also the transaction size (for rollback) in the transaction log will be a lot bigger.
This needs allocated and may need a log file size increase

Basically, update only the rows you need...