Clustering of users in a dataset
If your objective is to find clusters of users, then you are interested in finding groups of "similar" reviewers.

Therefore you should:


Retain information which relates to the users in a meaningful way - e.g. votes_for_user.
Discard information which has no meaningful relationship to a user - e.g. user_id (unless perhaps it contains some information such as time / order).
Be mindful of fields which may contain implicit relationships involving a user - e.g. vote may be a result of the interaction between user and ISBN.