Drupal function field_get_items vs user_load
No, You can't use field_get_items without the loaded user object. As the entity has to be passed as the second argument as illustrated in documentation.

Any reason why you hate user_load_multiple? As user_load itself uses user_load_multiple.

If you don't want to retrieve that huge amount of arrays. You can consider building your own query to retrieve just the field you want.