Entity reference to user doesn't work without 'administer users' permission
You could try creating an entityreference display in views which, with the 'access' options defined in views, handle user access however you please.

Here is an example view display to get you started:
https://gist.github.com/acao/7297342

Just configure the field settings to use this view instead of the default handler, and see if that gets you around your problem!