Create a view of nodes based on user record
For entity reference fields there are 2 types of relationships added to views: a Referenced entity and a Referencing entity. 


Referenced entity is used for a reference from the base table (e.g. A user view that references a fruit node).
Referencing entity is used for a reference to the base table (e.g. a fruit node view that references a user)


In your case, it's Referencing entity that you want. 

Create a node view and a Entity Reference: Referencing entity relationship (the description should read A bridge to the User entity that is referencing Content via field_FRUIT_NODE_REF) that refers to your fruit node reference.

That should handle the data/reference part. What remains is aggregation/filtering (e.g. there will duplicates if fruit node references aren't distinct for each user) and display (e.g. how to display marking a fruit node referencing a current user).

Depending on your needs, you might need a custom field/template since there is a limit of what you can do with field basic rewriting and default user contextual filters.