Entity Reference Views - Contextual Filters - Administrator override
I'm not really sure why this works but anyways.. It seems to be what you want.
You'll need Views Contextual Filters OR.
In my test, I used the contextual filter for author UID. So your entity reference field would only shows nodes created by the current user.

In your view, add the Content: Author relationship (don't check required) and Content: Author uid contextual filter (provide default: current user).
Enable Contextual filters OR in Query settings under Advanced.
Add a second contextual filter for Content: Author uid (provide default: current user)
Under More at the bottom of the filter, click exclude.

When added this an entity reference field, normal users only see nodes they've created but as admin, I see every node on the site. Weird but it works.