Filter content by own content or admin
Background

So I have an entity reference field on a node, the widget is a
select-box.

You (Risse) have already done this, but to make this clear for other users, this involves adding an entity reference field to a content type and in the field settings changing from the default 'Simple..' to 'Views..':

Now the select list used to choose a referenced entity (in this case a node), will use views to create the select list, allowing us to customise the nodes listed. In the example above I have created a view (and added an 'Entity Reference' display) and called it entity_reference_select.

I want so that an user can choose one of their own authored nodes as a
reference OR if an admin is logged in, they can choose any content. So
admins should be able to bypass this restriction (with a permission,
role or something).

It is easy enough to list only nodes by the current author using the filter &quot;Content: Author uid&quot; in the view (as per @molot above). Unfortunately this does allow us to differentiate the results for admin users.
An appropriate response might initially seen to involve an OR statement in the views filters to allow admin user to see all nodes in the select list, and other users to only see their own nodes. Unfortunately there are two issues with this:

There is no way to create a views Relationship to the current user in order to evaluate the current user's roles in the filters section
An OR conditional in the filters will remove options, not add them, so would  remove non authored nodes from an admin role user anyway

Solution
Provide a default value for a contextual filter which is defined by PHP. This means we can specify which user(author) we are going to filter the results by, which would be the current user (fetched from the global $user object) unless the current user has the 'administrator' role, in which case we list nodes authored by all users.