Commerce order access checks in Views
Your thing you came up with is certainly one way of doing it, though you're adding a step. If you can guarantee that your entity reference field will always be on the order, then you could just join your query to the entity reference table and check it against the current user's UID.

Something to the effect of:

INNER JOIN `field_order_user_reference` ON `commerce_order`.order_id = `field_order_user_reference`.entity_id 
...
WHERE
`field_order_user_reference`.user_id = ***CURRENT_USER***


Or something to that effect.

The way you have it implemented now, you will start to see performance degradation since you're adding a lot of overhead. Using pure SQL will help tremendously and you shouldn't see any slowdown.

Alternatively, since your order has an entity/user reference field attached to it, you should be able to just add it as a filter and set the filter user as the current one. Maybe not, but if you haven't tried it, it'd be worth a shot.