Views Filter: Has referencing entities
If you are willing to have an extra field, consider using Corresponding Entity Reference.


  It syncs the entity reference between two entity types which have an entity reference to each other, so double editing entities is no longer needed. If one entity has a reference, the other entity also receives a reference to the saved entity if it is referenced in that entity.


So basically, if properly configured, it will give you "is referred by" field you mentioned in your last paragraph.

For Drupal 6, use Corresponding node references module.