Register an anonymous user when entity form is submitted with rules
I thought the field validation module should have been able to do this but I couldn't make it validate against the user mail field. Maybe you could try that and have better luck.

The other option which definitely works and requires no coding is to use the Views Bulk Operations (VBO) module and use rules conditions to check the number of results for that email address.

a. VBO Mail Check View


Create a new view page to display users.
Add the "Bulk operations: User" field (no changes needed).
Add the contextual filter, "User: Email" and provide the default, "Raw value from Url, path component 2".
Test with the preview using a valid email address and make sure one result shows before proceeding.


b. New rule condition in your main rule for when the entity form is saved.


"Check number of results returned by a VBO View"
Choose the view and page display you created above.
Send the text field from the entity form as an argument. (This will be path component 2 in the VBO view above)
Set the minimum value to 1.
Negate the condition.


This will make the rule fire if the VBO view returns zero results ie. no email.

c. Create a new rule to show an error if the mail exists.


Same condition as above.
I don't really know why this works but anyways. Add an AND above the condition, negate it and indent the condition which is also negated. (move it to the right like it's a child). It will look like


Entity has field

NOT AND 

      NOT Check number of results returned by a VBO View


Action: Display site message and tell the user that the email is already in use.


It looks a bit convoluted but should only take a few minutes to get working. Good luck!