How do I cancel a half completed email configuration?
Reposting answer for future people!

Naturally, I figured it out 10 minutes after posting.

Can't answer for another few hours because I'm a new user. Under the security section of the settings, there is a Device Administrator page.

I deactivated the "Email" app as an admin. It warned that it would delete all my mail and stuff and I accepted. No factory reset required.

Thank you Stack Exchange for allowing me to express my question so I could figure it out myself :)