Mac Mail has very slow to send/receive mail (plus other bugs too)
I also had extremely slow mail using Yosemite until I did this:

From the Mail menu, click Preferences > Accounts > Mailbox Behaviors

Drafts:

Store draft messages on the server > leave unchecked

Sent:

Store sent messages on the server > leave unchecked

Junk:

Store junk messages on the server > checked
Delete junk messages when > Never

Trash:

Move deleted messages to the Trash mailbox > leave unchecked
Store deleted messages on the server > leave unchecked