How do I clear iBooks download queue on OS X?
Deleting the iBooks cache in /private/var/folders/randomLettersNumber/C/com.apple.iBooksX did the trick for me.

An easier way at least on OS X 10.11 is to click a book in the download queue, select all, then delete. Unfortunately iBooks will ask you to confirm the deletion of each book download.