Error when encrypting .dmg disk image
Can you go into your files and see what the general date of "Modification" for disk util, bluetooth, bootcamp, etc. All the SIP protected applications should all be roughly one date.

I ask because if you didn't get updated properly, disk util got left behind and you need the combo updater I believe.

10.11.3 if you didn't want to update: 10.11.3 Combo Updater
10.11.4: 10.11.4 Combo Updater

Ensure they're "combo" updaters. Even if you're at 10.11.3 or 10.11.4, download and run it anyway, it won't do any harm at all. It'll replace all those programs and files to be correct.

Disk Utility is either having issues with conflicting permissions, or its a bad flash drive. (Also check your FileVault certificate after you encrypt it, if it gives you one.) May not since it isn't internal disk encryption.