Recovery mode can't read fat32 sd card, only exfat
Have you tried considering a mutli-partition SD card? There are a few tools like SDMinste and Partition Wizard to help you create a dual partition setup on your SD card. You could create a small partition to store all your ROMs and updates, and a bigger partition for your user data.

However, there's a slight problem wherein Windows will only recognize the first partition. Refer this well written guide on XDA for a workaround.