Can I update my Galaxy Nexus manually?
AFAIK, the 4.0.4 is currently only available for the CDMA/LTE, but someone over at Modaco has a modified version of the 4.0.4 working for GSM.

It looks like everything is working on the GSM version with this modified release. I have also read that 4.0.5 will be released next month, so you may want to just wait.