Can I relocate my electric water heater under a basement window?
Is relocating the water heater under a window a problem? 


Not unless you need to get to that window for some reason such as egress as previously mentioned. 


  Would the pipes be more susceptible to freezing near the glass of the window? 


Yes. Since glass has a much lower insulative rating than just about anything else.


  Would I need to make sure I had heat from the furnace warming that utility/storage area?


How often is the water flowing in those pipes? In a full basement the temperature shouldn't drop much below 50°F in the dead of winter since most of the basement is below ground. However pipes close to outside walls (or windows) risk freezing in winter. Their risk of freezing goes up, as the amount and frequency of water flow goes down.

So, if you want to locate the water heater under that window and you don't need to use the window, make sure you block off and insulate the window, and insulate the pipes (especially the cold). If you heat the space that will preclude the insulation but may cost you more in the long run. Ultimately, doing both would ensure sleeping comfortable on cold nights without worrying about frozen broken pipes.

Good luck and Happy Turkey Day!