Increasing available RAM on Bootcamp under Win7
First, let me explain the problem a little: in 32-bit mode, the processor can only address 4GB of stuff. The problem is that "stuff" doesn't just include RAM, it includes RAM and memory-mapped I/O, most notably the video card. Since your new iMac has a bigger, better video card (with more video memory), it takes up more of the 4GB address space and there's less left for actual RAM.

There are several ways to make more RAM accessible to your program:


Switch to 64-bit Windows. This allows the OS to address all of your RAM, and is really the right way to solve the problem. If you can't recompile your custom program for 64-bit mode, it should be able to run in 32-bit mode thanks to the WoW64 compatibility layer.

Downsides: you'll have to do a custom install of the OS (meaning you basically start from scratch). There's also still a possibility that the program won't work (most likely if it includes 32-bit device drivers), and in this case you'd have to do another custom install to get back to 32-bit Windows. So, while this is really the best solution, you might want to try the other options first.
Enable physical address extension (PAE). If the driver supports it, this'll allow the video memory to be mapped above the usual 4GB limit, allowing you to use more of that 4GB for RAM. I haven't used this, but it looks pretty easy to enable and disable, so I'd be tempted to try this first.
Enable 4-Gigabyte tuning. This shifts how the address space is allocated, giving applications more and the kernel less. I'm not sure how this'd work (or even if it would work at all) in your situation, but it is another possibility. It can also be used together with PAE, so you can try different combinations to see what works.