Can a dedicated GPU use/share system RAM in addition to the VRAM included with it?
I believe "NO" is also an answer :(

The physical and design of the discrete (AMD in your case, nVidia in other MBPs) is such that their VRAM is on separate bus connected only to the GPU.

The IGPs (which are part of the CPU's die, ie. physical chip) shared the DRAM of the computer. That is where the term "shared" memory comes in, because the CPU and IGP have a shared access to the "computer's" RAM :)

Thus, if your application needs 512MB display memory, then you are out of luck with your current MBP ;(