How can I change the target resolution of a VNC Server
I had a similar issue years ago using VNC with an ancient Power Mac 7100.  The 7100 was headless and the video defaulted to 800 x 600.

I discovered that the VNC server doesn't care what the resolution is but that it is limited to resolutions that the remote machine will support.

In my case, after connecting the remote 7100, I then using the Displays Control Panel to crank the display to the highest resolution the remote machine could handle.  The VNC screen then changed to match the new remote resolution.

Unfortunately, this means you can only use the maximum resolution that the remote machine's hardware will support.