Modifying VGA Controller For My Monitor
You're missing an important detail -- the pixel clock for the new video mode is completely different! It requires a 109 MHz pixel clock, as compared to the 25 MHz pixel clock used for the 640x480@60Hz mode.

This 109 MHz clock cannot be derived directly from the 100 MHz input clock; you will need to use the clocking wizard to generate this signal.