FPGA VGA Buffer. How to read and write?
Sprites are not normally done with a frame buffer (as I understand the word). Instead you compare the x and y coordinate with the xmin,ymin and xmax,ymax of the sprite.  If the current scan position falls inside the sprite, output the relevant colour from the sprite memory.

If you are trying to display a frame buffer, take heart.  This was my first major FPGA project.  SDRAM shouldn't be a problem at 100MHz (I first did this about a decade ago, and silicon is waaay faster now), so multiply up your 50MHz clock. Writing your own controller will be educational :)

That gives you plenty of bandwidth to play with, you can then double buffer, no problem.  60Hz VGA needs an average of 18Mpixels/sec.  If you have a 16-bit wide device, you have 200MBytes/sec of peak bandwidth.  Even if you only manage to be 50% efficient (which should be doable) that's 100Mpixels/sec @ 16 bits per pixel or 50Mpixels/sec @ 32bits per pixel.  

For example, it may be that your RAM needs 60ns to setup a read, but can burst 8 words in 80 ns after that - that's 8 bytes in ~140ns.  IF you can get your RAM to do longer bursts, that would help amortise the cost of setting up the read.

Based on your comment that it's a byte-side RAM, that's a bit over 50MBytes/sec, only 16Mpixels/sec @ 24 bits per pixel :(  You just don't have enough bandwidth to do a truecolour display, even at VGA.  You could do 8 bits per pixel pretty readily, but that's only 2 or 3 bits per colour - which may be OK for your application, I don't know.  Or you could do a 256-colour lookup table like in the old days - after reading from the frame buffer you then use the value to lookup in an internal RAM clock to get the 24-bits (or 18 bits, which would fit nicely in a BRAM) of colour to output to the monitor. 

The double buffering still works:

Display a frame from address 0 (just read all the pixels out in turn, pausing the reads during the blanking intervals).

Write your next frame into another location.  For this double buffering, your DRAM controller will have to prioritise between the competing demands of the read and write channels.  Hint, prioritise the reads as they are time-critical :)