lavd/x11grab: check 32-bits color masks.

The X11 servers by VNC, at 32-bits depths, has the following masks:
R:0x000007ff G:0x003ff800 B:0xffc00000
This is not compatible with AV_PIX_FMT_0RGB32, and the result
is success with completely wrong colors.
