Best codec to use for RockPlayer?
You will get the best results using the built-in media player/decoder (called "system player" in RockPlayer, iirc) because it supports hardware decoding. For a list of supported containers and codecs see here: http://developer.android.com/guide/appendix/media-formats.html

I would recommend H264/AAC inside a MP4 container, like the website recommends, but with your device's native resolution (and a higher video bitrate, if necessary).

Though RockPlayer's ffmpeg decoder is able to play almost every media codec, the results will be quite poor (only software decoding).