Best codec to use for RockPlayer? What is the best codec should I use in convertion a movie from huge HD/BRay for watching it on Android's RockPlayer?

Also what are the optimal properties should I specify for better performance+quality? (bits-per-second, resolution, audio quality, etc...)
You will get the best results using the built-in media player/decoder (called "system player" in RockPlayer, iirc) because it supports hardware decoding. For a list of supported containers and codecs see here: http://developer.android.com/guide/appendix/media-formats.html

I would recommend H264/AAC inside a MP4 container, like the website recommends, but with your device's native resolution (and a higher video bitrate, if necessary).

Though RockPlayer's ffmpeg decoder is able to play almost every media codec, the results will be quite poor (only software decoding).