How to achieve this effect? Does it need a custom shader?
Its a open source fork of Blender called Bioblender, which can generate 3D models from protein database files.  As far as I can tell, the program uses the Blender Internal renderer to generate images.