Is there a way to use custom procedural textures in Cycles?
As mentioned, Open Shading Language is likely what you are after. The Blender Wiki has good OSL info. In addition, you may want to check out: 

www.openshading.com

If OSL is not your thing, you may want to check out the many Blender procedural experiments (incl. Python) by Jimmy Gunawan at: blendersushi.blogspot.com