How do you fade the edges of a model?
Suppose you sculpted this beauty out of a plane:



To make the flat area transparent you can create a node setup like this, where the transparency is controlled by the height (Z axis) of the model.



EDIT:

If you are working on an older version of blender that doesn't have the separate XYZ you can use a separate RGB node:



For finer control on the transparency use a Vector->Mapping node. By moving the Location values on the Z axis you can determine the height where the transition starts, and by moving the sliders on the color ramp you can determine how smooth the transition will be.