jQuery plugin for easy CSS3 transformation
I like the demo, from a once over:


the functionNames make my eyes glaze over, data tables variables should be sorted and tabbed in my mind:

  var functionNames = [                                   
    { name: "matrix",      type: functionParamTypes.length}, 
    { name: "matrix3d",    type: functionParamTypes.length}, 
    { name: "perspective", type: functionParamTypes.angle },
    { name: "rotate",      type: functionParamTypes.angle }, 
    { name: "rotate3d",    type: functionParamTypes.angle }, 
    { name: "rotateX",     type: functionParamTypes.angle }, 
    { name: "rotateY",     type: functionParamTypes.angle }, 
    { name: "rotateZ",     type: functionParamTypes.angle },
    { name: "scale",       type: functionParamTypes.number}, 
    { name: "scale3d",     type: functionParamTypes.length}, 
    { name: "scaleX",      type: functionParamTypes.number}, 
    { name: "scaleY",      type: functionParamTypes.number},
    { name: "scaleZ",      type: functionParamTypes.length},             
    { name: "skew",        type: functionParamTypes.angle }, 
    { name: "skewX",       type: functionParamTypes.angle }, 
    { name: "skewY",       type: functionParamTypes.angle }, 
    { name: "translate",   type: functionParamTypes.length}, 
    { name: "translate3d", type: functionParamTypes.length}, 
    { name: "translateX",  type: functionParamTypes.length}, 
    { name: "translateY",  type: functionParamTypes.length}, 
    { name: "translateZ",  type: functionParamTypes.length} 
  ];                                

It is preferred to have 1 single comma-separated var statement:

var $this = $(this),
    existingTransformation = $this.css("transform");

It is preferred to have either all single or all double quoted strings, I would suggest single quoted strings
You have some missing and some unnecessary semicolons, please check out http://jshint.com/
Other than that, a fun piece of code