Is there anything wrong with this double ternary?
Given your example I think that your double ternary operation should actually be a separate function. When you start getting complex one-liners like this, it is a sign that you need another function. Here's how I approached it:

function getStateName(state) {
  if(state is String) {
    return state as String;
  }

  // This will always return "null" if state.name does not exist.
  // I don't know about AS, but in JavaScript if something doesn't
  // exist it returns as undefined (not null), but you want the null
  // return. As far as I am aware, AS supports the || short-circuit and
  // truthy/falsey equivalence, like JavaScript.
  return state.name || null;
}


This could be used in your function like so;

public static function setStyleInState(styleName:String, value:*, state:*, target:UIComponent = null):void {
    var stateName:String = getStateName(state);

    if (!StateUtils.hasState(target, state)) {
        state = createState(stateName, null, target);
    }
    else if (!(state is State)) {
        state = getState(target, stateName);
    }
}


There's more to be said here; I think that exposing mutability like this on a static method (instead of on the instance of a state) is going to lead you into trouble later on down the road and you should probably re-evaluate your design.

Specifically, state in static methods is always a recipe for disaster when it comes to testability and debugging, especially when you branch into multiple threads.