Does crown direction really matter when framing a wall?
Yes it is.  If all your studs are oriented the same way, you'll have a gentle bow to the wall (either in or out) that is more or less consistent across the width of a wall.  

On the other hand, if you have them so that one stud curves in and the next curves out, you're going to have irregularities in your wall that are the twice the size of the curvature of the studs, and that will be much more noticeable.

Here's an article that explains it well.