How do we identify left handed and right handed crystals?
Yes, you are right that the handedness refers to the molecular 3D arrangement. Such handed molecules are stereoisomers known as enantiomers.

I think that the first part of your question will make more sense following an answer to the second part. So with regard to the part about discerning handedness by looking at it: 

(I couldn't get to the article right now due to network issues, so I hope I'm not repeating too much.)

A molecule is chiral when one of the atoms in it has four unique groups attached to it. 
Chiral molecules observably rotate plane-polarized light, either clockwise or anti-clockwise (see here for a picture of what plane-polarized light is). 

Pasteur observed this along with the cancelling effect of a racemic mixture. As in any scientific endeavor, Pasteur was building upon the efforts of previous work--in this case, that of Jean-Baptiste Biot in studying tartaric acid. Pasteur's careful work enabled him to deduce that the naturally occurring or biologically produced form of tartaric acid was enantiomerically pure because it rotated light and the mirror image crystals also rotated light opposite directions. This was in contrast to the racemic acid which did not rotate light since it was a 50:50 mixture that cancelled the rotation in either direction.

Bryn Mawr College has a helpful explanation of how enantiomers interact with light:


  You have probably [not] learned in lecture that chiral molecules have optical rotations and that enantiomers rotate plane polarized light in equal and opposite directions... This behavior upon exposure to plane polarized light is not some magical property that defies the general principles of enantiomers. The reason enantiomers interact differently with plane polarized light is that plane polarized light is chiral. Though polarized light is a planar phenomenon, it is actually comprised of two superimposed helical forms. You can think of the electrical component as oscillating in such a way that there are two electrical vectors rotating in a cork screw fashion in opposite directions away from the source of the propagation at the same rate. Since a helix is a chiral shape , the two versions of the light constitute an enantiomeric pair. If you resolve the vectors of the two superimposed helices, you end up with oscillation in a single plane...Enantiomers interact differently with the two forms of light. The basic idea is that when plane polarized light encounters enantiomerically pure, chiral molecules, one form of light will slow down more than the other as a consequence of the interaction. This results in one helix coming out of sync with the other. When the vectors are resolved, the resultant plane is rotated with respect to the original position. This change in position is what we mean when we say a compound has an optical rotation. 


So looking at a crystal being exposed to polarized light should allow you to observe whether it rotates light to the right or left. I'm not sure how easily detectable to the eye the rotation is, but polarimeters are instruments which are used to measure the angle of rotation.