Correctly interpreting Cosine Angular Distance Similarity & Euclidean Distance Similarity
If you look at the definitions of the two distances, cosine distance is the normalized dot product of the two vectors and euclidian is the square root of the sum of the squared elements of the difference vector.

The cosine distance between M and J is smaller than between M and G because the normalization factor of M's vector still includes the numbers for which J didn't have any ratings. Even if you make J's vector more similar, like you did, the remaining numbers of M (2 and 5) get you the number you get.
The number for M and G is this high because they both have non-zeroes for all the books. Even though they seem quite different, the normalization factors in the cosine are more "neutralized" by the non-zeroes for corresponding entries in the dot product. Maths don't lie.

The books J didn't rate will be ignored if you make their numbers zero in the computation of the normalization factor for M. Maybe the fault in your thinking is that the books J didn't rate should be 0 while they shouldn't be any number.

Finally, for recommendation systems, I would like to refer to matrix factorization.