What's the key to sawing straight by hand?
I can think of two possibilities:  1) you're beginning your cut with the saw out of alignment, or 2) the saw teeth are not set correctly. ("Set" is the distance of the tip of each saw tooth from the center.  If you look, you'll notice that the saw teeth are bent slightly away from the saw plate in alternating directions).

Let's assume you have already worked out your body geometry and you're holding the saw straight to begin with.  It's possible that the saw teeth are set more on one side than the other. This can cause a saw to drift off the intended cut line.