Circular saw blade clogs nonuniformly
You should only be worried about it if it's a problem with respect to your woodworking. Circular saws are by their nature not extremely precise tools. There's a lot of potential for movement as you work. 

Reasons for your findings could be a warped blade, a bent arbor, worn motor bearings, sloppy blade installation, or a number of other things.