Will taking 4 courses in a semester affect my chances of getting into grad school?
I don't think it should be a problem, provided that you are taking some challenging Math/CS classes among your 4 classes.  If it looks like you're trying to skirt by with the bare minimum difficulty of classes, then taking a couple fewer classes could reinforce that negative narrative.  But as long as it looks like you're taking a challenging course load and doing well, I don't know that people will even notice the 4 vs. 5 issue (since there are schools where 4 courses is a normal load).