Bike seems less stable when using handlebar bag
Yes, the bag can cause the instability. The physics is quite simple.

Ordinarily, a bike is self-stabilizing. If you turn the handlebars to the left, you steer to the left and the bike leans to the left. However, the turning gives a centrifugal force to the right, which causes the bike to want to lean to the right which, in turn, straightens it out.

With a bag on the handlebars, things change. With the bike perfectly upright, the weight of the bag acts completely downwards and does nothing to turn the handlebars. However, as the bike leans to the left, the bag can lower itself by turning the handlebars further to the left.  If the bag is heavy enough, this effect will overcome the self-righting stability of the bike and now you have a system that responds to a slight turn to the left by trying to turn more sharply (positive feedback) rather than the usual situation of responding to a slight turn to the left by trying to go straight again (negative feedback). This is the instability that you felt while riding.