Should I help an aggressive panhandler?
And that is ok. Suppose you would have helped him, would you have felt aversion towards yourself because you allowed yourself to be tricked and towards him because he tricked you? Would his life improve beyond imagination if you would have bought the item? 

Is it possible that you are allowing a trifling matter inflict exagerated suffering upon yourself? 

One can accept the past the way it is and let it go.