Is it OK to add citations from low impact factor journals?
Of course, you should cite any relevant work.

You can't ignore work just because it is in a journal that someone believes to be lesser. That doesn't mean the work is bad. 

As long you do a thorough literature review and avoid predatory journals, there should be no negative impact on your paper.