Reusing oil containers
Yes these are realistic concerns.


the tiniest amount of rancid oil can make your dish taste bad.
mold can grow on fat (e.g. penicillium in cheese).
oil only has residual amount of water, so it will not accelerate most bacterial growth.
shelf life for oil is about 6 months to a year in a dark cool cupboard
shelf life is about a year and more in fridge (but the oil clouds and solidifies). 


my suggestion would be that when your container is empty, just wash it with a natural degreaser (e.g. vinegar) instead of topping it off. Or top it off, but wash the smaller bottles every 6 months.

UPDATE

Be very careful about what kind of oil you use, in particular if it is not commercial-grade. In this other question you will see that oil or other canned products that provide anaerobic environment (=without oxygen) can be good conditions for some bacteria. Some of them are specifically bad. See also the wiki page for the C. Botulinum for an example.