Whether or not you have to file your taxes depends on your retirement income source, as well as how much you draw out of it within the tax year. Retirees whose only source of retirement income is social security won't owe any federal taxes. As such, they don't need to file a tax return.