More than a third of IRS employees will soon be eligible to retire, which could also lead to longer waits for things like taxpayer refunds and audits. Federal employees can retire at the age of 62 if they have worked five years, or even younger with 20 years of service. The average federal worker is 47 years old with about 14 years on the job, compared to the median age of 42 for all American workers, according to federal data.