Overturning a 2013 ruling, the judges did not, however, halt the programme but urged Congress to take action.
The NSA's spying was leaked by Edward Snowden, a former NSA contractor  who has since fled to Russia.
The NSA has collected data about numbers called and times, but not the content of conversations. It also allegedly spied on European firms.
Among individuals targeted was German Chancellor Angela Merkel.
Germany has "drastically reduced" internet surveillance for the NSA, reports say, after the US agency failed to provide "clear justification" for each search.
Members of Congress could "help reinforce the court's decision" next week, American University's Gordon Adams tells me. That's when House members are expected to vote on a bill, the USA Freedom Act, that would end the NSA's collection of bulk data.
Some senators, however, want things to remain the same. They've pushed for an extension of a provision, Section 215 of the USA Patriot Act that allows the NSA to collect the data.
Section 215 expires in June, and an extension would allow the NSA to carry on with its work.
US spy leaks: How intelligence is gathered
How vulnerable is the internet?
US states take aim at NSA facilities
The latest verdict, by the 2nd US Circuit Court of Appeals in New York, came after New York District Judge William Pauley had dismissed a legal challenge by the American Civil Liberties Union (ACLU) which argued that the way the NSA tracked million of calls contravened the US constitution.
The 97-page ruling says that "a provision of the USA Patriot Act permitting the Federal Bureau of Investigation to collect business records deemed relevant to a counterterrorism investigation cannot be legitimately interpreted to permit the systematic bulk collection of domestic calling records".
However, the appeals court stopped short of ruling on the constitutionality of the programme, launched after the 11 September 2001 attacks on the US.
The Snowden revelations in June 2013 caused an international outcry, despite US administrations insisting the programme has been fully authorised.
The measures - repeatedly approved in secret by a national security court since 2006 - are set to expire on 1 June.
Leaders of the lower US House of Representatives would prefer to pass a bill to end the government's bulk collection of phone records and replace it with legislation that supporters say protects civil liberties. But Republican Senate Majority Leader Mitch McConnell has indicated he wants to extend the Patriot Act and retain the bulk collection programme.
The White House supports "an alternative mechanism to preserve the program's essential capabilities without the government holding the bulk data", said Ned Price, a spokesman for the White House National Security Council.
But the ACLU's deputy legal director Jameel Jaffer said: "The appeals court's careful ruling should end any debate about whether the NSA's phone-records program is lawful."
A US appeals court has ruled that bulk collection of phone records by the National Security Agency is illegal.