Media playback is not supported on this device
They were denied a shot at the gold medal after making an illegal change-over in their semi-final win over Ukraine.
Pendleton was penalised for taking over half a bike wheel too early at the start of the second lap of the two-lap race.
Britain have won seven out of 10 track cycling gold medals at the London 2012 Olympics
Men's individual sprint: Jason Kenny (Gbr)
Women's individual sprint: Anna Meares (Aus)
Men's team sprint: Great Britain
Women's team sprint: Germany
Men's keirin: Sir Chris Hoy (Gbr)
Women's team pursuit: Great Britain
Men's omnium: Lasse Hansen (Den)
Women's omnium: Laura Trott (Gbr)
"We were probably just a bit too eager," Pendleton told BBC Sport. "It's one of those things. "It was very unfortunate; it was a mistake made in a fraction of a second."
China then finished ahead of Germany in the gold medal race, but were also relegated for an infringement at the change-over, leaving Germany as champions. China were left with silver, while Australia beat Ukraine in the bronze medal race.
The announcement that Pendleton and Varnish had been relegated to eighth place was greeted by loud boos at the packed London Velodrome.
Dave Brailsford, British Cycling's performance director, argued the pair's case with the judges in the centre of the track for more than 10 minutes before the news was relayed to spectators.
British cyclists won gold in seven out of 10 events and a total of 12 medals on the track at Beijing in 2008, Pendleton and Varnish were on course to keep the run going in London.
They broke the world record in their heat, in a time that was quickly bettered by China in the next race, and were second fastest in the semi-finals, again behind only China.
"It happens so quickly when you are full speed in a team sprint. Jess moves up and that's my cue to take over but it was a metre or so too early, unfortunately," said Pendleton.
"It was on the change. Jess got stuck a bit on the start and perhaps it put her a little bit on edge and perhaps she was trying to make up for the difference she lost.
"Maybe she judged the straight a bit wrong and I just saw the gap and just went for it because that's my cue.
"I'm desperately disappointed for Jess. She has done an incredible job in getting this far. Her team sprint today was the best of her life so far.
"It's the Olympic Games, it's the biggest thing. To spend four years getting ready for it only to lose in a fraction of a second is gut wrenching."
Mark Cavendish on BBC One
"I'm sorry for disappointing all the people that have come to support us and perhaps not offering the ride that we would have done."
Pendleton, who is retiring from cycling after the Olympics, competes in the keirin on Friday and will begin the defence of her individual sprint title on Sunday, but Varnish's Games are over.
London 2012 is the first time the women's team sprint has been staged at an Olympics.
Great Britain's Olympic track cycling campaign got off to a disastrous start when Victoria Pendleton and Jess Varnish were disqualified from the final of the women's team sprint.