The "surprising find" in Avebury, which is 30m (98ft) wide, is thought to be one of the site's earliest structures.
The square of megaliths appears to have been erected around the remains of a Neolithic house, which sat at the centre of the colossal stone circle -  the largest in Europe.
Archaeologists have hailed the discovery "unheard of".
For more archaeological digs and historical findings from around England follow our Pinterest board
Dr Mark Gillings, from the University of Leicester, said in their "wildest dreams" they had not been expecting to find what he has described as a "stone square circle".
The discovery comes 80 years after the archaeologist and marmalade magnate, Alexander Keiller, discovered a line of standing stones near the 6m (19ft) upright stone known as the obelisk.
The vast 330m (1,082ft) wide stone circle of Avebury was built between about 2850 BC and 2200 BC.
Containing three stone circles and boasting 100 huge standing stones originally, it has been the subject of considerable archaeological interest since the 17th century.
The discovery of previously unknown megaliths inside the monument has been greeted as a "great surprise".
"We have detected and mapped a series of prehistoric standing stones that were subsequently hidden and buried," Dr Gillings said.
"Together, these reveal a striking and apparently unique square megalithic monument within the Avebury circles that has the potential to be one of the very earliest structures on this remarkable site."
Dr Joshua Pollard, from the University of Southampton, said their geophysical survey had "finally completed" Keiller's work.
"It has shown the line of stones he identified was one side of a square of megaliths about 30m across and enclosing the obelisk," he said.
"Megalithic circles are well known from the time when Avebury was built, but square megalithic settings of this scale and complexity are unheard of."
The archaeologists think the stone square might commemorate and monumentalise the "ancestral house of the first people who lived in Avebury".
If proved correct, it may help to explain why the Avebury monument complex was built.
An ancient formation discovered within a Neolithic stone circle in Wiltshire is actually a square.