In 2014, the largest colony of the rare seabirds was at Winterton, near Great Yarmouth, Norfolk but the birds have since moved north to Sea Palling.
Two years ago there were 300 pairs of the rare seabird. This year there are just 190 pairs at Sea Palling.
The RSPB said the species is declining.
Little terns travel 3,000 miles from west Africa to breed on the UK coast but, according to the RSPB, the numbers returning to the UK have declined by between 30-50% since last year.
It said the site near Sea Palling is still the country's largest colony, despite the significant drop in numbers.
Fabienne Fossez, the Little Tern warden for RSPB East Norfolk, said: "We're extremely concerned, it's a species in decline and they are in real trouble.
"What we don't know is what goes on out in west Africa in the winter but it is something to do with the climate, weather and food.
"Nevertheless, we have a really good success story here near Sea Palling, despite all of the odds."
Ms Fossez said 380 chicks have hatched at the east Norfolk site since mid May.
Her team has been maintaining a 24-hour surveillance of the colony in order to protect the birds.
But she said the breeding birds have not had the same level of success elsewhere.
"The little terns have had a really difficult time this year," she said.
"On the north Norfolk coast they were completely washed out at the beginning of June. We were protected here on the east coast and we are fortunate that there are just little terns nesting here."
Wildlife conservators have said they are "extremely concerned" that the number of little terns nesting in the UK's biggest breeding colony dropped by almost half in a year.