At least 44 more have been injured in the crush, which was reportedly triggered by a road accident.
The pilgrims were returning from the Hindu shrine of Sabarimala, which is in a remote, mountainous, densely-forested area.
Kerala Governor R S Gavai has said he is "deeply shocked and saddened".
"We have recovered 102 bodies. The rescue work is almost over," local police official Sanjay Kumar told the Associated Press news agency.
The stampede happened at Pulmedu, near the town of Vandiperiyar in Idukki district, Indian media reported.
Hundreds of thousands of pilgrims had gathered at the hill shrine on Friday, the last day of an annual festival.
The festival runs for two months and attracts millions of visitors every year.
Reports differed as to exactly how the deadly crush was triggered.
The Indian PTI news agency said a jeep carrying pilgrims had driven into a crowd returning from the shrine, starting a panic.
However, the Times of India reported that the incident happened when the jeep broke down and overturned as pilgrims tried to move it.
It crushed a number of people and caused others to stumble, which then triggered the stampede, the report said.
Correspondents say deadly stampedes are not unusual at Indian temples, where huge crowds gather in small areas with few safety measures or crowd controls.
A similar tragedy happened at Sabaramila in 1999, when more than 50 people died in a stampede after a landslide on a crowded hillside.
Officials say 102 pilgrims have been killed in a stampede at a religious festival in the southern Indian state of Kerala.