Varadarajan was born in Thoothukudi, Madras Presidency in 1926 to a union leader who was shot to death by the police.
Varadarajan ran a parallel judicial system within the South Indian community in his strongholds.