According to a legend this temple was consecrated by a sage named Swetharshi. The legend says Agasthya Muni on his way to River for offering his customary ablution was confronted by two demons named Neela and Swetha. Agasthya Muni cursed them. On their request for penance, Agasthya Muni advised them to perform Tapas at two Shiva Temples. The place where Neela performed Tapas came to be known as Neeleswaram. Swetha performed his Tapas at Thiruvangad (Tiru-van-kad, meaning the sacred dense forest). They were later known as Neelarshi and Swetharshi. It is said that Swetharshi consecrated a Shiva temple & Vishnu temple at the place where he performed his Tapas. It is believed that the Shiva installed by Swetharshi was very fierce, hence to calm its fury another Shiva temple was consecrated just opposite. Now the temple complex consists of two Shiva temples facing each other on the north side and the Big Sri Rama temple on the South. There is also a slightly different version of the legend, Which says that Shiva Temple was consecrated by Neelarshi - a Shiva Bhakta and Sree Rama temple was consecrated by Swetharshi - a Vishnu Bhakta.