According to a local legend, the temple was built after the queen had a dream visitation by Lord Rama directing her to build a temple for Him; while Madhukar Shah was a devotee of Krishna, his wife's dedication was to Rama. Following the approval to build the Chaturbhuja Temple, the queen went to Ayodhya to obtain an image of Lord Rama that was to be enshrined in her new temple. When she came back from Ayodhya with the image of Rama, initially she kept the idol in her palace, called Rani Mahal, as the Chaturbhuj Temple was still under construction. She was, however, unaware of an injunction that the image to be deified in a temple could not be kept in a palace. Once the temple construction was completed and the idol of the lord had to be moved for installation at the Chatrubhuj Temple, it refused to be shifted from the palace. Hence, instead of the Chaturbuj temple, the Rama's idol remained in the palace whereas the Chaturbhuj Temple remained without an idol in its sanctum. As Rama was worshiped in the palace it was converted into the Ram Raja Temple; it is the only shrine in the country where Rama is worshiped as a king.