After completion of his graduate studies, Rao returned to Bangalore in 1959 to take up a lecturing position, joining IISc and embarking on an independent research program. The facility at the time was so meagre that he described it, saying, "You would get string and sealing wax and thats about it." In 1963 he accepted a permanent position in the Department of Chemistry at the Indian Institute of Technology Kanpur. He was elected Fellow of the Indian Academy of Sciences in 1964. He returned to IISc in 1976 to establish a solid state and structural chemistry unit. and became director of the IISc from 1984 to 1994. At various points in his career Rao has taken appointments as a visiting professor at Purdue University, the University of Oxford, the University of Cambridge and University of California, Santa Barbara. He was the Jawaharlal Nehru Professor at the University of Cambridge and Professorial Fellow at the Kings College, Cambridge during 1983–1984.
Rao has been working as the National Research Professor holding the positions Linus Pauling Research Professor and Honorary President of Jawaharlal Nehru Centre for Advanced Scientific Research, Bangalore, which he founded in 1989. He had served as chair of the Scientific Advisory Council to the Indian Prime Minister for two terms, from 1985 to 1989 and from 2005 to 2014. He is also the director of the International Centre for Materials Science (ICMS), which he founded in 2010, and serves on the board of the Science Initiative Group.