He is the first American Secretary of State to visit the city.
In a meeting with Somalia's president under tight security at the airport, he said: "Next time I come, we have to be able to just walk downtown".
The US backs the Somali government in its fight against Islamist militant group al-Shabab, which still controls many rural parts of southern Somalia.
African news as it happens: 5 May 2015
The meeting with President Hassan Sheikh Mohamud and civil leaders was held at Mogadishu airport, as Mr Kerry isn't leaving its heavily fortified perimeters.
Mr Kerry said: "I visited Somalia today because your country is turning around".
The more recent changes he pointed out were Somali forces pushing al-Shabab out of big towns and a "determined international effort" to put virtually all of Somalia's pirates out of business.
BBC Somalia analyst Mary Harper says the US plays a crucial if controversial role in Somalia, supporting the weak central government in its fight against al-Shabab.
Al-Shabab leaders have been killed in US drone strikes.
Our correspondent points out that America is careful to take a less visible role than it did in the 1990s, when the bodies of US servicemen were dragged through the streets of Mogadishu.
The US Secretary of State John Kerry has arrived in the Somali capital, Mogadishu, on an unannounced visit.