Japan administers the uninhabited Senkaku islands, which China also claims under the name Diaoyu islands.
Russian naval ships were also spotted in the area at the same time, Japanese local media said.
In 2012, relations between Japan and China deteriorated after Japan bought the islands from a private owner.
The islands are important because they are close to key shipping lanes, offer rich fishing grounds and lie near potential oil and gas reserves.
Reacting to the incident, Japan's Chief Cabinet Secretary Yoshihide Suga expressed "serious concern" at a news conference, saying it would escalate tensions.
Prime Minister Shinzo Abe has told his government to work closely with the US and other countries to deal with this, Mr Suga said.
How uninhabited islands soured China-Japan ties
"Around 00:50 am (15:30 GMT Wednesday), a Chinese naval vessel entered our nation's contiguous waters surrounding the Senkaku Islands," the Japanese foreign ministry said in a statement.
Japan's Vice-Foreign Minister Akitaka Saiki summoned China's ambassador Cheng Yonghua to lodge a protest.
Mr Saiki told the ambassador that the Senkakus are an inherent part of Japan's territory, in terms of history and international law, NHK reported.
He demanded the vessel swiftly move away from the area. Local media reported the vessel left the area around 03:10 local time.
Mr Cheng said the Chinese frigate was allowed to sail in those waters, said Kyodo news agency citing an unnamed source.
Three Russian military vessels were also seen in the waters around the disputed islands, according to Jiji Press, entering around 21:50 on Wednesday and leaving around 03:05 on Thursday.
Japan has lodged a protest with the Chinese ambassador in Tokyo, after a Chinese ship sailed close to contested islands in the East China Sea.