The regulator said the bid did not comply with the laws of having its principal place of business in the Chinese territory.
The city's other carriers had also objected, it said.
Qantas said it would reconsider basing the low-cost carrier in Hong Kong.
The carrier is a three-way joint venture between Qantas Airways, China Eastern Airlines and Hong Kong investment firm Shun Tak Holdings.
"Jetstar Hong Kong cannot make its decisions independently from that of the two foreign shareholders," said the Air Transport Licensing Authority when handing down its decision.
It also said the city's other carriers, including Cathay Pacific and Dragon Airlines, had objected to Jetstar's application.
Qantas chief executive Alan Joyce said the decision was disappointing for shareholders as well as "for the travellers that Jetstar Hong Kong planned to serve".
"It's the travelling public who have lost out, because the message from this decision is that Hong Kong appears closed to fresh aviation investment even when it is majority locally-owned and controlled," he said in a statement on Friday.
He added that Hong Kong was going in the opposite direction of other aviation markets in Asia that were "opening up".
"Given the importance of aviation to global commerce, shutting the door to new competition can only serve the vested interests already installed in that market."
The airline, along with its partners, said it had not yet decided on whether to appeal the decision.
The joint venture was valued at $10m Australian dollars ($7.7m; Â£4.9m), according to Qantas.
Shares of Qantas were down 1.4% in Sydney in reaction to the news.
Hong Kong authorities have rejected an application by Qantas-backed Australian budget carrier Jetstar to set up a local airline, after nearly two years of deliberation.