His comments were captured in a recording of a closed-door meeting, the Daily Beast reported.
The current round of direct peace talks between the Israelis and Palestinians appear to be at an impasse.
But President Barack Obama has said the US will not abandon its peace efforts.
By Paul DanaharBBC Washington bureau chief
The US state department is telling everyone that it's no big deal that John Kerry used the 'A' word to describe the impact on Israel if the two-state solution for the Israel-Palestinian conflict collapses.
A spokeswoman has pointed out that senior Israeli politicians have also referenced apartheid before when talking about the risks to Israel's reputation. This point though is frankly spin. It is one thing for Israeli politicians to use provocative language in their own political arena. When used by the US secretary of state it adds legitimacy to the debate about whether there is an equivalence between the old South African regime and the situation on the West Bank.
Whether the "apartheid" reference was a gaffe or deliberate there is no doubt that Secretary Kerry believes Israel's government is stubbornly ignoring his warnings that a failure to agree a peace deal will feed a campaign trying to delegitimise their state.
Mr Obama has previously rejected the use of the term "apartheid" in discussion of the Middle East peace process, calling it unhelpful and inaccurate.
But in remarks before a group of high-ranking officials from the US, Europe and Japan known as the Trilateral Commission, Mr Kerry used the word.
"A two-state solution will be clearly underscored as the only real alternative," Mr Kerry said, according to the Daily Beast.
"Because a unitary state winds up either being an apartheid state with second-class citizens - or it ends up being a state that destroys the capacity of Israel to be a Jewish state."
Mr Kerry also reportedly suggested a peace deal was more likely to come with a change in Israeli and Palestinian leadership, and strongly criticised Jewish settlement construction on occupied territory.
The American-Israel Public Affairs Committee (Aipac) condemned his comments as "deeply troubling".
"Any suggestion that Israel is, or is at risk of becoming, an apartheid state is offensive and inappropriate," it said in a statement.
In a statement to the BBC, the US state department noted that Israeli leaders had used the term "apartheid state" before, to warn of the consequences of the failure of the peace process.
"Secretary Kerry, like Justice Minister [Tzipi] Livni, and previous Israeli Prime Ministers [Ehud] Olmert and [Ehud] Barak, was reiterating why there's no such thing as a one-state solution if you believe, as he does, in the principle of a Jewish state," spokeswoman Jen Psaki said.
"He was talking about the kind of future Israel wants and the kind of future both Israelis and Palestinians would want to envision. The only way to have two nations and two peoples living side by side in peace and security is through a two-state solution. And without a two-state solution, the level of prosperity and security the Israeli and Palestinian people deserve isn't possible."
Last week, Israel suspended the current round of US-brokered peace talks, which resumed last July after a three-year hiatus, demanding the annulment of a reconciliation agreement between two main Palestinian factions.
Fatah, which is led by Palestinian Authority President Mahmoud Abbas, and Hamas, which governs Gaza, said they planned to form a unity government in the coming weeks.
Hamas rejects Israel's right to exist and is designated a terrorist group by the US, EU, Israel and other countries.
Mr Netanyahu told the BBC that Mr Abbas could "have peace with Israel or a pact with Hamas - he can't have both".
The US expressed "disappointment" over the unity deal, but pushed for the talks to continue.
Last week, Mr Obama said his administration would continue to offer "constructive approaches". But he added that "neither side had political will to make tough decisions" and that there might need to be a pause in the process.
The US Secretary of State, John Kerry, has warned that Israel risks becoming "an apartheid state" if a two-state solution to the conflict with the Palestinians does not emerge soon.