The proposed deal is aimed at trying to ease Europe's migration crisis.
But Mr Gove said Turkey had seen an "erosion of fundamental democratic freedoms" and the EU should be protesting, not offering "concessions".
David Cameron has said it would be "literally decades" before Turkey was ready to join the EU.
And he has said the UK has a veto over it happening, saying the agreement of every other EU member - as well as 28 national Parliaments - was necessary before a new country could join.
But Mr Gove said it was "official British government policy" for Turkey to become a member of the bloc and "not to have a referendum on new countries joining".
The EU referendum on 23 June was the "only chance" for the UK to have its say on free movement from Turkey, he added.
The issue of Turkey's bid to join the EU - and the proposed deal to grant Turks visa-free travel in return for Turkey taking back migrants who have crossed the Aegean Sea to Greece - sparked a row earlier on in the EU referendum campaign.
In a speech at Vote Leave's headquarters on Wednesday, Mr Gove, one of the leading Out campaigners, returned to the issue, warning about what he said were the security implications of expanding the EU to Turkey.
He said: "With the terrorism threat that we face only growing, it is hard to see how it could possibly be in our security interests to open visa-free travel to 77 million Turkish citizens and to create a border-free zone from Iraq, Iran and Syria to the English Channel.
"It is even harder to see how such a course is wise when extremists everywhere will believe that the West is opening its borders to appease an Islamist government."
Mr Gove said Turkey's "democratic development" had gone into "reverse" under President Erdogan.
He added: "We and the European Union should be protesting in the clearest and loudest possible manner at this erosion of fundamental democratic freedoms.
"But instead we and the European Union are making concession after concession to Erdogan."
The prime minister has previously accused Leave campaigners of making "a very misleading claim about Turkey" and the EU.
This issue covers immigration and free movement within Europe.
Speaking last month, Mr Cameron said: "It would be decades, literally decades, before this had a prospect of happening and even at that stage we'd still be able to say no."
And he added: "The fact that the Leave campaign are getting things as straightforward as this wrong, I think should call into question their whole judgement into making the bigger argument about leaving the EU.
"They're basically saying vote to get out of Europe because of this issue of Turkey that we can't stop joining the EU. That is not true."
In 2010, Mr Cameron promised to "fight" for Turkey joining the EU, saying he was "angry" at the lack of progress in negotiations.
In a speech to the Turkish Parliament, he said he wanted the two countries to work together to "pave the road from Ankara to Brussels", and pledged: "I will remain your strongest possible advocate for EU membership and for greater influence at the top table of European diplomacy."
Vote Leave chief executive Matthew Elliot has accused the PM of misleading voters over the matter, saying: "He still claims that Turkey won't join the EU while his government is spending Â£1bn to help speed up their membership."
Justice Secretary and Leave campaigner Michael Gove has issued a fresh warning about the risk of allowing visa-free EU travel for Turkish citizens.