Turkey accuses the cleric, Fethullah Gulen, of orchestrating last July's failed coup.
In a video interview, Mr Woolsey told the Wall Street Journal he was present at a discussion about removal methods beyond the legal extradition process.
Mr Flynn disputes Mr Woolsey's account.
The meeting took place last September at a New York Hotel.
Those present included Mr Flynn, then an adviser on national security to the Trump election campaign, the son-in-law of Turkish President Recep Tayyip Erdogan, and Turkish Foreign Minister Mevlut Cavusoglu, according to the WSJ (paywall).
Mr Woolsey, then also advising the Trump campaign, admitted he might not have heard "some kind of caveat" because he arrived late for the meeting.
But he told the WSJ: "There was serious discussion of finding some way to move Mr Gulen out of the US to Turkey.
"You might call it brainstorming. But it was brainstorming about a very serious matter that would pretty clearly be a violation of law."
He went on: "It was a serious and troubling discussion but it did not, repeat not, in my portion of being in the room, rise to a level of being a specific plan to undertake a felonious act."
In a later interview with CNN, Mr Woolsey called the meeting "suspicious" and "concerning", saying: "I felt I needed to say something to somebody, but was it a clear plot that they were going to seize him? No."
A spokesman for Mr Flynn, whose consulting company Flynn Intel Group carried out work for the Turkish government, disputed Mr Woolsey's version of events.
"At no time did Gen Flynn discuss any illegal actions, non-judicial physical removal or any other such activities," he said.
The presence of Mr Gulen in the US has become a major irritant in relations between Washington and Ankara. Turkey has repeatedly called for him to be handed over.
Mr Gulen, who has lived in Pennsylvania since 1999, denies involvement in the failed coup.
Fethullah Gulen: Powerful but reclusive Turkish cleric
Michael Flynn: Former US national security adviser
Mr Flynn was forced to step down as national security adviser in mid-February after just a few weeks in post over allegations he discussed sanctions with Russia before Mr Trump took office.
Private US citizens are barred from conducting diplomacy. Mr Flynn admitted providing the vice-president with an incomplete account of his conversation with the Russian ambassador.
President Trump's former national security adviser, Michael Flynn, discussed removing from the US a Muslim cleric wanted by Turkey, according to ex-CIA director James Woolsey.