His sister was married to Mohammad Sidique Khan - the oldest of the four suicide bombers who attacked London on 7 July 2005, killing 52 people.
Now he has decided to speak publicly for the first time, because he wants to help in efforts to prevent radicalisation.
Mr Patel says the actions of his brother-in-law, who killed himself and six others when he detonated a bomb on a Circle Line underground train near Edgware Road, were "disgusting".
Mr Patel struggles to find adjectives about him that are not expletives, adding: "He's a coward. How is his daughter supposed to live the rest of her life now? Terrorism hurts everybody."
He says that at first he did not register the attacks and only properly understood what had happened on 12 July.
"My mother, brother, sister and niece, came to my house. Scotland Yard and MI5 with them. My brother said to me 'that thing in London, they're telling us it was Sidique,'" he recalls.
"I actually asked him, 'What thing has happened in London?' I had to go research it.
"I nearly fainted. I can remember my mother the next two or three days fainting multiple times."
Profile: Mohammad Sidique Khan
7/7 attacks: What happened that day?
7/7 attacks: The victims
His first reaction was denial.
"You cannot believe this person that you knew is capable of that, simply because there were no signs, there were no indicators.
"I'm thinking 'maybe he's just one of the victims', because it was an ongoing investigation."
That period of denial came to an end when Mohammad Sidique Khan's "martyrdom video" was released.
In it he claimed to be acting in response to western involvement in Muslim countries.
Mr Patel is categorical that there were no signs he could have spotted in Khan to predict what he would go on to do.
He says Khan, who worked as a learning mentor for children before the attacks, would discuss events such as the Iraq War as part of normal conversation but never showed signs of being capable of violence.
But he does say that Khan looked down on the Muslim community in Dewsbury, where they were all living, and would refuse to visit local mosques because he disagreed with their non-political stance.
"His attitude was 'my prayers won't be accepted in these mosques'.
"The term he would use is that, 'You guys are hippy Muslims - you're like the Quakers, like the Salvation Army. All happy clappy Muslims, that's what you Sufis are.'"
Mr Patel has met two 7/7 survivors and has often considered paying his respects to victims at one of the annual commemorations but worries it would offend grieving relatives.
He says his family have also have been affected, with his children being bullied by other Muslim children.
"We were not the perpetrators, but we were ostracised from the community - that is proof that the community views terrorism with disgust - and they view the families of terrorists with disgust."
But Patel is clear that he is not trying to detract from the suffering of those who lost loved ones in the attacks.
"Their (suffering) is much worse. If people don't sympathise with me, ok, I don't blame them. I blame him [Mohammad Sidique Khan]."
Mr Patel believes there can at times be an unhelpful focus on foreign policy when discussing radicalisation.
He says Khan's claims that his actions were a response to foreign policy were a "pathetic excuse". "We all have grievances," he says.
He is similarly forthright in rejecting the idea that Muslim communities are mistreated in Britain and that is pushing them towards radicalisation.
"After 7/7 there's been very little blowback. There have been isolated incidents. It makes you put your faith back into humanity.
"Whenever somebody like the EDL or Britain First come around, the first people to actually oppose them, along with us, will be their fellow countrymen."
For Mr Patel - radicalisation is an issue for all of British society - not just the Muslim community.
Nevertheless, he believes mosques could do more.
He says there need to be more English-speaking imams and more co-ordination with youth workers who have "life experience".
"ISIS (so-called Islamic State) have to be discussed.
"Otherwise, because we're not addressing these issues in the mosque, they will go online and they will end up in a bad place. We're not to blame, but we're part of the problem because we're not doing what we can do ourselves."
Mr Patel says he is broadly supportive of government counter-radicalisation programme, Prevent.
He also hopes to be able to forge links with others working in de-radicalisation to be able to help prevent another atrocity happening here.
Asked if he is worried there could be another attack on British streets, he replies: "Yes, that's why I feel I have to speak out."
For more than a decade Ahmed Patel has lived with the legacy of the 7/7 attacks.