Baroness Tanni Grey-Thompson says the London 2012 games did change views of disabled athletes - but that has not been reflected in the wider community.
She told BBC Radio Wales' Sunday Supplement that some disabled people are still seen as benefit scrounges.
"We're still kind of lost behind other minority groups," she said.
The comments from the former athlete and House of Lords crossbencher come on National Paralympic Day.
She said the success of the games three years ago meant that Paralympians are now accepted as elite athletes, something she regarded as a hugely important step.
But she told Sunday Supplement's Vaughan Roderick: "The reality is that disability hate crime figures are the worse they've been in 10 years of reporting.
"Disabled people are portrayed in three different ways: you're either a Paralympian, you're a benefit scrounger, or you're a victim and you can't really crossover.
"The vast majority of disabled people aren't benefit scroungers. There's a huge number that don't claim anything at all.
"I think we're in a very difficult place, but that's partly because now you see disabled people integrated into society which you didn't see 20 years ago."
The peer said she had hoped the Equality Act that came into force three years ago would help signal a major shift in the way people approached disability issues.
"We're still a long way from equality and the disability rights movement has struggled so when the Equality Act came in, I was really hopeful that actually for disabled people, it would mean a really big step forward.
"But it hasn't been and we're still kind of lost behind other minority groups."
She said there have been improvements, but added: "I'd like to see more equality for disabled people and that then filters into sport and physical activity.
"You know disabled people need to be fit, healthy, have the same opportunities as everyone else and have the same opportunities to go into elite support.
"The Paralympics can help drive that change. It can't do it all, but it can help change attitudes."
More still needs to be done to shift perceptions towards people with disabilities, says Wales' most successful Paralympian.