Richardson hardly featured under Blackwell in the Leeds team at all due to injuries. Under new manager Dennis Wise, Richardson played more regularly in the Leeds starting line-up, replacing the injured Gary Kelly at right back. On the retirement of Kelly, Richardson was given the number two shirt and became the club's first choice right back and started all the club's matches in the 2007–08 season until injury ruled him out on 2 February. Under former Leeds manager Gary McAllister, Richardson also played in the left-back position, playing a few games there as a replacement for on-loan Leicester City defender Alan Sheehan. After a poor performance in this position against Cheltenham Town, Richardson was booed off the pitch by Leeds fans and did not play for a few weeks after. He returned in his usual right-back position as a replacement for the injured Darren Kenton in their 1–0 win over Doncaster Rovers. Richardson was appointed Leeds captain in August 2008 by McAllister, and remained club captain despite injury under new manager Simon Grayson. Richardson made his final appearance for Leeds as a late substitute against Northampton Town, this was to be his first game for Leeds for several weeks, and also it proved to be his last appearance for the Elland Road side. Leeds United confirmed on 16 June 2009 that they would not continue contract negotiations with him due to the signing of former Northampton captain Jason Crowe. Richardson was released due to his contract ending on 30 June 2009. In July 2010, Lloyd Sam revealed that Richardson had never wanted to leave Leeds. Charlton Athletic On 9 July, Richardson signed for Charlton Athletic on a free transfer on a two-year deal. He scored his first goal for the club in a 2–2 draw with Gillingham on 20 March 2010. In May 2010, he was selected in the Football League One PFA Team of the Year for the 2009–10 season. Richardson was part of the Charlton side which reached the League 1 playoffs, they were knocked out of the playoffs after losing on penalties to Swindon Town, Richardson scored his penalty, but Charlton lost and Richardson suffered his fourth playoff campaign defeat in his career. Southampton On 6 July 2010, Richardson signed for League One club Southampton for an undisclosed fee, in a deal reportedly worth £450,000. After the club's second pre-season game, against FC Sochaux-Montbéliard on 17 July