expressed his disappointment about the transfer saying that Duffy wanted to "develop" his career in England. Though BBC wrote that the transfer fee was £250,000, The Guardian'' said that he joined the club for an undisclosed fee. However, with limited opportunities at Hull, particularly after the departure of manager Peter Taylor, he joined Hartlepool United on loan in November 2006, scoring 5 goals in 10 games for the then League Two side. An even more productive loan spell followed in February 2007, when Duffy joined League One side Swansea City on loan, scoring five goals in as many starts, as the Swans narrowly missed out on the playoffs. In July 2007 Duffy joined Swansea permanently, signing a three-year contract for an initial fee of £200,000. Bristol Rovers signed Duffy from Swansea on 3 July 2008 for a transfer fee in the region of £100,000. Duffy scored twice in a 6–1 over Hereford United and in a 2–2 draw at Elland Road against Leeds United. At the end of the 2008–09 season, Duffy had scored 13 goals for Rovers. After this, however, Duffy's first team opportunities were restricted. He had a loan spell at Carlisle United where he scored once against MK Dons. On 30 August 2010, Duffy signed a season-long loan deal with SPL side Hibernian to team up with his former Falkirk manager John Hughes. Duffy was ostensibly signed by Hibs as a replacement for Anthony Stokes, but suffered a foot injury soon after signing for the club. The injury meant that Duffy was unable to play for Hibs before Hughes left the club by mutual consent. Duffy featured infrequently after he returned from injury and Hibs announced in April 2011 that he would return to Bristol after the loan deal was completed. Duffy was one of seventeen players released by Rovers in May 2011. Cheltenham Town announced in July that they had agreed to sign Duffy. The striker hit his first goals for the club in a 3–2 win at Northampton Town. He finished the 2011–12 season as top goalscorer with 15 goals. Cheltenham reached the League Two play-off final, but lost to Crewe. Duffy then fell out of favour at Cheltenham, making just five starting appearances in the 2012–13 season. He was offered a new contract by Cheltenham in May 2013, but he decided to leave the club due to his frustration at not playing regularly. India