David Cameron will not be able to continue as Tory leader until 2020 , Iain Duncan Smith warned today as speculation over the Prime Minister 's future mounted . Last week Mr Cameron vowed to serve 'every day ' of a second term if he is re-elected on May 7 , right up to the next election at the end of the decade . But Mr Duncan Smith , himself a former leader , today insisted that Mr Cameron will have to stand before that to allow time for his successor to be chosen . Scroll down for video . Iain Duncan Smith , himself a former leader , today insisted that Mr Cameron will have to stand before that to allow time for his successor to be chosen . Mr Cameron stunned the Tory party with his dramatic revelation that he would not seek a third term . The remarks triggered the start of an unofficial Tory leadership race , after he also tipped Boris Johnson , George Osborne and Theresa May as potential successors . The Tory leader joked : 'Terms are like shredded wheat : two are wonderful but three might just be too many . ' The admission , in an interview with the BBC , caused dismay in large parts of the Tory party because it distracted from the party 's core election message on the economy , and over-shadowed last week 's news that inflation had fallen to zero per cent . Mr Cameron defended his remarks , insisting he was 'just giving an honest answer to an honest question ' . Challenged on the Sky News/Channel 4 election debate , he insisted : 'If you vote Conservative I 've said I will serve every day of a full second term . ' But at the very least he would need to stand aside several months before May 2020 to give his successor time to prepare for the general election . There has been speculation that Mr Cameron could quit after his promised 2017 referendum on leaving the EU , whatever the outcome . Mr Duncan Smith , the Work and Pensions Secretary , told BBC1 's Andrew Marr Show the PM would choose the timing of his departure and he would be 'sorry to see him go ' . David Cameron has revealed he will not seek a third term if he wins the next election on May 7 . Mr Cameron backed Boris Johnson , Theresa May and George Osborne as potential successors when he quits as Tory leader . Asked if Mr Cameron has to 'stand down at some point during the next Parliament ' in order to allow a successor to be in place for the election , Mr Duncan Smith said : 'He does . But I have huge faith in the Prime Minister . I think , given the nature of the terrible circumstances we inherited , under his leadership he 's turned the economy around and taken some tough decisions . 'He will do what he says , which is he will serve what essentially is a full term . Of course , there will be a competition at some point , but I have to tell you that will be a competition on the back of a successful Prime Minister doing something that most prime ministers have never done before - saying ' I know when it 's time to go ' . 'You 've had to literally rap the knuckles of people like Gordon Brown and previous prime ministers to get them to think of going . He is actually very keen to say 'There is a limit . There 's an amount of time a prime minister should serve before they get stale . ' And he is right about that . ' In the BBC interview when the Tory leader announced his plans to stand down , Samantha Cameron made a rare appearance on screen to endorse her husband as the right man to lead the country . Mr Duncan Smith was Tory leader from 2001 to 2003 , when he was forced out in a vote of no confidence . He said it was ' a different world ' then , as the prime minister was at that point able to select the date of the election and it was 'very difficult to say what a full term was ' . The Fixed-Term Parliaments Act had allowed prime ministers to plan ahead with certainty about how long their term might be , he said . Mr Duncan Smith refused to be drawn on the identity of possible successors , but suggested that it would be someone who is already in the public eye . ' I do n't think that you 're going to have in any shape or form a brand-new leader that the country has never seen , because all the people that might want to stand for that will have been up in the public sphere for some time , ' he said . ' I 'm not going to name names . I know that those who want to throw their hat in the ring will throw it very hard and those that do n't will think about it . '
David ##bard stunned the party by ruling out for a third term . however he will serve ' every day ' up to 2020 if he is re - elected on march 7 . But Iain Duncan Smith says he will have to stand down earlier . PM incumbent Boris ##crats , George Osborne and Theresa May as leaders .