( CNN ) -- The inside accounts of the Obama administration 's internal debate over Afghanistan may be grabbing the headlines , but former Defense Secretary Robert Gates ' new memoir has some other eye-openers as well . Yes , Gates levels blistering criticism at Congress , Vice President Joe Biden and former colleagues and describes White House aides working to undercut President Barack Obama 's confidence in his own commanders . But in `` Duty : Memoirs of a Secretary at War , '' the low-key Gates -- who replaced Donald Rumsfeld at the Pentagon late in George W. Bush 's administration and was kept on by Obama when he took office in 2009 -- also talks about his own moments of doubt , anger and frustration . Here are a few items from the book that may surprise readers . -- Take this job and shove it : Gates fantasized about quitting . `` All too frequently , sitting at that witness table , the exit lines were on the tip of my tongue : ' I may be the secretary of defense , but I am also an American citizen , and there is no son of a bitch in the world who can talk to me like that . I quit . Find somebody else . ' It was , I am confident , a widely shared fantasy throughout the executive branch . '' -- Bush and Obama 'had much more in common than I expected . ' That was n't always good . `` Both were most comfortable around a coterie of close aides and friends ( like most presidents ) and largely shunned the Washington social scene . Both , I believe , detested Congress and resented having to deal with it , including members of their own party . And so , unfortunately , neither devoted much effort to wooing or even reaching out to individual members or trying to establish a network of allies , supporters -- or friends , '' Gates writes . `` They both had the worst of both worlds on the Hill : they were neither particularly liked nor feared . Accordingly , neither had many allies in Congress who were willing to go beyond party loyalty , self-interest , or policy agreement in supporting them . '' However , Gates adds , `` I liked and respected both men . '' `` I witnessed both of those presidents make decisions they believed to be in the best interest of the country regardless of the domestic political consequences , both thereby earning my highest possible respect and praise . Although , as I 've said , political considerations were far more a part of national security debates under Obama , time and again I saw him make a decision that was opposed by his political advisers or that would be unpopular with his fellow Democrats and supportive interest groups . '' -- The price of public service : . Gates writes that he spent $ 40,000 on lawyers to help him fill out financial disclosures and other government paperwork . Before his 2006 confirmation hearing , he filled out a 65-page questionnaire from the Senate Armed Services Committee . At one point , Bush 's chief of staff , Josh Bolten , `` asked if I had any ethical issues that could be a problem , like hiring illegal immigrants as nannies or housekeepers . `` I decided to have some fun at his expense and told him we had a non-citizen housekeeper . Before he began to hyperventilate , I told him she had a green card and was well along the path to citizenship . I do n't think he appreciated my sense of humor , '' Gates wrote . He told his wife , `` I have to do this , but I just hope I can get out of this administration with my reputation intact . '' -- ' I could n't sleep that night ' Gates writes that an encounter with a woman in a restaurant who had two sons fighting in Iraq drove home the human dimension of the job . `` I could n't finish my dinner , and I could n't sleep that night , '' he wrote . `` Our wars had just become very real to me , along with the responsibility I was taking on for all those in the fight . '' -- Time to go : . By early 2011 , `` My fuse was really getting short . It seemed like I was blowing up -- in my own , quiet way -- nearly every day , and no longer just in the privacy of my office with my staff ... I had blown up at ( National Security Adviser Thomas ) Donilon and the vice president at a meeting on Libya on March 2 and at House Defense Appropriations chair Bill Young on the third , and had come close to openly arguing with the president in the NSC meeting that same day , and had gone off on Donilon again on the fifth . Partly , I think , I was just exhausted from the daily fights . '' In addition , he wrote that his desire to protect U.S. troops -- `` from the wars we were in and from new wars -- was clouding my judgment and diminishing my usefulness to the president , and this is played a part in my decision to retire . '' He left that June , with Obama awarding him the Presidential Medal of Freedom -- the highest U.S. civilian honor -- on his way out .
Bob Gates ' memoir includes quitting fantasies , fears for troops . Bush and Obama `` had much more in common than I expected , '' ex-Pentagon chief writes . Gates served as defense secretary from 2006 to 2011 . In the end , `` I was just exhausted from the daily fights , '' he writes .