Based on multiple books, this was at the southwest corner of 17th and F streets NW.  Photo of it <a rel="nofollow" class="external text" href="https://www.loc.gov/pictures/item/npc2007000059/">here</a>, in the 1920s, after a major 1885 alteration. In 1865, this was two buildings, and I think <a href="https://en.wikipedia.org/wiki/Ulysses_S._Grant" class="extiw" title="w:Ulysses S. Grant">Grant's</a> headquarters were the right one (on the corner) in this photograph. It is in <i>Capital Losses</i> as "Towson House"; it was built in 1827 by <a href="https://en.wikipedia.org/wiki/Nathaniel_Towson" class="extiw" title="w:Nathaniel Towson">Nathaniel Towson</a>, paymaster general of the US Army.  He died in 1854, and the government leased the building from then on.  It was <a href="//commons.wikimedia.org/wiki/Winfield_Scott" title="Winfield Scott">Winfield Scott</a>'s headquarters at the outset of the Civil War (and was the house where Scott asked <a href="//commons.wikimedia.org/wiki/Robert_E._Lee" title="Robert E. Lee">Robert E. Lee</a> for allegiance to the Union; Lee resigned the following day). Portraits of Scott <a rel="nofollow" class="external text" href="https://www.loc.gov/pictures/item/cwp2003001690/PP/">here</a> and <a rel="nofollow" class="external text" href="https://www.loc.gov/pictures/item/cwp2003001689/PP/">here</a> were taken on the rear porch of the house. Scott left in October 1861. Immediately after <a href="https://en.wikipedia.org/wiki/Battle_of_Appomattox" class="extiw" title="w:Battle of Appomattox">Appomattox</a>, Grant returned to Washington and made his headquarters there. Another book has a "Washington City, 1869" engraving which describes it as being his headquarters at the time, still. In 1885, the government needed more space, and built a new wing joining the two buildings in this photo, and added a full third floor to the Towson House on the right. It was occupied by various government offices, then sold and torn down in 1947. The site was afterwards an office for the United Steelworkers of America, and today is the site of the FDIC headquarters.  The taller building in the background on the right is the Winder Building, on the northwest corner of F and 17, which still exists. - (lightly edited from remarks by <a href="//commons.wikimedia.org/wiki/User:Clindberg" class="mw-redirect" title="User:Clindberg">Carl Lindberg</a>, originally made at <a href="//commons.wikimedia.org/wiki/User_talk:Jmabel/Stereo_cards_of_Washington,_D.C." title="User talk:Jmabel/Stereo cards of Washington, D.C.">User talk:Jmabel/Stereo cards of Washington, D.C.</a>)
<div class="fn value">
E. &amp; H.T. Anthony (Firm) -- Publisher</div>
<p>Created: 1860-1880.
</p><p><br>
Coverage: 1860-1880.
Source Imprint: 1860?-1880?.
</p>
Digital item published 1-25-2006; updated 2-13-2009.
This image is available from the <a href="//commons.wikimedia.org/wiki/New_York_Public_Library" title="New York Public Library">New York Public Library</a>'s Digital Library under the digital ID G90F083_061F: <a rel="nofollow" class="external text" href="http://digitalgallery.nypl.org/nypldigital/id?G90F083_061F">digitalgallery.nypl.org</a> → <a rel="nofollow" class="external text" href="https://digitalcollections.nypl.org/search/index?keywords=G90F083_061F">digitalcollections.nypl.org</a>