[96] The farm was not successful and to earn an alternate living he sold firewood on St. Louis street corners.[97]
In 1856, the Grants moved to land on Julia's father's farm, and built a home called "Hardscrabble" on Grant's Farm. Julia described the rustic house as an "unattractive cabin", but made the dwelling as homelike as possible with the family's keepsakes and other belongings.[98] Grant's family had little money, clothes, and furniture, but always had enough food.[99] During the Panic of 1857, which devastated Grant as it did many farmers, Grant had to pawn his gold watch in order to buy Christmas gifts for his family.[100] In 1858, Grant rented out Hardscrabble and moved his family to Julia's father's 850-acre plantation.[101] That fall, after suffering from malaria, Grant finally gave up farming.[102]
The same year, Grant acquired a slave from his father-in-law, a thirty-five-year-old man named William Jones.[103] Although Grant was not an abolitionist, he was not considered a "slavery man", and could not bring himself to force a slave to do work.