In Baltimore , Poe lived with his aunt , Maria Poe Clemm . Before entering West Point , Poe moved back to Baltimore for a time , to stay with his widowed aunt Maria Clemm , her daughter , Virginia Eliza Clemm ( Poe 's first cousin ) , his brother Henry , and his invalid grandmother Elizabeth Cairnes Poe .