His mother was Ann Phoebe Penn Dagworthy ( Charlton ) and his father was Captain John Ross Key . Francis Scott Key was born to Ann Phoebe Penn Dagworthy ( Charlton ) and Captain John Ross Key at the family plantation Terra Rubra in what was Frederick County and is now Carroll County , Maryland .