When does my new family member occupy a new room?
According to the rulebook (pg 9):


  After Family Growth, also 1 Minor Improvement (Stage 2): A player may only use this Family growth space if he has more empty
  rooms than Family members. It is irrelevant how the family has grown
  before and whether the Family members are on the game board or in the
  farmyard.


Since you had four family members and four rooms, you couldn't have done your second Family Growth action, even though the new family member had not returned home yet to "claim the room".