at the gate of the palace goring , laud and cavendish left for their permanent visiting room in st. james , while hudson the dwarf left to sulk in the garden , as he was not allowed in the dining room .