 He married a member of the Courtauld family, textile magnates.In 1932 he acquired the 2,500 acre Ashcombe estate in Devon, still owned by his descendants, where in 1935 he built as his residence Ashcombe Tower House, situated on a spur of Little Haldon above the stream known as the Dawlish Water, so named after the tower built there in 1833 as an observatory.Brian O'Rorke was chosen as the architect for the project on the grounds that he had never designed such a house before and would therefore be open to Lady Rayner's ideas.