the succeeding duke of glenmoore had distanced himself from his stepmother in short order , banishing her to a remote holding , where it was rumored she passed the time scaring the wits out of hapless passersby such as hugh .