 While there, he unsuccessfully attempted mediate between the Chōshū samurai and the forces of the Tokugawa shogunate at the time of the Kinmon incident.This opened Kaga Domain to a charge of collusion with enemies of the state, and an alarmed Nariyasu had Yohsiyasu sent back to Kanazawa under house arrest, which lasted until April 1865.On April 4, 1866, on the retirement of his father, he became "daimyō" of Kaga Domain; however, he was only a figurehead as his father continued to control all power.