From 1996 to 1998 , Gillard was Chief of Staff to John Brumby , at that time the Victorian opposition leader . From 1996 to 1998 , Gillard served as Chief of Staff to John Brumby , at that time the Victorian opposition leader .