One of the arguments against appointing an Australian, particularly a former politician, had always been that they would be too closely involved with Australian personalities and issues to perform their constitutional role impartially.
Casey left office in 1969 and he and his wife retired to their farm at Berwick in Victoria.