Smith's first significant adventure was in 1964 when he sailed with a group of Rover Scouts to Balls Pyramid in the Pacific Ocean - the highest sea spire in the world. Failing to top it on this occasion, Smith returned in 1980, completed an ascent and, together with Hugh Ward and John Worrall, formally claimed the land for Australia by unfurling the New South Wales state flag.: He went on, in 1988, to sue the state government over its ban on recreational climbing of the spire because the restrictions really attack people's right to have an adventure. His action was taken when the government refused an application by Greg Mortimer, the first Australian to climb Mt Everest, to make the climb.