Fort Victoria is situated on Sconce Point west of Yarmouth. Its position overlooks the whole of the Needles Passage and approaches to Yarmouth, and is almost opposite Hurst Castle on the mainland. Around 1 kilometer to the southwest lies Fort Albert.