The low-budget Ouija, starring Bates Motel actress Olivia Cooke, made $20m (Â£12.4m) between Friday and Sunday, according to studio estimates.
That put it ahead of John Wick, Keanu Reeves' latest thriller, which made $14.2m (Â£8.8m) over the same period.
Last week's top film, Brad Pitt's World War II tank drama Fury, fell to three.
Made for just $5m (Â£3.1m), Ouija's success is testament to US moviegoers' appetite for modestly-budgeted supernatural fare without big-name stars.
Blumhouse Productions, the company behind the film, has enjoyed similar success with the Insidious, Purge and Paranormal Activity series.
Meanwhile, John Wick, in which Reeves plays a retired assassin out for revenge, exceeded industry expectations with its second place opening.
This week's top five was rounded out by David Fincher's Gone Girl and animated title The Book of Life.
Next week's chart is likely to be dominated by Interstellar, the latest sci-fi epic from British director Christopher Nolan.
Matthew McConaughey and Anne Hathaway star in this tale of deep space exploration, which has its UK premiere in London on Wednesday.
A horror film about young people who use a Ouija board to contact the spirit world has topped the North American box office in the run-up to Halloween.