Jumping to conclusions (JTC) distinguishes patients with schizophrenia from both healthy and psychiatric controls. JTC is typically assessed using the beads task, which, however, faces a number of limitations as to its interpretability and reliability. The present study set out to validate a new paradigm to assess JTC: the box task. We adopted a psychometric psychosis proneness approach and divided a large population sample into participants who scored high versus low on a scale tapping psychosis-like experiences. Participants performed a variant of the original beads task along with a new JTC task, the box task, with or without time pressure. The box task requires participants to infer which of two ball colors will be more prevalent in a matrix of boxes. The box task and the beads task were significantly correlated at a medium effect size, thus demonstrating criterion validity for the box task. As hypothesized, participants who scored high on psychosis-like experiences showed particularly strong JTC and a decreased decision threshold relative to low scorers, especially in the box task version with time pressure; in contrast, group differences in the beads tasks only achieved trend-wise significance. Mediation analyses showed that fewer draws to decisions were predicted by either a lower decision threshold or by higher initial probability estimates for the dominant item. The study establishes the criterion and construct validity of a new JTC task. Its advantages over the traditional beads task are better comprehensibility and usability; multiple parallel versions can be created thus raising reliability.