What is the computer science interpretation of a qubit?
A qubit is a vector in a 2-state complex space. Its vector form is something like Qubit = $ax + by$, which by convention in Dirac notation is $|Q\rangle = a|0\rangle + b|1\rangle$, where $|0\rangle$ and $|1\rangle$ represent linearly independent basis vectors and the coefficients $a,b$ are complex numbers. $a^2$ and $b^2$ represent the probability of measuring 0 or 1, and they can take on any complex value as long as $|a|^2 + |b|^2 = 1$ (which must be true because we only have one qubit).
In this framework, the qubit acts like a vector pointing from the origin which  naturally evolves or rotates around the infinitely many surface points of a ‘Bloch Sphere’ except when it’s measured it only ever tells you 0 or 1 (which represent the north and south poles of the sphere).
In short, a qubit is a vector.
The ‘infinitely many’ language comes from the ability of $a,b$ to take on any complex value so long as their squares equal 1. Measuring a qubit (meaning when you read it, touch it, look at it, record its value etc) only gives you 0 or 1, not both and not a decimal in between.
If you also think of the qubit as a wave, then the goal of quantum computing is to take a superposition of all possible results of a qubit string (000, 001, 010, 011 etc), weight the components appropriately to constructively and destructively interfere with one another, so that at the very end you’re biased to get the right result... with a nonzero probability that you will get the wrong answer even if you do everything right.
Incidentally a string of three qubits, you’ll notice, is the tensor product of three qubit vectors. This creates a complex space with $2^3 = 8$ basis vectors which our 3-string qubit can happily explore.