The difference between a bit and a Qubit
Probably the easiest analogy is to probabilities. If your computer can flip fair coins, you can think of each coin being in state Tails with probability 1/2 and in state Heads with probability half. So it's appropriate to think of a coin not as a bit but as a vector of two elements $(p_T, p_H)$, where $p_T$ is the probability of tails, $p_H$ is the probability of heads and we have that $p_T \geq 0$, $p_H \geq 0$, and $p_T + p_H = 1$. Once we sample a coin its state "collapses" to either tails or heads.

Just about the same happens with qubits. You can also think of quibit as being "in between" two states. It's also easiest to think of qubit not as a bit but as a vector of two elements $(q_0, q_1)$. Now, however, we allo $q_0$ and $q_1$ to be negative, and even further, to be any two complex numbers, as long as $|q_0|^2 + |q_1|^2 = 1$. Just as with sampling coins, once you measure a qubit, it collapses to either $0$ or $1$, and in fact you get $0$ with probability $|q_0|^2$ and $1$ with probability $|q_1|^2$. However, and this is the crucial part, these quantum probabilities can be made to cancel each other because they take negative values; this cannot happen with classical probability. This is the power of qubits.

BTW these complex valued quantities describe amplitudes and are related to the wave-particle duality, i.e. the discovery that particles in fact behave as waves, until you measure them, and then they behave as particles. The cancelation I am talking about is in fact interference between waves. So another answer to your question is that a qubit is not a discrete object that's either in one state or not, it's a wave..until  you measure it.