Understanding the state vector in Quantum Computing I have just started to learn QC. It is said that 


  The quantum state of $N$ qubits can be expressed as a vector in a space
  of dimension $2^N$


If there is $1$ qubit then we have two possible state vectors $|0\rangle$ and $|1\rangle$ or $(0,1)$ and $(1,0)$ respectively. Getting to $2$ qubits we have $4$ possible state vectors $(1,0,0,0), (0,1,0,0), (0,0,1,0), $ and $(0,0,0,1)$. 
Note that in each case, all entries are zero except 1. The point I am trying to get to is that:


$2^N$ seems like a big space but given a vector in this space - all
    components will be zero except $1$. So there are only $2^N$ possible
    values the state vector can take.  Is this not correct? If not, why?
Why don't we say the space is $N$-dimensional. A $N$-bit string has $2^N$
possible values.
No, it's not correct.  It's not true in general that one entry will be 1 and all others 0; that is true for the basis vectors, but there are other states (other vectors) where that isn't true.  You can have linear combinations of the basis states, e.g., the following is a possible state:

$${1 \over \sqrt{2}} |0\rangle + {1 \over \sqrt{2}} |1\rangle.$$

It corresponds to the following vector in the vector space:

$${1 \over \sqrt{2}} (0,1) + {1 \over \sqrt{2}} (1,0) = (1/\sqrt{2},1/\sqrt{2}).$$

Because of this, you can have infinitely many possible states in this vector space.

The vector space is not $N$-dimensional.  The dimension of a vector space has a formal definition, and if you apply it, you will discover that the dimension of the vector space is $2^N$.