Sql Server 2008 : What is the cost of accessing an index?
That is because SQL Server reads entire pages instead of records and when a index is used SQL Server might need to follow the bookmark to fetch the fields that are not included in the index.

So if your query isn't selective enough, it might end up reading all the pages of the index, and following the bookmarks to read all the pages of the actual table too. This ends up being more expensive than just reading all of the pages of the table.

This effect can be somewhat mitigated using included columns in which some columns are included in the index, but aren't part of the index (i.e. they have no effect on the sorting of the index).

SQL Server estimates the selectivity of your query based on statistics.

This article explains a lot of it. And the stairways series really is a good starting point to understand how indexes work.