Efficient way to fetch records since a given criteria
To add a bit to what @SQLFox said if ID happens to be the clustered index then it should be blazing fast.  Even if it isn't the clustered index, if it is even included in an index then you are libel to at least get an index scan and go fairly quickly.  

If I had to guess I would say it's your ORDER BY that is slowing you down.  With the ORDER BY in place it's going to have to check every row and check to see if it should be part of that top 100.  Without it SQL is just going to check enough rows until it gets 100 of them that match the criteria.