What are limitations on the number of objects the eye determine at a glance?
The answer to this question is complicated, but assume it's not for the moment. The easy answer is that our eyes can detect patterns we're familiar with and "produce" an answer in the form of a number. Unfamiliar patterns do not produce this 'number'.

Take a die. We are immediately aware of the number of dots in the typical arrangement of one to six. We need expend no counting to do so. If the skittles are arranged in the familiar die pattern, we will have no problem knowing if there are 4 or 6 skittles.

Given a pair of dice, any gambler can tell you what number any two faces adds up to without counting the spots. Do it long enough, and your brain is on auto-recognize. In your example, if Sam is a gambler, and the skittles are arranged like the spots on two dice, again, no problem telling between 2-12 skittles.

If you arrange things in a comfortable pattern, this can go on into the hundreds.



The above was easily recognized by you as nine, I presume.

 

Someone good with pattern recognition will know that the above left is 25 dots. However, they will not know that there are 25 dots in the above right.

So, the easy answer is that the eye can grasp high numbers of things if they are presented in a familiar pattern. The normal mind, however, is not able to do so with higher numbers arranged in a random pattern. And, random is how most things present themselves. So, we need to count.