How can I know how far a button is pressed?
Following on from Mark's comment, I'm not convinced that you can measure the size of the beam, using a single optical (analogue) device - generally optical switches, like other switches, are binary, either on or off, and as such would use one of the Arduino's digital inputs. 

The type of (hypothetical) button that you describe, taking a mechanical analogue (rather than your optical beam example), would require the use of an analogue input pin, and your "switch" would actually be a linear potentiometer, i.e. a slider, or lever, and the analogue input could then measure the input voltage, which would be proportional to the amount that the button (or lever) has been pressed (or moved). However, the usual linear potentiometer is not spring loaded, so the button would not really have the usual expected push button behaviour, unless, of course, you rigged up some sort of spring loaded button, with a long throw, mechanically connected to a linear potentiometer.

An analogue version of this sort of set up, as far as I am aware, does not exist for a optical solution.

What you could use is a series of optical digital (not analogue) switches (or a single light source and a series of receivers), arranged in a line (perpendicular to the base along which the coin is rolled along) in such a way that they can detect the size of the coin*. If the light is broken for a particular detector, then you know that the radius/size/width of the coin is at least the same as that detector's distance from the base.



These optical detectors are then connected to a series of digital inputs, i.e. D5, D6, D7 and D8, and a series of if statements would enable your code to determine the size of the coin.

Assuming that the broken light beam results in a low, then the pseudo code would be something like this:

if D5 = low &amp; D6 = high
 then coin is a cent
if D5 = low &amp; D6 = low &amp; D7 is high
 then coin is a nickle
if D5 = low &amp; D6 = low &amp; D7 = low &amp; D8 = high
 then coin is a dime
if D5 = low &amp; D6 = low &amp; D7 = low &amp; D8 = low
 then coin is a quarter


Obviously you would need a trigger when to take the "snapshot" of the coin passing the detector at the correct time, when it is fully blocking the lights, and not just partially blocking the light beams (which could result in a false reading).



* I have just realised that this is exactly what Jot described in his comment.