Question about viability of building a Bill Counter machine
There are many more things to consider than just put some parts together if you try to build a counter from scratch. Automated currency handling has come a long way to today's bill counters or any vending machines. You may perfectly handle the computing part of task with an embedded system like the arduino, but paper handling is an art of itself. 

If you think of this task as a challenge to acquire and hone engineering skills, go ahead. It may be one of the best training tasks one can think of. But while you may learn a lot from this conundrum, you will propably end up without an usabale device. :)

You have to consider, there are subtasks in


mechanics
electronics
programming


which may include


paper handling (separation, traction, alignment)
pattern recognition
motor control
high speed sensors
signal detection and processing
error and fault processing and handling
motor and gear connection
bearings and mechanical substructure
user interface


and many more.

To modify an existing device solves a lot of mechanical problems in advance. But it may still be difficult to change the firmware of a bill counter to your needs. 

You will have to


break into the system (may be impossible)
reverse engineer the firmware
understand the mechanical constraints
find a way to adapt the system to your needs.


A third way, which in my eyes is rather intriguing, is to take an existing bill counter, rip out the controller and try to plug in your own controller, e.g. an arduino. This gives you a real chance to reach your goal while still having the opportunity to learn a lot.