What kind of Arduino should I pick for a calculator?
Start with the biggest (AVR) Arduino you can afford, and worry about miniaturizing it once you have the basics in place. You may find that you don't actually want an Arduino in the end, but something like the ATmega169PA instead so that you have native LCD support (instead of having to use a discrete LCD module).


  P.S. I have some experience with x86 assembly; does that mean that 86duino or Intel Galileo are better for me?


Doesn't matter. Most of your x86 assembly experience will easily transfer to AVR (but not all, since the AVR core is simpler than x86).