Should I use an Arduino or not?
Short answer: Yes. After all, Arduinos are made by starting with an ATMega (or other AVR) IC. 

Longer answer: People - including myself - routinely build with a bare ATMega328p (or other MCU), programming them off-board on another system, or providing connections to allow programming them in place. Beyond that you only need the soldering skills for the chip you choose (if you're building the boards yourself). I use DIP ICs and off the shelf through-hole PCBs and hand-build one or a couple at a time. Quad Flat-Package or Surface-Mount chips are another step up in complexity for the hand-builder.