How do I build a USB joystick/button encoder board without using an Arduino or Teensy
If you have it working with an Arduino, the obvious next step is to use the same microcontroller and the same or similar software. 

ATmegas can be programmed using a SPI programmer like STK500 or USBasp. You could load your software directly that way, or just load the Arduino boot loader and use the Arduino loader to load your software.