USB: what are the advantages (or disadvantages) or using HID over serial-over-USB?
To determine what solution will be best for throughput, you need to find the bottleneck and improve that until it's no longer the bottleneck until you can't remove anything anymore.


Your AVR has a fixed maximum frequency, aim to make its (optimized) processing routines the limiting factor.  If your communications protocol can handle this speed, then there's nothing more you can do.
USB  at 1.5Mbps represents moving 1 byte every 160 clocks.  That's quite a lot of time. to do processing, and if you're triple-buffering your data or sending out full blocks you'll want this to be even faster.  At 12 Mbps, you get 13 clock cycles per byte, which is much more difficult to achieve, and would be a reasonable goal.  
The interface between USB (and even USB itself) will carry some overhead.  Sending ASCII data over an asynchronous serial interface is a big overhead.  A parallel FIFO USB interface chip will be much faster. A micro with built-in USB peripheral will be better yet.


To get more reliability, you need to stick with what works and what you know best.  Relax the specs.  Here, a tried-and-true serial interface is a fine choice.



How does "serial over USB" work in your application?

If it's a software driver on the embedded Linux device, and a USB &lt;-> parallel IC (or software serial port on a USB-enabled AVR, which sounds likely since you've tried HID mode), then you'll have no problems with higher bit rates.  

However, true USB &lt;-> serial ICs do exist and are popular (the FTDI232R on older Arduinos comes to mind...) which turn USB into logic-level serial RS232 communication. You don't need or want this layer for your application.  It will reduce your throughput and add an additional asynchronous delay section into your application.