What's more aerodynamic on flat bars: backpack or pannier?
A back pack is faster than a single pannier.  This exact scenario was tested in the Specialized "Win Tunnel" in their "The Brew Master" episode. They suggested the time savings was 2 minutes over 40 km, which isn't that much for casual cycling considering how uncomfortable a heavy back pack can be!   Also see their "Areo at Slow Speeds" to understand how the results can be generalized to a wide array of average speeds.

Extrapolating to the "real world"

Answering @JPMay comment request


  In the "Brew Master" video they don't really quantify things in the most helpful way (if you are not racing). In terms of percentages, the least aero option they tested is only about 2% slower than the fastest (assuming you go between 20-30 km/h). – JP May


The time difference would likely be more useful as a percentage so it could be applied to different speeds.  Drag (what you have to overcome by pedaling) is directly proportional to the drag coefficient:

drag = CdA * p * v^2


where drag (and the power required to ride a speed) goes up by the square of velocity (v), but scales linearly by the drag coefficient (CdA). Whether you ride with a back pack or panniers impacts the CdA value only.  For example, reducing your CdA by 5% will reduce your required effort by 5% whether you are riding at 20 kph or 40 kph.  This is why aerodynamics still matter at slower speeds (its just harder to "feel").

What we don't know is how fast the hypothetical 40 km was ridden in the wind tunnel testing. Assuming a race pace of 40kph, this would work out to about 3.33% longer for a single pannier vs a backpack.  

Therefore, to understand the impact on real world conditions take your ride duration with a backpack and multiple the time by the following to determine the time it would take to ride with pannier(s): 


one small rear pannier multiply the time by 1.0333  (3.33% longer)
two small rear panniers multiply the time by 1.0667  (6.67% longer)