Wiring diagram check up for usb interfaced avr
You really need a switch to control where the power is coming from, at the very least you want some sort of current limiting.  

Without such control both the LDO and the USB bus can be providing +5V but there will be small offsets between the two +5V levels which can cause large currents to flow as the resistance between the supplies in your design is very low.  

Current from the USB host controller should be limited to 100mA if you haven't requested more power but bouncing off that current limiter may cause issues depending on the implementation of the limiting.  This could be limited if the LDO has an internal protection diode or you supply one externally.

If the +5V is slightly higher on the board when being supplied from the jack, then current will flow back at the host controller, which appears to be the issue your seeing.  This could cause lots of issues of various forms on the host side.

Basically what you need is to OR the USB bus power with the output of LDO and prevent reverse current flow.  

This can be done with a pair of diodes, one on each supply line, pointing toward the micro-controller.  However, the forward voltage drop from the diode is going to be roughly 0.5V.  With a USB input that can go as low as 4.75V your supply voltage at the micro-controller may be as low as 4.25V which may be outside the operational range of the micro-controller.

In that case there are special devices designed for this purpose, here is an example.  Such a device will reduce your forward drop to 50mV or less in most cases.  

If you also need to support USB device suspend mode then you need to also handle a power limit of 500uA in this mode.  This will require an external power enable switch which basically shuts everything down, including the regulator.

EDIT: If you don't mind it not being automatic, you can just put a jumper on the board to pick between the power sources.

EDIT2:

Excuse the poor picture editing but hopefully this clears up what I'm saying.  This is what you would do to OR the two +5V rails with diodes: