How much drag do the gear doors create during retraction?
Of course gear doors create drag when open.  The question is "how much?"

As you suspect, door drag is not the critical factor here.  As usual the answer to your question is aircraft-specific, and your question is pointing to a generalized "old wive's tale" which is only true sometimes.  

OK, what is the story behind the story?  Look at this Youtube video.  Watch the main gear as it enters/exits the storage wells in the fuselage.  The axles of the wheels are close to parallel to the direction of flight, which means the wheels are being dragged broadside in the slipstream.  A very high drag scenario which will affect aircraft performance.  The POH recommendation is (from memory, now) to climb over critical obstacles on takeoff with the gear down, and raise the gear only when the obstacle is cleared.  The attached video shows why.

This advice does not pertain to all planes.  Know your plane, read the POH.  Oh... and ask questions here!