Any process that decreases the frequency, rate or extent of a process involved in the formation, arrangement of constituent parts, or disassembly of an organelle.
negative regulation of organelle organization and biogenesis