Optimizing a vector loop
Educated guesses are: in this case, loop unrolling will probably feature in a big way, and may account for most of the 20X improvement. Additionally, with the loop unroll it may make other things like instruction-level-parallelism more effective, and even cache-line management.
There are essentially only two comments to make on your situation:

non-optimized code is going to be slower.... you pay money for the compiler for a reason, lots of smart people make it 'go fast'.
you need to inspect the instruction-level code (asm) generated by the compiler in order to understand the main differences between the two versions ... anything else is just educated guessing.

Edit
FWIW, Purely for interest sake.... , you may want to try to manually unroll your inner-most loop, and see if this makes a significant difference:
        for (auto it = units.begin() + n; it != (units.begin() + (n + j + 1)); it++)
        {
            sizesMap[1] += (*it).sizes[1];
            sizesMap[2] += (*it).sizes[2];
            sizesMap[3] += (*it).sizes[3];
            sizesMap[4] += (*it).sizes[4];
            sizesMap[5] += (*it).sizes[5];
            sizesMap[6] += (*it).sizes[6];
            sizesMap[7] += (*it).sizes[7];
            sizesMap[8] += (*it).sizes[8];
            sizesMap[9] += (*it).sizes[9];
            sizesMap[10] += (*it).sizes[10];
            sizesMap[11] += (*it).sizes[11];
        }