Can you sum and difference in a single op-amp?
One potential issue I see with this approach is that since the inputs of the op amp will not remain at a fixed voltage, some of the voltage seen on the "sum" inputs of your circuit will be "visible" on the all of the inputs (sum and difference alike).  If those inputs are not being driven by low-impedance sources, that could pose a problem.

A better design in some cases, if you have the power-supply margin to accommodate it, may be to have the summing inputs feed into one amplifier wired as an inverter (with a non-inverting input that sits at a fixed voltage), and have the output of that inverter fed as one of the inputs to the amplifier that accepts all the difference inputs.  If your sum and difference inputs are intended to be grouped as differential pairs this approach won't be great (the non-inverting inputs will flow through two op amps while the inverting inputs only flow through one) but your original approach won't be either.  Your best approach in that case would be to either pass all the sum inputs into one inverting amp, all the difference inputs into another amp, and then take the difference of the two outputs, or else to pass each differential pair through its own instrumentation amp and sum the results.  Using a separate instrumentation for each pair would give the by far the best CMRR, but would of course require more amplifiers.