How can STATCOM output voltage higher than system voltage?
You can make the DC voltage theoretically as high as you want for such a topology because it is a boot-type converter. The basic relationship for the DC-link voltage is

Udc = 2*Unmax/M

with modulation index M in a range [0 .. 1.15], dependent on the modulation signal of your PWM

Unmax ... amplitude of converter input phase voltage (not line-line but line-ground)