Is center of symmetry only valid for symmetric substituents?
You assume too much, starting with the words "The given compound". There is no given compound. The picture corresponds to quite a few different compounds, and here is why.

As I told you before on a remotely similar question, the biphenyl fragment totally can't be planar. The xylene substituents must be rotated so their ortho methyl group points either up or down relative to the plane of the central ring. Since there are two of them, we have three options: "both up", "both down", and "one up, one down". The first two compounds are indeed chiral, and enantiomers of each other. The third one is centrosymmetric, just as you've reasoned.