Why can the IS31FL3731 only handle 32 RGB Leds when it can handle 144 Regular ones?
if you look closely at the matrixes you will see that they are not regular square crosspoint matrixes, but instead charlieplexed. because of this each logical row and column has only 8 positions which limits the number of RGB leds that can be fitted.

thus there are only 16 places in each matrix where common-cathode or common-anode RGB leds will fit, charlieplexing also has voltage concerns where different led colours are mixed.