How can you test what color different people perceive?
One way we can get evidence qualia are the same or very similar for different people is by reactions to it, beyond just the word. 

For example, beyond the word "pain", we have other strong reactions to pain. So nobody 
suspects other people might experience pain as pleasure and vice versa. Obviously not!
There are no obvious signs for colour qualia, so it makes sense to suspect some people experience red as blue and vice versa. 

Still, there are weak reactions to colours, like sometimes certain colours are associated with certain emotions. But, this is very weak evidence and could easily mismeasured, or be bound up with culture rather than biology.

We could test it: take pairs of identical twins at birth (so that they haven't learned words for colours yet), re-wire optical nerves of one of each pair so that red and blue receptors were switched (and perform a placebo operation on the other one, so they don't know who is who), and see whether there is any statistically significant change in the attitudes to different colours or the like as they grow up. 

The experiment has various technological and ethical problems :-) But it could be done in principle, so to me this shows the question isn't meaningless, just that it's hard to find the answer.