Load voltage and current of relays
... The voltage during the "blocking mode" will have the inverter
  switching frequency component.


Not a good idea as Charles Cowie said, you can damage the inverter output. But moving on to your questions...


  For a common relay like this, does it matter if the "conducting"
  voltage and current have a frequency of a few kHz?


As a gross generalized rule, it is permissible (from the contactor's standpoint) for it to be used on the output side of an inverter, it is generally done on safety systems. But contactors used in machine safety systems have other requirements, such as "force guided contacts" that this particular contactor does not have. So be careful about what you are planning on here. 


  In the relay product page I attached above, it only says the coil volt
  is 24 V. Do relays also have a "breakdown voltage" or "blocking
  voltage" like IGBT or MOSFET?


Not in the same way as electronic components. Relays have a "working voltage", in this case 600VAC maximum. There is also a dielectric breakdown voltage, usually much higher than the actual working voltage, but that does come into play when attempting to interrupt an arc coming from an inverter that is firing at a high carrier frequency. this contactor will be fine, but again, the INVERTER may not like having an arc opening on the output under load.


  Also, what about solid state relays?


Absolutely out of the question.

Describe what you are attempting to accomplish with this, there may be better ways to get there.