Setting class weights for categorical labels in Keras using generator
For categorical data, it is best to use sample_weight instead of class_weight argument. This can be done by simply giving all samples of a particular class the same weight. sample_weight works for categorical data because it takes a numpy array as its value as opposed to a dictionary (which won't work for categorical class labels) in case of class_weight.

See: Keras sequential model methods


  You can either pass a flat (1D) Numpy array with the same length as
  the input samples (1:1 mapping between weights and samples), or in the
  case of temporal data, you can pass a 2D array with shape (samples,
  sequence_length), to apply a different weight to every timestep of
  every sample. In this case you should make sure to specify
  sample_weight_mode="temporal" in compile().


The other way, as you mentioned is using custom weighted loss function. A detailed discussion on it can be found here.