Issue title: Encounter NAN value when calculating the metrics
 Issue body: We train and validate the Swin-Transformer on a custom datasest. The number of classes is set as 2. The model can be trained as normal, but encounter NAN value when calculating the metrics, shown in the figure below. It seems like the images and annotations are correctly loaded, since the validation program can be executed as normal.
![图片](https://user-images.githubusercontent.com/36557512/168205692-9f85e53b-0af8-465a-8236-4bf61d006f64.png)

 Comments: 
 Comment 0: OK, I found the problem. The annotation classes are mislabeled. 
{'source': 'GithubIssues', 'repo_name_with_owner': 'open-mmlab/mmsegmentation', 'issue_id': 1234699002, 'repo_stars': 4519.0}