Specialized NER models. The three BERT-based models, used for fine-tuning in this work, have the same architecture and mainly differ in the data used to pre-train the models. BERT is pre-trained on generic domain data from Wikipedia and Book-Corpus datasets. SciBERT uses data from scientific domains by pre-training on a subset of more than a million full-text documents retrieved through SemanticScholar , including 18% of computer science papers and 82% of medical papers. MatSciB-ERT uses datasets closer to the fatigue domain, pre-training on a materials science corpus containing 40% words from papers on inorganic glasses and ceramics, and 20% each from metallic glasses, alloys, and cement. As can be seen in Figure , SciBERT and MatSciBERT both significantly outperformed BERT by 2-4% F1-score. Since the models differ mainly by pretraining datasets, the performance increase can be attributed to domain-specific pre-training. Domain specific pre-training has been shown to be helpful for various downstream tasks in domains including biomedicine (BioBERT ), finance (FinBERT ), law (CaseHold ), as well as materials science (MatBERT and MatSciBERT). This is especially beneficial if the dataset used for the downstream task is smaller . However, it is to be noted that the pre-training data closest to the materials fatigue domain, i.e., the MatSciBERT corpus, is not specific for this subdomain which makes the model struggle to identify fatigue-domain specific terminologies. This could also be the reason behind the modest performance improvements for MatSciBERT compared to SciBERT. The inclusion of scientific writing styles in the datasets used in MatSciBERT and SciBERT might be primarily causing the performance improvement compared to BERT which used a diverse pretraining dataset with varying writing styles. Some common confusions of the best conventionally fine-tuned MatSciBERT illustrated in Figure also motivate the need for even more domain-specific pre-training. For instance, the wrong prediction of "Defect" and "Microstructure" as "Damage" highlights that the model cannot learn the fine-grained differences between these classes. This could also be complicated by the presence of ground truth entities of these types that contain common words. For example, 'multi-GB cracks' (Figure 2.) and 'dislocation substructures' (Figure 7.) are of type "Damage" while 'low-S GBs' (Figure 2.) and 'trapped dislocations' (Figure 7.) are "Defect" entities. Our reasoning on the distinction between these semantically close entity types is provided in the accompanying data set publication . The presence of common words in example entities and their semantic closeness could also be the reason for the model confusion between "PhysicalQuantity" and "MechanicalTestParameter" -'stress', 'strain', 'temperature', etc. are physical quantities (state variables) while 'strain rate', 'stress amplitude', 'testing temperature', often refer to mechanical test parameters (process metadata). Entities of this nature are typical in materials science, thus underscoring the necessity for pursuing nested NER approaches. The model missing predictions and performing relatively poorly on the "Mechanism" entity type, as seen in Figure , can likely be ascribed to the broad interpretation of the "Mechanism" class which resulted in the inclusion of entities with a large semantic scatter in the annotated datasets. This motivates the refinement of the sub-hierarchy of the "Mechanism" class in the materials mechanics ontology 31 and replacing "Mechanism" entity type with more fine-granular entity types.