Building a Japanese Document-Level Relation Extraction Dataset Assisted by Cross-Lingual Transfer
Youmi Ma An Wang Naoaki Okazaki







§ INTRODUCTION


Document-level Relation Extraction (DocRE) aims to identify all semantic relationships between entities in a document <cit.>.
The task promotes Relation Extraction (RE) to a more practical setting, where relations can reside between entity pairs document-wise, i.e., within and beyond the sentence boundary.
DocRE is worth spotlighting as it not only inherits the significance of RE in benefiting knowledge graph completion and question answering but also showcases how models comprehend long text <cit.>.
Even in the era of large language models (LLMs), the task deserves more attention as in-context learning of DocRE was considered not yet feasible <cit.>. 

DocRE research has been conducted mainly in English <cit.>.
This work aims to promote DocRE in other languages with the help of English resources.
Specifically, we utilize existing resources of English DocRE to construct datasets and models for non-English DocRE.
We chose Japanese as our target language for the following two reasons.
Firstly, despite Japanese being a widely used language for web content, there is currently a notable absence of general-purpose Japanese DocRE resources.
Our work thus contributes to the community by establishing the foundation for Japanese DocRE.

Secondly, Japanese stands out as one of the most linguistically distant languages from English <cit.>. 
The dissimilarity encompasses various aspects, including script models and word order. 
Therefore, our research setting is highly representative, and the insights we gain will hold value when acquiring resources for other languages.





We first explore if DocRE resources of high quality can be obtained with zero human effort.
To this end, we automatically construct a Japanese DocRE dataset with cross-lingual transfer.
Specifically, we translate Re-DocRED <cit.>, a popular English DocRE dataset of high quality, into Japanese with a machine translator.
An automatically constructed dataset (hereafter referred to as Re-DocREDja) can thus be obtained without human annotators.
The translation-based cross-lingual transfer has been successfully applied to other information extraction (IE) tasks, including named entity recognition and sentence-level relation extraction <cit.>.
However, we observe that models trained on Re-DocREDja suffer from low recalls when extracting relation triples from raw Japanese text.
We investigate the error cases and attribute the failures to the discrepancies between documents in Re-DocREDja and those composed by native speakers.
The discrepancies include deviations of topics and wording. 
Our observation underscores the uniqueness and complexity of DocRE in comparison to other IE tasks.



Given that Re-DocREDja is not suitable for immediate practical application, we explore if the dataset can assist human annotation.
As in Figure <ref>, we adopt a semi-automatic, edit-based annotation scheme, where annotators edit machine recommendations by removing incorrect instances and supplementing missed instances <cit.>.
In contrast to previous works where only relation instances from an existing knowledge base are recommended <cit.>, we recommend instances with a state-of-the-art DocRE model trained on Re-DocREDja.




The collected dataset is named as JacRED (Japanese Document-level Relation Extraction Dataset), with statistics shown in Table <ref>.
We quantitatively analyze recommendations from the model trained on Re-DocREDja and those from knowledge base queries and find the former reduces the human edit steps to half of the latter.

We employ JacRED as a benchmark for evaluation.
Firstly, we evaluate the performance of existing models on Japanese DocRE. 
While models trained using the train set of JacRED perform fairly on the test set, the scores fall short of those achieved on Re-DocRED.
The result indicates that JacRED introduces extra challenges in addition to Re-DocRED.
Notably, we observe that in-context learning of LLMs yields poor performance on JacRED, in line with the findings of <cit.>.
Next, we quantify the performance gap between models trained on Re-DocREDja and those trained on JacRED.
The results further demonstrate that, although translation-based cross-lingual transfer appears effective for a range of IE tasks, it does not hold true for DocRE, especially for distant language pairs.
Additionally, JacRED also enables the evaluation of cross-lingual DocRE.
We assess the cross-lingual transferability of existing DocRE models between English and Japanese, from which we observe challenges due to the complexity of document semantics.
Our dataset will be publicly available[The dataset is available at <https://github.com/YoumiMa/JacRED>].




§ DATASET CONSTRUCTION





  
Task Definition. 
For each document D consisted of n sentences 𝒳_D = x_1, x_2, …, x_n, entities within the document are given as ℰ_D = {e_1, e_2, …, e_k}, where each entity e_i ∈ℰ_D is a collection of all its proper-noun mentions e_i = {m_1^i, m_2^i,…, m_l^i }.
A DocRE model is expected to extract all relation triples within the document in the form of (e_h, r, e_t), where e_h is the head entity, e_t is the tail entity, and r is a relation label chosen from a predefined set. 
Additionally, we also expect the model to perform evidence retrieval, where evidence for each relation prediction is provided at the sentence level.
In other words, for a predicted triple (e_h, r, e_t), the model should be able to return the evidence sentences 𝒱_e_h,r,e_t⊆𝒳_D.



  
Approach. We explore ways to construct language resources for Japanese DocRE using existing English resources.
To do so, we start by automatically building a dataset with cross-lingual transfer(Section <ref>).
The approach has been reported successful in other IE tasks <cit.>; If the transferred dataset portrays the characteristics of Japanese DocRE well, there is no need to recruit human annotators.
However, we observe that the DocRE models trained with such a dataset err on raw Japanese text.
Nevertheless, the model yields predictions of fair quality.
We thus adopt the model trained on the transferred dataset as an intermediary tool to assist human annotation (Section <ref>).






 §.§ Automatic Construction



We build a Japanese version of Re-DocRED <cit.>.
Re-DocRED revises DocRED <cit.>, the first and most popular DocRE dataset collected from English Wikipedia.





  
Translation and Annotation Projection.
We translate the complete train/dev/test splits of Re-DocRED into Japanese with the help of machine translators.
As shown in Figure <ref>, XML tags are inserted around each entity.
Documents are translated from English to Japanese with the tags so that entity spans are projected jointly during the translation process.
Relations associated with can be thus directly inherited from the English dataset.
This mark-then-translate method has been reported to work well for multiple structured prediction tasks <cit.>.
We utilize DeepL to perform translation, as it enables translation while preserving XML tag markups[<https://api.deepl.com/v2/translate>].



  
Post-processing for Case Markers.
Given the translation as in Figure <ref>, we recognize the necessity of post-editing due to the presence of case markers in entity spans.
Case markers (“kaku-joshi” in Japanese) are special linguistic units attached to the end of nouns to indicate the relationship between words.
A case marker only reveals the grammatical role but does not contribute to the semantics of the noun phrase it is attached to.
UTF8min
For example, in entity span <e0> of the Japanese translation, a topic marker “は” following “モロゴロ州” (Morogoro Region) indicates the noun phrase to be the topic of this sentence.

We detach case makers from the entity span with the Japanese morphological analyzer MeCab <cit.>[<https://taku910.github.io/mecab/>], removing tokens identified as particles at the end of each span.
The obtained dataset is denoted as Re-DocREDja.





  
Limitations of Transferred Dataset.
When utilizing Re-DocREDja as the training data and test bed, we witness DREEAM <cit.>, the current state-of-the-art DocRE model, achieving an F1 score of 72.74 (cf. the same architecture scores 77.94 on the original Re-DocRED).
However, when “real” Japanese documents from Japanese Wikipedia are fed into the model, we observe relation triples being left out in the predictions, with typical examples demonstrated in Figure <ref>.
Two possible reasons can be raised to explain why the model trained on Re-DocREDja fails: 
(1) Topic Shift of Contents: Re-DocREDja cannot represent the real topic distribution of Japanese documents.
Collected from English Wikipedia, Re-DocRED consists of contents that English speakers are concerned about, which do not necessarily match the interests of Japanese speakers.
As in Figure <ref>, Re-DocREDja lacks documents about Japanese culture, preventing the DocRE models from being localized.
(2) Gap of Surface Structures: The surface structures, i.e., how words are organized in the sentence, of Re-DocREDja follow the logic of English, which is distinct from that of Japanese.
Figure <ref> showcases a typical example of how Japanese differs from English in surface structures regarding the omission of subjects.
Re-DocREDja thus cannot reproduce the surface structures of “real” Japanese, resulting in failures of the trained model.



 §.§ Semi-Automatic Construction



Having observed drawbacks of Re-DocREDja, we postulate that human annotations are necessary to better depict Japanese DocRE.
We thus involve human annotators in constructing a Japanese DocRE dataset, which we call JacRED.
The annotation process consists of two phases: the entity mention annotation phase and the relation annotation phase.
Both phases follow an edit-based scheme <cit.>: Annotators only need to edit machine recommendations instead of listing all relation instances from scratch.

The quality of machine recommendation is crucial under the edit-based scheme: Poor recommendations require more edits, which will drastically increase the annotators' workload and affect the dataset's quality.
The problem is recognized in DocRED as the false-negative issue, where too many relation instances are left out in the recommendations to be mended by human edits <cit.>.
We propose to mitigate this issue using Re-DocREDja, utilizing a model trained on Re-DocREDja to recommend relation instances.





  
Documents. 
JacRED is built on top of the Japanese edition of Wikipedia.
We clean up the dump and extract the opening text of each page as the document[2023-01-01 dump at <https://dumps.wikimedia.org/jawiki/>], with only those longer than 256 characters kept in our annotation pool. 





  
Annotators. Given the complexity of the task, we recruit native Japanese speakers with expertise in annotating language resources instead of crowdsourcing[Measures including the Inter Annotator Agreements (IAA) are thus not reported in this paper.].
The annotators first work individually on different data and then cross-check the worked annotations.
The annotation tool is BRAT <cit.> during both phases.



  §.§.§ Entity Mention Annotation


The purpose of the entity annotation phase is two-fold: (1) to obtain high-quality entity mention annotations for each document and (2) to filter out documents involved with few entities and relations.



  
Entity Types. We adopt the definition of IREX (Information Retrieval and Extraction
Exercise, <cit.>) with 8 types, whose scope is similar to that of DocRED.
A list of entity types is provided in Table <ref> of Appendix A.



  
Machine Recommendations.
We parse each document and obtain machine predictions of named entity mentions using KWJA <cit.>, a unified analyzer for Japanese.



  
Document Filtering.
Another round of document filtering is performed based on the machine prediction to remove documents that are likely to contain few cross-sentence relations.
To this end, we first link each mention to Wikidata entities <cit.>.
If an edge with label r connects a certain entity pair (e_h, e_t) in the knowledge base, we treat (e_h, r, e_t) as an extractable relation triple from the document, following the distant-supervision assumption <cit.>.
Only documents with more than 4 cross-sentence relations are preserved in the annotation pool.
We employ mGENRE <cit.> for entity linking and KGTK <cit.> for connectivity check.



  
Human Edits.
We randomly select 2,000 documents from the annotation pool for human annotation.
Human annotators review recommendations in each document, correcting wrongly predicted entity mentions and supplementing missed ones.



  §.§.§ Relation Annotation


Relations and coreferences are annotated based on entities.
Our approach differs from existing works in that (1) we define a smaller relation label set that covers a sufficient number of relation instances, and (2) we provide machine recommendations with a model trained with Re-DocREDja.



  
Coreference Recommendations.
For each entity e_i, we treat all its mentions {m_1^i, …, m_l^i} as coreferences of each other.
As introduced in the task definition, we only consider proper nouns as mentions while excluding the pronouns.
Mentions linked to the same Wikidata entity are recommended as coreferences.



  
Relation Types.
(Re-)DocRED's relation label set ℛ contains 96 relation types.
However, it is hard for annotators to comprehend such a large label set, which will eventually affect the annotation quality.
We thus reduce the relation label set based on the following principles:
(1) All relation categories defined in ERE <cit.> should be covered;
(2) Explicitly-defined inverse relation pairs, e.g.,  has_part and part_of, are merged into one;
(3) Relations frequently appearing in Re-DocRED are preserved as much as possible.
This results in a label set ℛ' of 28 relations covering over 88% relation instances in Re-DocRED.



  
Relation Recommendations.
We project the label set of Japanese Re-DocRED from ℛ to ℛ' and retrain a DREEAM model.
Predictions of the model are employed as machine-recommended relations.
We expect our recommendations to be more accurate than those in previous works obtained from knowledge base queries, primarily due to two factors: (1) Wikidata only stores a limited number of relation facts, while a model can, in principle, assign relation(s) to each entity pair in the document; (2) Relation facts in Wikidata are independent of the document's content, while model predictions are contextually sensitive.
A quantitative comparison of recommendations from the model trained on Re-DocREDja and those from querying Wikidata can be found in Section <ref>.




  
Human Edits.
Coreferences and relations are revised during human annotation.
For coreferences, human annotators remove irrelevant mentions and supplement missed mentions for each entity.
For relations, human annotators first examine the existence of each recommended relation.
As showcased in Figure <ref>, a pair of mentions m^h_i, m^t_i, representing entity e_h, e_t respectively, along with their relation r is shown in the interface.
If annotators consider relation triple (e_h, r, e_t) as true, they need to provide the evidence sentence 𝒱_e_h,r,e_t within the document[Sentences where mention m^h_i and m^t_i resides are treated as evidence by default. Only evidence sentences other than those need to be provided.];
Otherwise, the triple should be deleted from the dataset.
Finally, the annotators supply missing relation triples and evidence sentences with their best effort.




  
Post-processing. 
Among all 28 relation types, 7 have inverse relations defined in Wikidata.
We automatically augment triples of inversed relation types after human edits.
For example, if triple (e_h, part_of, e_t) is present in the revised annotation and relation type part_of is an inversion of has_part, a new triple (e_t, has_part, e_h) will be automatically added into the annotation.
JacRED thus includes 35 relation types eventually.
A detailed list of relation types is provided in Table <ref> of Appendix B.



§ DATASET ANALYSIS


This section reports the analysis results of JacRED to provide a deeper understanding of the collected dataset.
Firstly, we compare the statistics of JacRED against (Re-)DocRED.
The comparison suggests that JacRED combines the advantages of DocRED and Re-DocRED (Section <ref>).
Next, we calculate the number of edits human annotators made before reaching the final annotations.
We observe that significantly more edit steps would be necessary if the human annotation started from machine recommendations suggested by knowledge base queries (Section <ref>).




 §.§ Detailed Statistics







Table <ref> details the agreements and differences between (Re-)DocRED and JacRED.



  
Document Complexity.
As for document length, JacRED shares a similar scale with (Re-)DocRED at both token and sentence levels.
On one hand, documents in JacRED contain more relation instances than DocRED on average, implying that the false negative issue is mitigated in JacRED compared to DocRED.
On the other hand, documents in JacRED contain fewer relation instances than in Re-DocRED.
One possible reason is our re-definition of relation types, where symmetric relation types are merged into one as they represent the same knowledge.



  
Evidence Annotation.
Re-DocRED revises DocRED to alleviate the false negative issue by supplying missed relation instances.
However, evidence sentences for those supplied instances are not included in Re-DocRED.
In contrast, we collect human-annotated evidence sentences during the relation annotation phase.
JacRED thus better portrays the correlation between relation and evidence sentences than Re-DocRED.




 §.§ Number of Human Edits






We quantify the distance between machine recommendations and human annotations of relation instances.
To this end, we compare machine recommendations against final human annotations to see how many edits have been made.
Specifically, we randomly sample 400 documents from JacRED and calculate the number of recommendations being deleted/substituted/supplied as in Table <ref>. 


  
Human Annotations v.s. Machine Recommendations.
We observe that more than 20% of machine recommendations (1,490 out of 6,500) were regarded as inappropriate, whose relation labels were deleted or substituted by human annotators.
The human annotators also supplied another 2,740 relation instances, taking up more than 40% of the recommendations.
We thus conclude that DocRE models trained on the automatically constructed dataset still lag behind human performance considerably, suggesting the importance of a manually collected dataset.




  
Cross-Lingual Transfer v.s. Knowledge Base Queries.
We further measure the distance of human annotations from relations recommended by querying Wikidata, a de-facto method used in previous works <cit.>.
Compared with model predictions, Wikidata provides only half as many recommendations: To reach the human annotations, 50% (1,572 out of 3,200) of the recommendations need to be revised, with another 200% instances to be added.
In total, it takes 7,805 steps to align Wikidata recommendations with the final annotation, while only 4,230 steps are needed when employing cross-lingual transfer.
These statistics reveal the usefulness of Re-DocREDja in reducing human efforts.



§ EXPERIMENTS




  
Purposes.
We employ JacRED as a benchmark to examine the capability of existing DocRE models.
Our major concerns are:
(1) How well can existing DocRE models perform Japanese DocRE?
(2) How different can a DocRE model perform when trained on Re-DocREDja and JacRED?
Additionally, we evaluate the cross-lingual transferability of existing DocRE models with JacRED.


  
Settings.
We split JacRED into train/dev/test sets with 1400/300/300 documents.
Models are trained and evaluated on a single Tesla V100 16GB GPU.
For evaluation, we follow previous works to compute the micro-averaged F1 scores for relations and evidence sentences <cit.>.
Additionally, we compute Rel F1 Ign, a variant of F1, where relation instances seen in the training set are ignored during evaluation.
Average scores of 5 runs initialized with different random seeds are reported throughout this paper.



 §.§ Models Trained on JacRED



We measure the performance of existing models when supervised by the training split of JacRED.
Specifically, we train and evaluate 4 popular models on top of tohoku-nlp/bert-base-japanese-v2 available on Huggingface[<https://huggingface.co/tohoku-nlp/bert-base-japanese-v2>], with results summarized in Table <ref>.
Among these models, DREEAM is the current state-of-the-art model on (Re-)DocRED for extracting both relations and evidence sentences.
We also evaluate the performance of LLM with in-context learning.



  
JacRED introduces extra challenges beyond those in Re-DocRED.
In Table <ref>, all DocRE models score above 60 on Relation F1. 
Although acceptable, the performance of each model is worse than their equivalents trained on Re-DocRED, with a gap of 10 F1 points (cf. Table <ref>).
The result suggests potential challenges in JacRED that are absent from Re-DocRED, possibly due to the characteristics of the Japanese language, such as the omission of subjects.
Addressing such characteristics may be essential to better tackle Japanese DocRE.




  
In-context learning of LLMs on JacRED is non-trivial.

Apart from models specially designed for DocRE, we also evaluate how LLMs can tackle the task via in-context learning.
Specifically, we pre-define the relation label set and include a pair of (document, relations) in the prompt to guide the LLM in conducting DocRE.
In our experiments, we utilize models provided by OpenAI, namely (1) the instructed version of GPT-3.5 accessed via the API key gpt-3.5-turbo-instruct <cit.> and (2) GPT-4 accessed via the API key gpt-4[Details of the prompt is provided in Figure <ref> of Appendix C.].
As in the last two rows of Table <ref>, GPT-3.5 exhibited much lower performance than the DocRE models.
GPT-4 improved over GPT-3.5 but still lagged behind the supervised DocRE models.
Similar insights have been provided by <cit.>, where in-context learning of DocRE could not be conducted due to the length restriction of the prompt.
We succeeded in instructing LLM to conduct DocRE, while the performance is limited.
The experiment results thus highlight the challenge of DocRE as a task that LLMs cannot easily tackle.



 §.§ Models Trained on Transferred Re-DocRED





Section <ref> has mentioned limitations in the dataset automatically constructed from cross-lingual transfer.
Specifically, we showcased how DocRE models trained on such a dataset fail to extract relation triples from raw Japanese documents.
Here, we quantify the performance gap between a model trained on the automatically constructed dataset (Re-DocREDja) and the human-annotated dataset with machine assistance (JacRED).
The test set of JacRED is adopted as the benchmark, with results shown in Table <ref>.




  
Models trained on Re-DocREDja suffer from low recalls.
From Table <ref>, we witness that DREEAM trained on Re-DocREDja underperforms its equivalent trained on JacRED.
Taking a closer look at the scores, we find the gap in recalls (73.29 v.s. 53.67) is more significant than that in precisions (64.76 v.s. 56.14).
The result corresponds to our observation in Section <ref> that models trained on the transferred dataset cannot identify some relation instances due to the limitation of texts translated from English.



  
The gap between models trained on Re-DocREDja and JacRED is evident under the same setting.
We further train DREEAM on Re-DocREDja with 1,400 documents, aligned with the number of documents in JacRED.
The F1 score drops from 54.87 to 53.56, lagging behind that of the model trained on JacRED with a gap of 15 F1 points.
The results indicate that JacRED provides better supervision than Re-DocREDja.




 §.§ Crosslingual DocRE





JacRED also enables the evaluation of cross-lingual DocRE.
Although DocRE datasets have been collected in Chinese <cit.> and Korean <cit.>, they lay in different domains than (Re-)DocRED.
In contrast, JacRED is collected from Wikipedia following a pipeline similar to DocRED.
The domain and label sets of JacRED and (Re-)DocRED thus match each other, enabling the evaluation of cross-lingual DocRE.
Here, we take the first attempt to measure the cross-lingual transferability of existing models using Re-DocRED and JacRED.



Specifically, we train models on the training set in one language and evaluate them on the test set in another.
The relation label set of Re-DocRED is projected onto JacRED using the same method as in Section <ref>.
To ensure the multilingualism of trained models, we adopt multilingual BERT (mBERT, <cit.>) as the encoder.
Evaluation results are shown in Table <ref>.



  
Cross-lingual performance of existing models is limited.
All models exhibited a decreased accuracy in the target language.
Different from sentence-level tasks, DocRE requires not only an understanding of individual sentences but also inter-sentence semantics within the whole document, which improves the difficulty of building cross-lingual models.
This may offer a potential explanation as to why translation-based cross-lingual transfer is ineffective for DocRE, despite its successful application in sentence-level RE and OpenIE <cit.>.








§ RELATED WORK





  
DocRE corpora in English. 
The most well-known definition of DocRE was proposed by <cit.>, along with a dataset collected from English Wikipedia named DocRED.
While two document-level relation extraction datasets, namely CDR <cit.> and GDA <cit.>, have been proposed ahead of DocRED, they were collected in the biomedical domain, thus unsuitable for developing general-purpose DocRE models.
DocRED suffers from the false negative issue where a considerable amount of relation instances are absent from the ground-truth annotations <cit.>.
<cit.> randomly selected 96 documents from DocRED and relabeled them from scratch, while <cit.> revised the whole dataset as Re-DocRED with machine assistance.
This work follows a machine-assisted annotation process as DocRED and Re-DocRED while paying extra attention to providing better machine recommendations with the model trained on a dataset transferred from Re-DocRED.



  
DocRE corpora in other languages.

<cit.> constructed HacRED from Chinese DBpedia to promote relation extraction from complex contexts.
<cit.> focused on Korean historical RE research and collected HistRED from a travel diary written between the 16th and 19th centuries.
These datasets were collected independently from DocRED with distinct domains and label sets.
Apart from these studies, <cit.> released a system for medical relation extraction on Japanese documents, while the dataset is not publicly available.
In this work, we explore how existing resources can help construct DocRE resources in other languages.
We share the insights that models trained under cross-lingual transfer techniques are not ready for practical use.
However, they serve as good assistants for aiding human annotations.



  
Cross-lingual transfer for structured predictions.
Several works have adopted translation-based cross-lingual transfer approaches to solve cross-lingual and multi-lingual structured prediction tasks <cit.>.
More recently, <cit.> constructed MultiTACRED, a multilingual version of TACRED tacred, using similar approaches as ours.
They confirmed the dataset's quality to be high enough even without human modifications.
Our work examines the approach's usefulness in the literature of DocRE and reports its shortcomings. 
Unlike other sentence-level IE tasks, DocRE involves understanding not only single sentences but also the whole document, improving the difficulty of cross-lingual transfer.





§ CONCLUSION


This work publishes JacRED, the first benchmark for general-purpose Japanese DocRE.
In the process of building JacRED, we explore how to utilize existing English DocRE resources to construct resources for other languages, using Japanese as the representative. 
Starting from constructing a dataset by translation-based cross-lingual transfer, we have shown how and why such a dataset is not ready for practical use.
Nevertheless, models trained on the dataset can replace existing approaches, i.e., knowledge base queries, to provide better recommendations for human annotation.
Our insights can benefit the development of DocRE resources for other languages.
Benchmarking with JacRED portrays the challenge of not only Japanese but also cross-lingual DocRE.

In the future, we plan to utilize models trained on JacRED to help downstream tasks such as question answering and reading comprehension.



§ ETHICS STATEMENT


In this work, we collected a dataset from Wikipedia, whose text content can be used under the terms of the CC-BY-SA[<https://en.wikipedia.org/wiki/Wikipedia:Text_of_the_Creative_Commons_Attribution-ShareAlike_4.0_International_License>].
We thus presume that no copyright issues are involved in constructing and publishing our dataset.


  
Automatic Annotations.

For the machine translator, we adopted DeepL API at the cost of 2,500 JPY (approx. 16$) per 1 million characters.
For the LLM, we tested with the instructed version of GPT-3.5 provided by OpenAI at the cost of $0.0015 per 1 thousand tokens for input and $0.002 per 1 thousand tokens for output.
For existing DocRE models, all resources we adopted are publicly available and free of charge.



  
Human Annotations.
Before the annotation, we arranged meetings in advance to (1) explain the purpose of collecting the dataset and (2) adjust the workload.
The annotators understand and agree that their work will be used to train neural networks.
For both the entity and relation annotation phases, we explained the purpose of building the dataset and provided a detailed annotation guideline.
During the annotation, we frequently discussed with the annotators how to handle irregular cases and adjust the guidelines when necessary.
7 annotators are involved in the entity mention annotation phase, and 6 annotators are involved in the relation annotation phase.
Each annotator is paid 5,000 JPY (approx. 30$) per hour, which is higher than the standard salary in Japan.




§ ACKNOWLEDGEMENTS


This paper is based on results obtained from a project, JPNP18002, commissioned by the New Energy and Industrial Technology Development Organization (NEDO).
*


§ BIBLIOGRAPHICAL REFERENCES


lrec-coling2024-natbib




§ LANGUAGE RESOURCE REFERENCES


lrec-coling2024-natbib
languageresource





§ APPENDIX A: ENTITY LABEL TYPES



In this section, we list all entity types in JacRED as in Table <ref>, together with those defined in (Re-)DocRED.





§ APPENDIX B: RELATION LABEL TYPES




In this section, we list all relation types included in JacRED as in Table <ref>.






§ APPENDIX C: PROMPT FOR IN-CONTEXT LEARNING



We showcase the prompt used for the in-context learning of LLM in Figure <ref>.
In previous work where LLM are utilized for relation extraction <cit.>, the prompt has been designed to return all relation triples within a document.
However, it is hard to identify all relation triples across a document at once.
Furthermore, most supervised approaches tackle DocRE by classifying relation types entity-pair wise <cit.>. 
We thus design prompts to reduce the task complexity by querying one relation type for each API call[In early experiments, we evaluated the performance when querying: 1) one relation type; 2) all relation types of one entity pair; 3) one relation type of one entity pair during each API call, where 2) yields good performance at a low cost.].
By using our prompt, GPT-3.5 yields better performance than reported in existing works.







