Update README.md

README.md

@@ -13,18 +13,26 @@ thumbnail: https://github.com/Marcosdib/S2Query/Classification_Architecture_mode
 # MCTI Text Classification Task (uncased) DRAFT
 
 Disclaimer: 
-sentences in the original corpus, and in the other cases, it's another random sentence in
-## According to the abstract,
-
-With probability 0.5, sentence A and sentence B correspond to two consecutive sentences in the original corpus, and in
 
-Text classification is a traditional problem in Natural Language Processing (NLP). Most of the state-of-the-art implementations require high-quality, voluminous, labeled data. Pre- trained models on large corpora have shown beneficial for text classification and other NLP tasks, but they can only take a limited amount of symbols as input. This is a real case study that explores different machine learning strategies to classify a small amount of long, unstructured, and uneven data to find a proper method with good performance. The collected data includes texts of financing opportunities the international R&D funding organizations provided on theirwebsites. The main goal is to find international R&D funding eligible for Brazilian researchers, sponsored by the Ministry of Science, Technology and Innovation. We use pre-training and word embedding solutions to learn the relationship of the words from other datasets with considerable similarity and larger scale. Then, using the acquired features, based on the available dataset from MCTI, we apply transfer learning plus deep learning models to improve the comprehension of each sentence. Compared to the baseline accuracy rate of 81%, based on the available datasets, and the 85% accuracy rate achieved through a Transformer-based approach, the Word2Vec-based approach improved the accuracy rate to 88%. The research results serve as asuccessful case of artificial intelligence in a federal government application.
+## According to the abstract,
 
-This model focus on a more specific problem, creating a Research Financing Products Portfolio (FPP) outside of
-the Union budget, supported by the Brazilian Ministry of Science, Technology, and Innovation (MCTI). It was 
-introduced in ["Using transfer learning to classify long unstructured texts with small amounts of labeled data"](https://www.scitepress.org/Link.aspx?doi=10.5220/0011527700003318) and first released in
-[this repository](https://huggingface.co/unb-lamfo-nlp-mcti). This model is uncased: it does not make a difference
-between english and English.
+Text classification is a traditional problem in Natural Language Processing (NLP). Most of the state-of-the-art implementations
+require high-quality, voluminous, labeled data. Pre- trained models on large corpora have shown beneficial for text classification
+and other NLP tasks, but they can only take a limited amount of symbols as input. This is a real case study that explores 
+different machine learning strategies to classify a small amount of long, unstructured, and uneven data to find a proper method
+with good performance. The collected data includes texts of financing opportunities the international R&D funding organizations
+provided on theirwebsites. The main goal is to find international R&D funding eligible for Brazilian researchers, sponsored by
+the Ministry of Science, Technology and Innovation. We use pre-training and word embedding solutions to learn the relationship
+of the words from other datasets with considerable similarity and larger scale. Then, using the acquired features, based on the
+available dataset from MCTI, we apply transfer learning plus deep learning models to improve the comprehension of each sentence.
+Compared to the baseline accuracy rate of 81%, based on the available datasets, and the 85% accuracy rate achieved through a 
+Transformer-based approach, the Word2Vec-based approach improved the accuracy rate to 88%. The research results serve as 
+asuccessful case of artificial intelligence in a federal government application.
+
+This model focus on a more specific problem, creating a Research Financing Products Portfolio (FPP) outside ofthe Union budget,
+supported by the Brazilian Ministry of Science, Technology, and Innovation (MCTI). It was introduced in ["Using transfer learning to classify long unstructured texts with small amounts of labeled data"](https://www.scitepress.org/Link.aspx?doi=10.5220/0011527700003318) and first released in
+[this repository](https://huggingface.co/unb-lamfo-nlp-mcti). This model is uncased: it does not make a difference between english
+and English.
 
 ## Model description
 
@@ -241,23 +249,12 @@ learning rate warmup for 10,000 steps and linear decay of the learning rate afte
 
 ### Model training with Word2Vec embeddings
 
-Now we have a pre-trained model of word2vec embeddings that has already learned relevant meaningsfor our classification problem. We can couple it to
-our classification models (Fig. 4), realizing transferlearning
-and then training the model with the labeled
-data in a supervised manner. The new coupled model
-can be seen in Figure 5 under word2vec model training.
-The Table 3 shows the obtained results with related
-metrics. With this implementation, we achieved
-new levels of accuracy with 86% for the CNN architecture
-and 88% for the LSTM architecture.
+Now we have a pre-trained model of word2vec embeddings that has already learned relevant meaningsfor our classification problem.
+We can couple it to our classification models (Fig. 4), realizing transferlearning and then training the model with the labeled
+data in a supervised manner. The new coupled model can be seen in Figure 5 under word2vec model training. The Table 3 shows the
+obtained results with related metrics. With this implementation, we achieved new levels of accuracy with 86% for the CNN
+architecture and 88% for the LSTM architecture.
 
-When fine-tuned on downstream tasks, this model achieves the following results:
-
-Glue test results:
-
-| Task | MNLI-(m/mm) | QQP  | QNLI | SST-2 | CoLA | STS-B | MRPC | RTE  | Average |
-|:----:|:-----------:|:----:|:----:|:-----:|:----:|:-----:|:----:|:----:|:-------:|
-|      | 84.6/83.4   | 71.2 | 90.5 | 93.5  | 52.1 | 85.8  | 88.9 | 66.4 | 79.6    |
 
 Table 1: Results from Pre-trained WE + ML models.
 
@@ -270,49 +267,34 @@ Table 1: Results from Pre-trained WE + ML models.
 
 ### Transformer-based implementation
 
-Another way we used pre-trained vector representations
-was by use of a Longformer (Beltagy et al.,
-2020). We chose it because of the limitation of the
-first generation of transformers and BERT-based architectures
-involving the size of the sentences: the
-maximum of 512 tokens. The reason behind that
-limitation is that the self-attention mechanism scale
-quadratically with the input sequence length O(n2)
-(Beltagy et al., 2020). The Longformer allowed the
-processing sequences of a thousand characters without
-facing the memory bottleneck of BERT-like architectures
-and achieved SOTA in several benchmarks.
-For our text length distribution in Figure 3, if
-we used a Bert-based architecture with a maximum
-length of 512, 99 sentences would have to be truncated
-and probably miss some critical information.
-By comparison, with the Longformer, with a maximum
-length of 4096, only eight sentences will have
-their information shortened.
-To apply the Longformer, we used the pre-trained
-base (available on the link) that was previously trained
-with a combination of vast datasets as input to the
-model, as shown in figure 5 under Longformer model
-training. After coupling to our classification models,
-we realized supervised training of the whole model.
-At this point, only transfer learning was applied since
-more computational power was needed to realize the
-fine-tuning of the weights. The results with related
-metrics can be viewed in table 4. This approach
-achieved adequate accuracy scores, above 82% in all
-implementation architectures.
+Another way we used pre-trained vector representations was by use of a Longformer (Beltagy et al., 2020). We chose it because
+of the limitation of the first generation of transformers and BERT-based architectures involving the size of the sentences:
+the maximum of 512 tokens. The reason behind that limitation is that the self-attention mechanism scale quadratically with the
+input sequence length O(n2) (Beltagy et al., 2020). The Longformer allowed the processing sequences of a thousand characters
+without facing the memory bottleneck of BERT-like architectures and achieved SOTA in several benchmarks.
+
+For our text length distribution in Figure 3, if we used a Bert-based architecture with a maximum length of 512, 99 sentences
+would have to be truncated and probably miss some critical information. By comparison, with the Longformer, with a maximum 
+length of 4096, only eight sentences will have their information shortened.
+
+To apply the Longformer, we used the pre-trained base (available on the link) that was previously trained with a combination
+of vast datasets as input to the model, as shown in figure 5 under Longformer model training. After coupling to our classification
+models, we realized supervised training of the whole model. At this point, only transfer learning was applied since more 
+computational power was needed to realize the fine-tuning of the weights. The results with related metrics can be viewed in table 4.
+This approach achieved adequate accuracy scores, above 82% in all implementation architectures.
 
 
 Table 2: Results from Pre-trained Longformer + ML models.
 
-ML Model Accuracy F1 Score Precision Recall
-NN 0.8269 0.8754 0.7950 0.9773
-DNN 0.8462 0.8776 0.8474 0.9123
-CNN 0.8462 0.8776 0.8474 0.9123
-LSTM 0.8269 0.8801 0.8571 0.9091
+| ML Model |  Accuracy | F1 Score  | Precision |   Recall  |
+|:--------:|:---------:|:---------:|:---------:|:---------:|
+| NN       |  0.8269   |  0.8754   |0.7950     |  0.9773   |
+| DNN      |  0.8462   |  0.8776   |0.8474     |  0.9123   |
+| CNN      |  0.8462   |  0.8776   |0.8474     |  0.9123   |
+| LSTM     |  0.8269   |  0.8801   |0.8571     |  0.9091   | 
 
 
-##< Checkpoints >
+## Checkpoints
 - Examples
 - Implementation Notes
 - Usage Example
@@ -320,21 +302,20 @@ LSTM 0.8269 0.8801 0.8571 0.9091
 - >>> ...
 
 
-##< Config >
-
-##< Tokenizer >
+## Config
 
-##< Training data >
+## Tokenizer
 
-##< Training procedure >
+## Training data
 
-##< Preprocessing >
+## Training procedure
 
-##< Pretraining >
+## Preprocessing
 
-##< Evaluation results >
+## Pretraining
 
-##< Benchmarks >
+## Evaluation results
+## Benchmarks
 
 ### BibTeX entry and citation info