Improving Semantic Parsing with Neural Generator-Reranker Architecture
Huseyin A. Inan Gaurav Singh Tomar Huapu Pan


Glass engineering to enhance Si solar cells: a case study of Pr^3+-Yb^3+ codoped tellurite-tungstate as spectral converter
    Marcos Paulo Belançon^a
    December 30, 2023
==========================================================================================================================




Semantic parsing is the problem of deriving machine interpretable meaning representations from natural language utterances. Neural models with encoder-decoder architectures have recently achieved substantial improvements over traditional methods. Although neural semantic parsers appear to have relatively high recall using large beam sizes, there is room for improvement with respect to one-best precision. In this work, we propose a generator-reranker architecture for semantic parsing. The generator produces a list of potential candidates and the reranker, which consists of a pre-processing step for the candidates followed by a novel critic network, reranks these candidates based on the similarity between each candidate and the input sentence. We show the advantages of this approach along with how it improves the parsing performance through extensive analysis. We experiment our model on three semantic parsing datasets (, , and ). The overall architecture achieves the state-of-the-art results in all three datasets. 





§ INTRODUCTION



Semantic parsing is the task of deriving machine interpretable meaning representations such as logical forms or structured queries from natural language utterances. These meaning representations can be executed in various environments, making semantic parsing applicable in many frameworks such as querying data/knowledge bases for question answering <cit.>, generating regular expression <cit.>, instruction following <cit.>, and communicating with robots <cit.>.

Conventionally, semantic parsing has been done with a two step approach: first, a large number of potential candidates are generated using deterministic methods and combinatorial search (the generator), and then the best candidate is selected among them with a probabilistic method or scoring (the critic) <cit.>. With recent advancement of neural networks, neural models with encoder-decoder architectures has obtained impressive improvements <cit.>. These encoder-decoder based neural semantic parsers produce one candidate given an input sentence, essentially acting as both the generator and the critic. However, undesirable prediction errors still occur, e.g. predicting a wrong comparative or superlative structure (such as < instead of ≤). On the other hand, neural semantic parsing models have a high recall, i.e. top-n predictions of the model cover the gold-standard meaning representation most of the time (c.f. Table <ref> in Section <ref>).

In this work we propose a generator-reranker architecture that uses two neural networks for semantic parsing: a generator network, which generates a list of potential candidates, and a reranker system, which consists of a pre-processing step for the candidates followed by a novel critic network that reranks these candidates based on the similarity between each candidate and the input sentence. An advantage of separating the semantic parsing process into a generator network and a critic network is that the critic observes each candidate and the input sentence entirely, taking into account bidirectional representations of both sentences and can globally reason over the entire candidate. This may be more effective in terms of choosing the right candidate and mitigating some of the errors arising from auto-regressive decoding in the generator. Furthermore, the critic can leverage extra data sources for training, e.g. a paraphrase dataset and allow for better transfer learning. Our key contributions in this work are the following:


  * We introduce a neural critic model, which reranks the candidates of a semantic parser based on their similarity score with respect to the input utterance.

  * We propose various pre-processing methods for the candidates to leverage the pre-trained representations of their tokens using the critic model.

  * We show through extensive qualitative and quantitative studies how the critic model helps mitigating errors of a state-of-the-art neural semantic parser and improves the performance. 




Evaluation results of our approach on three existing semantic parsing datasets (see Table <ref> for a sample
input-output pair for each dataset) show that our model improves upon the state-of-the-art results and the 
generator-reranker architecture can substantially improve parsing performance.




§ RELATED WORK



The semantic parsing problem has received significant attention and has a rich literature <cit.>. 
While traditional approaches  <cit.> rely on high-quality lexicons, manually-built templates, and/or domain or representation specific features, in more recent studies neural models with encoder-decoder architectures show impressing results
<cit.>.



Among recent results, <cit.> achieves the best performance on the  dataset <cit.>, which consists of 8 various domains. The model in <cit.> is an attention-based sequence-to-sequence model and it is trained jointly over 8 domains. Very recently, <cit.> presents an approach that uses a Graph Neural Network (GNN) architecture, which successfully incorporates information about relevant entities and their relations in parsing natural utterances. Similar to <cit.>, the decoder has a copying mechanism, which can copy an entity to the output during parsing. This model achieves the best performance in  dataset <cit.> and is competitive with state-of-the-art in  dataset <cit.>. We note that this model was not applied to the  dataset. 





We provide in Table <ref> the comparison between top-10 (25) oracle[The fraction of test examples where the gold-standard logical form is present in one of the top-10 (25) predicted logical forms.] and top-1 (greedy decoding) accuracy for the state-of-the-art sequence-to-sequence models. We note that while it may not be possible to reach top-10 (25) accuracy for all datasets[For instance, 
<cit.> reports 17% rate of incorrect labels in the  dataset.], there is certainly room for improvement.

Motivated by these observations, we consider a generator-reranker architecture for semantic parsing. The reranker system consists of a critic network, which reranks candidate logical forms based on their similarity scores to the input utterance. We propose various techniques for processing candidate logical forms before the critic scores their similarity to the input utterance. For the generator, we use the model in <cit.> and for the critic, we use the BERT model <cit.> for all three datasets. Although the setting is parser agnostic, we would like to investigate if the generator-reranker architecture can further improve the performance of the best performing parser. 



To the best of our knowledge, our reranker system is novel and has not been considered before in semantic parsing. However, the idea of reranking the candidates of a generator model appears in various applications in the literature. We refer the reader to <cit.> for reranking based on a combination of feature functions for various NLP tasks. 

For semantic parsing task, <cit.> applies reranking based on predicted answer type. We note that this method may not be effective for the cases where the candidate logical forms return the same answer type but the top prediction is wrong (such as predicting a wrong comparative, e.g. < instead of ≤). We believe choosing the best candidate by assessing its relevance to the input utterance entirely would be a more effective method. In a more recent work by <cit.>, reranking is applied by two main quality-measuring features of candidate logical forms. The first one is the reconstruction feature, using the probability of reproducing the original input utterance  from . The second is the discriminative matching feature, which is based on pair-wise associations of tokens in  and . In our work, reranking of candidate logical forms is applied based on their similarity with the input utterance directly using a critic model which can leverage the pre-trained representations of processed logical forms as well as extra data sources to learn similarity.







§ MODEL ARCHITECTURE


Our goal is to learn a model which maps an input utterance  to a logical form
representation of its meaning . The input utterance  is a sequence
of words _1, _2, …, _n_1∈𝒱^(in) where 𝒱^(in)
is the input vocabulary and the output logical form  is a sequence of tokens 
_1, _2, …, _n_2∈𝒱^(out) where 𝒱^(out)
is the output vocabulary.



In this work, we consider a combination of two models for semantic parsing. Figure <ref> illustrates the model architecture along with an example. We describe the model architecture in what follows.



 §.§ Generator



To generate candidate logical forms, we use the model recently introduced in <cit.>, which is based on the Transformer architecture <cit.>, with the self-attention layer extended to incorporate relations between input elements, and the decoder extended with a copy mechanism similar to <cit.>. The model is trained with natural utterances paired with logical forms and learns shared representations from these pairs. Candidate logical forms are generated using beam search for a given input utterance.



 §.§ Processing candidates



Before scoring the similarity between a logical form and an input utterance, we preprocess the candidate logical forms. We propose the processing methods in an increasing order of their complexity and how close they map the logical forms to natural text. Note that the critic can leverage the pretrained representations of more processed logical forms better and can be more effective when scoring similarities. Figure <ref> in Appendix <ref> illustrates these methods along with an example. 



  §.§.§ Raw Logical Form



In this method, we consider raw logical forms without any processing for calculating similarity with respect to the input utterance. This is the simplest method that is applicable to any dataset.



  §.§.§ Natural Language Entity Names



In this method, we simply convert entities to natural text. The output tokens are often self-explanatory and easy to be converted to simple text, e.g., “num_assists" → “number of assists", “en.location.greenberg_cafe" → “greenberg cafe" in  dataset, and “_arrival_time" → “arrival time" in  dataset etc. This approach requires an additional step but can be applied in a straightforward manner. The advantage of this approach is that the critic can leverage pre-trained representations for these tokens. We describe the exact procedure in Appendix <ref> for each dataset. 



  §.§.§ Templated Expansions



In this method, logical forms are converted to canonical utterances using a deterministic template  (e.g.  to
“player that has the largest number of rebounds"). The purpose of the canonical utterances is to capture the meaning of the logical forms. Here the assumption is that while it is nearly impossible to generate a grammar that parses all utterances, it is possible to write one that generates one canonical utterance for each logical form <cit.>. We use this method only for  dataset as there is an available template introduced in <cit.> to convert the logical forms to canonical utterances. In this case, the critic can be considered as a paraphrase model, producing a score based on the equivalence of two sentences semantically. During test time, candidate logical forms that do not have a corresponding canonical utterance are treated as incorrect candidates and they are not included in ranking.



 §.§ Critic



The critic network takes two sentences as an input and outputs a score s ∈ [0, 1] based on the similarity of the sentences. The input may consist of two arbitrary sentences (coming from different language models, vocabularies etc.) depending on the processing of the logical forms described in the previous section. We use the critic to rerank the candidate logical forms based on their similarity score with respect to the input utterance. We use the BERT model <cit.> for this task[The logical forms are tokenized using the public BERT wordpiece tokenizer as wordpieces often contain useful information. Different tokenization methods can be considered, which we leave for future work.]. 





To train the critic, positive and negative examples are generated as follows. Each natural utterance in the training set has a gold-standard logical form, which forms the positive examples. For the negative examples we apply the generator, which is trained over the training set, to generate logical forms for each natural utterance in the training set with beam search. The incorrect logical forms are paired with the natural utterance for negative examples. Additionally, we pair any two logical forms among beam candidates as negative examples to increase training data and to help the model learn subtle differences among the candidates. The logical forms are processed according to the method we use in the model. The positive pairs are labeled as 1 and the negative pairs are labeled as 0. The model performs binary classification. Figure <ref> illustrates this approach with an example.

As aforementioned, we can leverage existing paraphrase datasets to pretrain the critic and fine tune it over the examples we generate. For instance, we use the Quora question pairs[See <https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs>.], which contains over 400K annotated question pairs with binary paraphrase labels. Furthermore, we may choose not to apply reranking in certain cases. From the error cases of the critic over the training examples, two cases stand out. The first one is where each candidate is scored below 0.5 (no candidate is similar to the input utterance according to the critic) and the second one is the case where at least two candidates are scored very high and close to each other. It appears quite natural not to do ranking if all scores are below 0.5 to address the first case. On the other hand, one can address the second case by setting a threshold between the best and second best score and choose not to do ranking if the difference is less than this threshold. However, setting this threshold is prone to overfitting and may look somewhat arbitrary if chosen by maximizing the accuracy over the evaluation set. Therefore, we set this threshold as 0.001 once and keep it same for all experiments. Our goal here is merely to show various strategies one can take with the critic model. We may choose not to rerank if either or both of these cases occur at inference and instead output the top prediction of the generator.




§ EXPERIMENTS




 §.§ Dataset


We use three semantic parsing datasets in our experiments. Table <ref> presents an example for each dataset.

GeoQuery () contains natural language questions about US geography along with corresponding logical forms <cit.>. We follow <cit.> and use 600 training examples and 280 test examples. We use logical forms based on Functional Query Language (FunQL) <cit.>. We use logical form exact match when reporting accuracy.

ATIS () contains natural language queries for a flights database along with corresponding database queries written in lambda calculus <cit.>. We follow <cit.> and use 4473 training examples and 448 test examples. We compare normalized logical forms using canonical variable naming and sorting for unordered arguments <cit.> when reporing accuracy.

Overnight () contains 13,682 examples of language utterances paired with logical forms across eight domains <cit.>. In this dataset, each logical form has a corresponding canonical utterance, which we use in templated expansions method when we process the candidate logical forms to canonical utterances. We evaluate on the same train/test split as <cit.> with the same accuracy metric, i.e. the fraction of test examples for which the denotations of the predicted and gold logical forms are equal. 



 §.§ Training Details


For the generator, we follow the settings of <cit.> for  and  datasets. The model was not applied to  dataset, therefore, we configured the hyperparameters based on performance cross-validated on the training set. We provide the final setting and training process in full detail in Appendix <ref>. 

For the critic, we use the BERT_LARGE model in <cit.>, where we feed two sentences as an input and feed the  representation into an output layer for binary classification. We either directly train the model with the examples generated by the generator (see Section <ref>) or train the model on Quora question pairs first and fine tune it over our examples. We apply the early stopping rule based on the evaluation set accuracy to determine the total training steps. We use the learning rate 1e-6 and batch size 32 when we train and fine tune the critic.



 §.§ Results








  
: We compare our model with the state-of-the-art models in Table <ref>. All models use the same training/test splits, therefore, we directly take the reported best performances from their original papers for fair comparison. We use the model in <cit.> for the generator. As Table <ref> shows, this model alone (without any reranking) improves the state-of-the-art performance from 79.6% to 82.15% accuracy and sets a new state-of-the-art as a sequence-to-sequence model. In this experiment, we use templated expansions method (Section <ref>) when processing candidate logical forms for reranking. We do our experiments over the generator-reranker architecture on three settings: 


  * Beam size - This determines the number of candidates produced by the generator.

  * Initialization of the critic - This determines how the critic is initialized. We use a pre-trained model (pQ), which is trained over Quora question pairs and fine tuned over the generated examples by the generator. We compare this with directly training the critic over the generated examples by the generator without Quora dataset.

  * Reranking with a threshold rule - We apply reranking based on various threshold rules. We compare reranking at all times with reranking when there is at least one candidate with score above 0.5 (TH1) and reranking if the difference between best score and second best score is above 0.001 (TH2).

From the overall results, we can see that:


  * Increasing the beam size improves the performance as expected. As the generator outputs more candidates, it is more likely that the correct form is among them. Therefore, this allows the critic to identify a higher number of true positives and improve the performance. Increasing the beam size further does not significantly improve the performance (beam size 50 achieves 83.80% accuracy), hence we conclude the experiment with beam size 25.

  * Using a pre-trained model improves the performance as well. We note that the task of the critic is to infer the similarity of two input sentences, therefore, we can initialize it with one that has been trained over an existing paraphrase dataset.

  * Reranking with a threshold rule may be helpful for the overall architecture. We observe that reranking by the critic at all times may not be the best approach. We note that choosing not to rerank when all scores are below 0.5 increases the performance further. On the other hand, reranking if the difference between the best score and second best score is above the threshold we set does not help in this case.


The overall architecture improves the performance of the generator (82.1% accuracy) to 83.7% accuracy. We note that this improvement is significant for  dataset as it is an average over 8 domains with an improvement for each one of them.















We next apply raw logical form (Section <ref>) and natural language entity names (Section <ref>) methods when processing the candidate logical forms and show that the critic improves the performance in these cases as well. While our best result with templated expansions (Section <ref>) method achieves 83.71% accuracy in Table <ref>, the best results we achieve with the first two methods are 82.81% and 83.16% respectively. We note that the critic improves upon the performance of the generator (82.1% accuracy) in all three cases. As the logical forms are processed more towards natural text, the performance gets better. This is expected since it helps the critic to measure the similarity with respect to the input utterance. 









































  
 and : We continue with  and  datasets and provide the best set of our results.

For the  dataset, we set the beam size as 10 for the number of candidates produced by the generator. We do not process the candidate logical forms and use the raw versions as the output tokens are already in processed form. We pretrain the critic over the Quora dataset and fine tune it over the generated examples
by the generator on the training set. We produce 25 candidate logical forms for each example in the training set when generating examples to train the critic.

For the  dataset, we set the beam size as 10 for the number of candidates produced by the generator. We process the candidate logical forms with natural language entity names method in a straightforward manner (see Appendix <ref>). We pretrain the critic over the Quora dataset and fine tune it over the generated examples by the generator on the training set. We produce 25 candidate logical forms for each example in the training set when generating examples to train the critic.

The results are shown in Table <ref>. We observe a performance gain in both datasets and achieve the state-of-the-art performance with the generator-reranker architecture. We note that in 
 dataset, there is a significant improvement upon the baseline and the overall architecture obtains the state-of-the-art result over <cit.>, where the approach is not based on neural models.





 §.§ Error Analysis


In this section, we categorize the types of errors the generator model makes and analyze which ones are corrected
by the critic to understand what types of errors can be mitigated via our approach. We use the  dataset as it has a much larger test set. In the following examples,
the top prediction of the generator is wrong, but the gold-standard logical form is among the beam candidates and correctly scored highest by the critic. We provide the corresponding canonical utterances instead of the logical forms for the sake of presentation.



  * Wrong comparative structure: 

Input utterance: meetings after january 2 or after january 3

Top prediction: meeting whose date is  at least jan 2 or jan 3 

Corrected to: meeting whose date is  larger than jan 2 or jan 3


  * NP-shift to wrong position: 





Input utterance: what article citing multivariate data analysis was in annals of statistics 

Top prediction: article whose venue is annals of statistics and  that multivariate data analysis cites 

Corrected to: article whose venue is annals of statistics and  that cites multivariate data analysis


  * Similar word confusion in elliptical constructions: 

Input utterance: show me recipes with a preparation time equal to or greater than rice pudding 

Top prediction: recipe whose preparation time is at least  cooking time of rice pudding 

Corrected to: recipe whose preparation time is at least  preparation time of rice pudding


  * Incorrect matching to semantically non-equivalent phrase: 

Input utterance: housing that is cheaper than 123 sesame street 

    Top prediction: housing unit whose  size is smaller than size of 123 sesame street 

Corrected to: housing unit whose  monthly rent is smaller than monthly rent of 123 sesame street


Although the critic helps fixing these errors and improves the performance in semantic parsing, there are few cases where it is ineffective. Therefore, a stronger critic may further increase the performance. We provide here the cases where the top prediction of the generator is correct, but the critic scores another prediction higher and this leads to an error. 



  * Comparative by trivial algebra: 

Input utterance: friends of people who joined their jobs before 2005 

Top prediction: person that employee whose start date is  at most 2004 is friends with 

Best scored candidate: person that employee whose start date is  smaller than 2004 is friends with 


  * Comparative by common sense: 

Input utterance: who is younger than alice 

Top prediction: person whose birthdate is  larger than birthdate of alice 

Best scored candidate: person whose birthdate is  smaller than birthdate of alice




§ CONCLUSION

In this paper, we proposed a generator-reranker architecture for semantic parsing. We introduced a novel neural critic model that reranks the candidates of a semantic parser based on their similarity scores with respect to the input utterance. We proposed various processing methods for the candidate logical forms, enabling the critic to leverage the pre-trained representations for the tokens of the candidates. Our architecture further enables leveraging extra data resources in a direct fashion. We showed that the proposed architecture improves the parsing performance and achieves the state-of-the-art results on three semantic parsing datasets.  


As the model is generator agnostic, in the future work we plan to try (a combination of) neural and/or traditional parsers as the generator and apply our architecture on more benchmark datasets. In addition, we believe that our architecture may also be effective in the cross-lingual semantic parsing setting <cit.>. One can leverage paraphrase datasets available in source-target language and pretrain the critic, which could help choosing the right form among the generator candidates. 





iclr2020_conference





§ PROCESSING CANDIDATES







 §.§ Natural Language Entity Names



We apply this method to  and  datasets as the  dataset does not require processing since the output tokens already has English words. 

For the  dataset, we apply a very simple procedure. We take the tokens that start with an underscore (i.e. “_") and remove it (e.g. “_from" → “from", “_to" → “to" etc.). Additionally, for such tokens, if there are underscores in between, we replace them with space to break into further tokens (e.g. “_departure_time" → “departure time", “_fare_amount" → “fare amount").  

For the  dataset, each entity token has a corresponding English text available in <cit.>, which is used in the process of building the dataset. These are used within a set of rules for the logical form, canonical utterance conversion as well. We note that this method is rather straightforward compared to templated expansions method since we only replace each entity token with its corresponding English text.



§ TRAINING DETAILS OF THE GENERATOR ON  DATASET


The final setting has 2 encoder and 2 decoder layers. The source embedding and hidden dimensions are chosen as 64 and the target embedding and hidden dimensions are chosen as 128. We set the corresponding feed forward layer hidden dimensions 4 times higher. We employed dropout at training time with probability 0.2. We used 8 attention heads for each task. We used a clipping distance of 8 for relative position representations <cit.>. 

We used the Adam optimizer <cit.> with β_1 = 0.9, β_2 = 0.98, and ϵ = 1e-9. We set the learning rate as 5e-5. We used the same warmup and decay strategy for learning rate as <cit.>, selecting the warmup steps as 4000. We trained the model for 50000 steps with batch size 32. We used a simple strategy of splitting each input utterance on spaces to generate a sequence of tokens. We mapped any token that did not occur at least 2 times in the training dataset to a special out-of-vocabulary token. Token embeddings are taken from a pre-trained BERT <cit.> encoder. We freeze the pre-trained parameters of BERT for 6000 steps, and then jointly fine tune all parameters, similar to existing approaches for gradual unfreezing <cit.>. When unfreezing the pre-trained parameters, we restart the learning rate schedule.

Motivated by the best performing approach in <cit.>, the generator is trained using the natural utterance, logical form pairs over all training examples, i.e., jointly over all domains. Therefore, all model parameters are shared across domains and the model is trained from all examples. Since there is no explicit representation of the domain that is being decoded, to help the model learn identifying the domain given only the input, we add an artificial token at the beginning of each source sentence to specify the target domain, similar to <cit.> for neural machine translation. 

