Modeling Selective Feature Attention for Representation-based Siamese Text Matching
Jianxiang Zang Hui Liu


The Directed Landscape is a Black Noise
    [
    April 29, 2024
=======================================




Representation-based Siamese networks have risen to popularity in lightweight text matching due to their low deployment and inference costs. While word-level attention mechanisms have been implemented within Siamese networks to improve performance, we propose Feature Attention (FA), a novel downstream block designed to enrich the modeling of dependencies among embedding features. Employing "squeeze-and-excitation" techniques, the FA block dynamically adjusts the emphasis on individual features, enabling the network to concentrate more on features that significantly contribute to the final classification. Building upon FA, we introduce a dynamic "selection" mechanism called Selective Feature Attention (SFA), which leverages a stacked BiGRU Inception structure. The SFA block facilitates multi-scale semantic extraction by traversing different stacked BiGRU layers, encouraging the network to selectively concentrate on semantic information and embedding features across varying levels of abstraction. Both the FA and SFA blocks offer a seamless integration capability with various Siamese networks, showcasing a plug-and-play characteristic. Experimental evaluations conducted across diverse text matching baselines and benchmarks underscore the indispensability of modeling feature attention and the superiority of the "selection" mechanism. [Codes available:https://github.com/hggzjx/SFA]







§ INTRODUCTION


The goal of the text matching task is to assess the semantic relevance between pairs of sentences and to determine their relationship. More specifically, it involves creating a classifier ξ that calculates the conditional probability P(label|s^a,s^b), thereby predicting the relationship between the sentence pair s^a and s^b. Here, label∈Ω represents different levels of sentence pair relationships, which can be {relevant, irrelevant} or {entailed, neutral, contradicted}. Representation-based Siamese networks <cit.> use dual encoders to compute text embeddings offline and aggregate them downstream for prediction. They offer the benefits of having low parameter sizes and reduced inference latency, making them extensively applicable in industrial contexts, including search engines and recommendation systems <cit.>. To enhance the post-interaction of text pairs, researchers have introduced various downstream attentions in Siamese networks <cit.>. 
Notably, these attention strategies solely capture word-level dependencies, neglecting the modeling of intricate relationships among embedding features. Each feature in text embeddings captures certain semantic or syntactic properties of the vocabulary, though these properties are usually not directly interpretable. For example, particular dimensions in the embedding vector might be related to parts of speech, contextual information (the relationship of a word to surrounding words), or other linguistic attributes.




 
The Word-level Interaction Attention shown in Figure <ref>(a) is the most commonly used downstream attention in Siamese matching networks <cit.>, and is a mapping (a,b)→(x,y), where a,b,x, y∈ℝ^L × D. Here, a,b represent the text embeddings of the text pair s^a,s^b. x,y represent the embeddings of the text pair containing rich word level interaction information. L denotes the length of the text sequence, and D represents the dimension of the embedding features. To enhance the sensitivity of the Siamese network to the embedding features then construct a more robust downstream attention, as illustrated in Figure <ref>(b), we advocate further building a single-branch symmetric Feature Attention (FA) based on the Word-level Interaction Attention. The FA block is a mapping that does not change the tensor size: x→u or y→v, where u, v∈ℝ^L × D. It is noteworthy that, despite the FA blocks in both branches sharing the same form, we advocate against sharing their parameters.

The FA block incorporates a "squeeze-and-excitation" approach, which concentrates on the most influential embedding features, enhancing their significance in the final classification.
Moreover, inspired by neuroscience, where the size of receptive fields in visual cortical neurons is modulated by external stimuli, we integrate a dynamic "selection" mechanism into Feature Attention based on the stacked BiGRU Inception structure. This results in the creation of Selective Feature Attention (SFA). The SFA block stimulates the network to dynamically adapt its focus on semantic information and embedding features across various levels of abstraction. Simultaneously, the "selection" within SFA effectively addresses the challenge of consistent gradient flow arising from the diverse scale semantic extraction in multi-branch Inception, achieving more efficient gradient flow management. 

The FA and SFA blocks preserve tensor shape in their mappings, capable of seamless integration with virtually any Siamese network, offering a plug-and-play characteristic. 
In the experimental evaluation, we combine the FA and SFA blocks with six of the most commonly used baseline Siamese networks from 2020 to 2023, assessing their performance across various text matching benchmarks. Extensive experiments demonstrate that the integration of SFA with all networks significantly improves inference accuracy across all text matching benchmarks.
Our primary contributions are highlighted in the following: (1)Based on our survey, we are the first to model dependencies at the embedding feature level for text matching. (2)We present the Feature Attention block and enhance it with a "selection" mechanism based on the stacked BiGRU Inception structure, resulting in the Selective Feature Attention. Extensive experiments confirm the superior performance of the "selection" in SFA block. (3)FA and SFA blocks are offering a plug-and-play characteristic, allowing them to be integrated with almost any Siamese network.     


















 

























































§ FEATURE ATTENTION




 §.§ Squeeze-and-Excitation Network




Channel-level attention mechanisms have demonstrated exceptional performance in image classification <cit.> and segmentation <cit.>. The Squeeze-and-Excitation Network (SE-Net) <cit.> pioneered channel attention by effectively constructing interdependencies among channels through the compression of each feature map. CBAM <cit.> further refined this concept by introducing spatial information encoding via convolutions with large-size kernels. Subsequent studies such as GENet <cit.>, GALA <cit.>, TA <cit.> expanded on this idea by adopting various spatial attention mechanisms or designing advanced attention blocks. While related work <cit.> models feature dependencies through higher-dimensional semantic spaces, we devised a squeeze-and-excitation style Feature Attention to model dependencies among semantic features.
 



 §.§ FA Block






As illustrated in Figure <ref>(b), FA block constitutes a 2D to 2D single-branch mapping that computes x→u or y→v. For the sake of simplicity, our discussion will focus solely on the computational mapping of the x branch. 
 
For the input x∈ℝ^L × D, to capture feature-level dependencies, we first execute a "squeeze" step using average pooling (ℱ_AP(·)) to compress global information into a feature descriptor s. Formally, s∈ℝ^1 × D is generated by averaging x along the spatial dimension L, where the d^th element of s is computed as formulated in Equation <ref>. Throughout this paper, ℱ(·) represents the mapping that does not involve trainable weights, while ℱ(·,W) represents the mapping that involves trainable weights W. Symbols with the subscript ·_l denote spatial (word-level) descriptors, while those with the subscript ·_d represent feature descriptors.



    s_d = 1/L∑_l=1^Lx_l  ,   d ∈ [1,...,D]



In the subsequent "excitation" step, aimed at enhancing the model's sensitivity to features, we filter out the embedding features that contribute more significantly to the final classification. For the aggregated information s from the previous steps, the "excitation" step is tasked with constructing a nonlinear, non-mutually exclusive gating mechanism. To ensure the excitation of multiple features, we have designed two fully connected layers for nonlinear mapping (ℱ_FC(δ)(·,W) and ℱ_FC(σ)(·,W)), namely a dimension-reducing layer with a Tanh function followed by a dimension-increasing layer with a Sigmoid function. Here, δ, σ represent the Tanh and Sigmoid function respectively, and s' ∈ℝ^1 ×D/r is a transitional vector. The decay factor r introduces a bottleneck in the network to control parameter redundancy. 





    e = σ(δ(sW_FC_1)W_FC_2)





The vector e∈ℝ^1 × D signifies the features that have been activated. Ultimately, this vector is merged with x through element-wise multiplication, enabling x to further develop into an embedding feature representation u = [u_1, u_2,..., u_D] ∈ℝ^L × D that is more finely attuned to the final classification.



    u_d = e_d*x_d  ,   d ∈ [1,...,D]




§ SELECTIVE FEATURE ATTENTION




 §.§ Inception Structure








In the visual cortex, neurons' ability to gather multi-scale spatial information within the same processing stage stems from the varying receptive field sizes in the same region <cit.>. The Inception structure <cit.> leverages this characteristic, achieving superior performance in computer vision by directly concatenating features extracted from multiple scales. However, this linear aggregation may be insufficient to model the neurons' robust adaptability. Furthermore, the uniform treatment of semantic extraction at different scales gradient flow management during training, affecting training stability.

Related research has shown that stimuli also influence neuronal responses <cit.>. The size of these receptive fields is not fixed but correlates with the stimulus contrast: lower contrast corresponds to a larger effective receptive field <cit.>. Selective Kernel Networks (SK-Net) <cit.> were the first to model this phenomenon in the field of computer vision, achieving significant success in image classification and semantic segmentation.
Motivated by this theory, we introduce the Selective Feature Attention, which encourages the network to dynamically adapt its focus on semantic information and embedding features across different levels of abstraction. 




 §.§ SFA Block




The SFA block comprises three phases: "split-and-fusion", "squeeze-and-excitation", and "selection", as illustrated in Figure <ref>. Initially, considering the potential complexity introduced by a multi-branch Inception structure, we advocate the incorporation of bottleneck structure at both ends of the SFA block for feature dimension scaling. Specifically, we employ a one-dimensional convolution kernel of size 1 as an auto encoder ℱ_AE(·,W) to map the size of the input x to ℝ^L ×D/r_1, where r_1 acts as a dimension reduction factor controlling the feature dimensionality of embeddings.




In the "split-and-fusion" phase, for x∈ℝ^L ×D/r_1, we introduce an N-layer stacked BiGRU (ℱ_S-BiGRU(·,W^(n))) to capture the semantic representation at each layer, effectively "splitting" the original embedding into vectors {x^(n)}_n=1^N, with x^(n)∈ℝ^L ×2D/r_1, as formulated in Equation <ref>. Here, h_l^(n) represents the hidden state at position l in the n^th layer of either the forward or backward GRU and <;> denotes concatenation along the feature dimension.



    x_l^(n) = <   GRU^(n)(h_l-1^(n), x^(n-1)_l,W^(n)_GRU);
       GRU^(n)(h_l+1^(n), x^(n-1)_l,W^(n)_GRU)>
       ,n ∈ [1,...,N],l ∈ [1,...,L]









The shallower layers of BiGRU excel at capturing short-range dependencies between words, such as understanding the combination of words in compound words or phrases. On the other hand, deeper layers of BiGRU are capable of processing and capturing long-range word dependencies. This includes discerning a sentence's theme, which may hinge on words at the beginning and end of the sentence or require consideration of the entire sentence's content for accurate interpretation.  The core idea behind "fusion" is to use a gating mechanism to enable information carrying different levels of semantic abstraction from multiple branches to flow towards the neurons of the next layer. To comprehensively and holistically preserve the semantic information of each branch, we employ stacked concatenation to amalgamate the results of all branches, as formulated in Equation <ref>. Here [;] denotes the stacked concatenation.



    x^(n)_l=[x_l^(1);x_l^(2);...;x_l^(n)],    n ∈ [1,...,N],l ∈ [1,...,L]





In the subsequent "squeeze-and-excitation" phase, for x∈ℝ^N × L ×2D/r_1, we recommend the combined use of global average pooling (ℱ_GAP(·)) and global max pooling (ℱ_GMP(·)) to  compress information simultaneously at both the BiGRU layer and word levels. As formulated in Equation <ref>, we obtain the sum s from the global average pooling and max pooling results, and then apply fully connected layers for activation.



    s_d = 1/N× L∑_n=1^N∑_l=1^Lx^(n)_l+n=1Nmaxl=1Lmax (x^(n)_l),   d ∈ [1,...,2D/r_1]






It is noteworthy that the essence of SFA block is to capture and excite features of text embeddings at different levels of abstraction, while adaptively adjust their relative importance. To accomplish this, we employ a single dimension-reducing fully connected layer (ℱ_FC(δ)(·,W)) alongside a series of dimension-increasing fully connected layers, with the count matching the number of branches (ℱ_FC(σ)(·,W^(n))) for excitation. This process results in the excited vectors {e^(n)}_n=1^N, e^(n)∈ℝ^1 ×2D/r_1, as formulated in Equation <ref>. Similarly, δ,σ represent the Tanh, Sigmoid function, respectively. The decay factor r_2 creates a bottleneck in the network to avoid parameter redundancy.



    e^(n) = σ(δ(sW_FC_1)W^(n)_FC_2))  ,   n∈[1,...,N]





In the most critical "selection" phase, we apply vector-level softmax normalization (ℱ_Norm(·)) to the excitation vectors {e^(n)}_n=1^N. The normalized results {e^(n)}_n=1^N serve as adaptive weights for the different branches, and are used for element-wise multiplication with each branch's representation {x^(n)}_n=1^N. The summation of these products yields the weighted sum representation of each branch u=[u_1,u_2...,u_2D/r_1] ∈ℝ^L ×2D/r_1.




    u_d =∑_n=1^Ne^(n)_d*x^(n)_d = ∑_n=1^Nexp(e^(n)_d)*x^(n)_d/∑_n=1^Nexp(e^(n)_d),d∈ [1,..., 2D/r_1]
     

      

 



Finally, to align the input and output dimensions of the SFA block, we also employ the same auto encoder (ℱ_AE(·)) to transform the shape of u to ℝ^L × D. To ensure the expressive power of the embedding feature, the feature dimension reduction caused by the bottleneck layer in the entire text must satisfy <cit.>:






    D/r,2D/r_1,2D/r_1r_2>8.33log L











 §.§ Efficient Gradient Management






The traditional Inception structure aggregates multi-scale information linearly.
However, this uniform updating of weights across different Inception layers impedes the differentiated flow of gradients, thereby impacting training stability. The distinct mapping branches ℱ^(n)_S-BiGRU(· ,W^(n)): x→x^(n) represent semantic extraction at various scales from the text embeddings, acknowledging that semantic extraction at different scales contributes differently to the final classification. In this section, we analyze the backpropagation within the SFA block during training, focusing on how the "selection" mechanism impacts training stability in the context of the gradient flow ∂u/∂x.

w/o selection Firstly, an SFA block without "selection" implies that there is only a single feature weight e, thus we have u = e * x. Consequently, the gradient flow chain is jointly determined by the gradient propagations of both e and x, specifically ∂u/∂x = e and ∂u/∂e = x. The gradient flow in the direction of e is formulated in Equation <ref>. Since the process involves a straightforward addition, it results in ∂s/∂ℱ_GMP(x) = 1, ∂s/∂ℱ_GAP(x) = 1.




    ∂u/∂x =  ∂u/∂e * ∂e/∂s' * ∂s'/∂s* (∂ℱ_GMP(x)/∂x+∂ℱ_GAP(x)/∂x)





Combining the backward propagation of gradients from x→x, the computation of the overall gradient flow is formulated in Equation <ref>. In this context, the transformation from x^(n)→x is a stacked concatenation, hence it follows that ∂x/∂x^(n) = 1. 




    ∂u/∂x=2*∂u/∂x*∑_n=1^N∂x/∂x^(n)*∂x^(n)/∂x=2*∂u/∂x*∑_n=1^N∂x^(n)/∂x





It can be observed that each semantic extraction mapping ℱ^(n)_S-BiGRU: x→x^(n) represents a gradient flow ∂x^(n)/∂x, which impacts the overall gradient flow ∂u/∂x in a uniform proportion of 2*∂u/∂x. This implies that the feature weights of each BiGRU layer are updated to the same degree. However, this uniformity does not align with the necessity for differentiated gradient flow management across various semantic scales, resulting in unstable training.



w/ selection The "selection" mechanism of the SFA block introduces an adaptive weight for each branch, leading to the entire gradient flow chain being jointly determined by the gradient propagations of both x^(n) and e^(n). This is expressed as ∂u/∂x^(n)=e^(n) and ∂u/∂e^(n)=x^(n). The gradient flow in the direction of e^(n) is formulated in Equation <ref>. Since the process involves a straightforward addition, it results in ∂u/∂e^(n) = 1, as formulated in Equation <ref>.


  

    ∂u/∂x =   ∑_n=1^N ( ∂u/∂e^(n) *∂e^(n)/∂e^(n) * ∂e^(n)/∂s' ) 
        
       * ∂s'/∂s * (∂ℱ_GMP(x)/∂x+∂ℱ_GAP(x)/∂x)





After backpropagating the gradients in Equation <ref>, a complete gradient chain is formulated in Equation <ref>. It is evident that, upon the introduction of a selection mechanism, the coefficient preceding each ∂x^(n)/∂x becomes a linear function of e^(n). This equation reflects how ∂x^(n)/∂x directly and, through the adaptive weights e^(n), indirectly influences the total gradient flow. In contrast to the scenario without the "selection", where ∂x^(n)/∂x affects the total gradient flow uniformly, the adaptive weights e^(n) manage the gradient flow across different branches in an adaptive and differentiated manner. This approach leads to a more robust training process, as reflected in Section <ref>.




    ∂u/∂x   =∑_n=1^N(∂u/∂x^(n)*∂x^(n)/∂x)+∂u/∂x*∑_n=1^N(∂x/∂x^(n)*∂x^(n)/∂x)
       =∑_n=1^N(e^(n)+∂u/∂x)*∂x^(n)/∂x







§ EXPERIMENTAL RESULTS & ANALYSIS










 §.§ Main Results


We selected the six most commonly used lightweight baselines in text matching tasks from 2020 to 2023. All these baselines reported parameter sizes in the articles with values less than ten million. The selected baselines are: BiMPM <cit.>, ESIM <cit.>, CAFE <cit.>, RE2 <cit.>, DIIN <cit.>, and DRCN <cit.>. In the experiments, we introduce FA block and SFA block to these baseline networks and evaluated their performance on following benchmarks: QQP <cit.>, MRPC <cit.>, BoolQ <cit.>, SNLI<cit.>, MNLI <cit.>(matched&mismatched), QNLI <cit.>, and Scitail <cit.>.

Table <ref> reports the evaluation accuracies of six lightweight text matching baselines, as well as their performances following the integration of FA and SFA blocks. Particularly, since DRCN is composed of multiple identical modules in series, we incorporated two FA and SFA blocks in each branch to align with the number of autoencoders present in DRCN. To minimize the impact of increased parameters on performance, we carefully controlled the values of r and N to maintain a consistent increment in parameters caused by the FA and SFA blocks. By regulating these hyperparameters, the additional parameters introduced by FA and SFA amounted to approximately 5%-10% of the network's original parameters. The sentence-level inference latency caused by the SFA block is higher than that of the FA block, due to the increased model complexity resulting from the stacked BiGRU. The incorporation of the FA block lead to overall improvements across all baselines, while the introduction of the SFA block significantly boost inference accuracy. The SFA block consistently demonstrate the best performance across all baselines compared to the FA block. It provides the most substantial average performance boost for the ESIM model (from 78.71% to 80.90%). With the integration of the SFA block, DRCN achieved the highest accuracy (81.87%) among all baselines, albeit at the cost of having the highest network parameters and inference latency.



Table <ref> reports a comprehensive comparison of six SFA-enhanced lightweight baselines against several high-parameter networks, (Residual Stacked <cit.>, LSTM-Max <cit.>, LM-Transformer <cit.>, BERT-base/large <cit.>, AlBERT <cit.>, and RoBERTa <cit.>). The comparison is based on evaluation accuracies, parameter sizes, and inference latencies across the QQP and SNLI benchmarks. In the QQP benchmark, it is observed that after integrating SFA, networks such as ESIM, CAFE, RE2, DIIN, and DRCN not only retain a substantial advantage in parameter volume and inference latency but also surpass the performance of pre-trained models like AlBERT (89.31%) and BERT (90.06%). Particularly noteworthy is RE2-SFA, which, with only 1.5% and 4.2% of the parameters and 5.4%, 12.3% of the inference latency, surpasses the accuracy of BERT-large (90.45%) and RoBERTa-base (90.73%) at 90.97%. In the SNLI benchmark, our networks with SFA outperform Residual Stacked, LSTM-Max, LM-Transformer, and AlBERT in terms of accuracy. Notably, DRCN, with just 6.9% and 2.2% of the parameter size and 36.2%, 10.3% of the inference latency, surpasses both BERT-base (90.16%) and BERT-large (90.82%). Despite having only 6.0% of the parameters and 23.2% of the inference latency, DRCN closely approaches the accuracy of RoBERTa-base (91.13%).















 §.§ Ablation Study


To investigate the key factors contributing to the superiority of the SFA block over the FA block, we conducted detailed ablation studies on the various components of the SFA block. As indicated in Table <ref>, ESIM and RE2 demonstrated the most significant average performance improvement following the integration of the SFA block. Thus, for a clearer representation of the results, we chose these two networks, along with QQP and SNLI, as the baselines and benchmarks for our ablation experiments. 

Figure <ref> illustrates the changes in prediction accuracy when components such as auto encoder, global max pooling, global average pooling, and the "selection" within the SFA block are individually removed. It is evident that the exclusion of each of these components resulted in a decrease in performance. Notably, the exclusion of the 'selection' step resulted in a substantial drop in performance, dipping below the outcomes achieved with the FA block. This impact was pronounced enough to cause ESIM's inference accuracy on QQP and SNLI to fall below that of their original baseline networks. Furthermore, the error bars indicate increased training instability after the removal of the "selection".



Figure <ref> illustrates the loss-epochs curves (dev.) for the ESIM model trained on two datasets. It is evident that the introduction of the FA and SFA blocks not only maintains or even improves training convergence speed but also enhances overall convergence performance. On the other hand, introducing a SFA block without the "selection" leads to numerous inefficient training processes. The loss exhibits little variation, significantly slowing down the convergence compared to previous models. As explained in Section <ref>, the omission of the "selection" yields uniform gradient updates across different Inception branches during feature extraction, leading to reduced training efficiency. This obstructed backpropagation process also hinders effective convergence during training, resulting in outcomes that may even be inferior to those of the base network.









 §.§ Inception Networks


Section <ref> corroborates that the "selection" is the pivotal element in the SFA block, which is built upon the foundation of multi-branch Inception networks. In this section, we discuss the effects of semantic multi-scale mapping using various Inception networks. These networks can be categorized into three types based on their fundamental architecture: CNN, RNN, and Transformer. For the CNN series, we investigate three forms: (1) CNNs with varying one-dimensional kernel sizes (K=2N+1) across different branches, (2) CNNs with a fixed kernel size of 3 but varying dilation factors (Dilation=N) across branches, and (3) CNNs with a fixed kernel size of 5 and varying dilation factors (Dilation=N). For the RNN series, we explore stacked BiRNN, stacked BiGRU, stacked BiLSTM. For the Transformer series, we examine three forms of stacked Transformer blocks with 1, 2, 3 attention heads, respectively. Specifically, we continue to use RE2 and ESIM as baselines and introduce SFA blocks with different Inception networks on top of them. Figure <ref> illustrates the average accuracies of the networks on the QQP and SNLI datasets. A significant increase in accuracy is observed when the number of branches changes from 1 to 2 in each Inception structure, further emphasizing the criticality of the "selection" based on multi-branch Inception. Compared to the RNN and Transformer series, SFA blocks built on the CNN series do not exhibit superior performance, possibly because RNN and Transformer architectures are suited for capturing sequential characteristics. When the branch amount reaches 3, the RNN and Transformer series generally achieve the highest inference accuracy.

Figure <ref> illustrates the increase in parameters and inference latency introduced by different Inception based SFA blocks, under the constraint of identical bottleneck factors (r_1, r_2). It is observed that at N=3, stacked BiLSTM and stacked Transformer blocks incur relatively high parameters and inference delay, contradicting the principle of lightweight text matching. In comparison, stacked BiRNN and stacked BiGRU strike a balance between performance and computational costs. SFA blocks based on stacked BiGRU exhibit superior and more stable accuracy compared to stacked BiRNN. This is the reason why we opted for a stacked BiGRU as the Inception network within the SFA block.







 §.§ Attention Analysis





The interaction of sentence embeddings directly influences the performance of matching networks. To visually illustrate the impact mechanism of FA and SFA blocks on the network, We selected two sets of sentence pairs from the MRPC, both of which have an 'irrelevant' relationship between them: "Robin Saunders, head of the bank's London-based principal finance unit, is also expected to quit." & "Robin Saunders, head of the principal finance unit, has made clear she has funding to buy parts of the business." and "In the second quarter, Anadarko now expects volume of 46 million BOE, down from 48 million BOE." & "Production for the second quarter was cut to 46 million barrels from 48 million barrels.".

We encoded the two sentence pairs using three types of trained ESIM (base, +FA, +SFA) respectively, and visualized the word-level dot product matrices of the base network x and y, as well as those of the network u and v with FA and SFA integrated. For visualization purposes, we performed feature-level average pooling, as illustrated in Figure <ref>. It can be observed that the base network only activates attention between synonyms, such as "head of," "principal finance unit", "second quarter", "46 million" and "48 million". When the FA block is introduced, this situation remains unchanged, as focusing only on synonyms does not make the network aware of their lack of relevance. In contrast, the introduction of the SFA block prompts the network to activate additional segments, such as "is also expected to quit" & "has funding to buy parts of the business" and "now expects" & "was cut to". These segments are essential for the network to determine the "irrelevant" relationship between sentences. This is because the FA block only adjusts the weights among individual features of each word, not affecting the values post-average pooling of all features (the weight of the sum of word-level features), and thus cannot directly influence word-level interactions. On the other hand, the SFA block aggregates embedding features across different scales with weighting. The mappings at these different scales are nonlinear, which leads to changes in the weight of the sum of word-level features. This directly activates words that were previously not focused on by the network, enabling the extraction of semantic information at a finer granularity and capturing the semantic focus.





§ CONCLUSION


In this paper, we introduce innovative attention modelling at the embedding feature level within lightweight text matching networks, presenting the FA and SFA blocks. The structures of the FA block and SFA block are concise, plug-and-play, and offer vast opportunities for expansion. In terms of structural design, beyond being a selection mechanism, the feature attention structure can support various forms, enabling finer-grained feature modeling. In terms of task format, feature attention is merely an activation of feature dependencies, unaffected by task formats. This indicates its applicability to various other semantic embedding-based tasks in NLP, such as text classification, entity recognition. We hope to encourage more researchers to explore diverse forms of feature-level attention across a broader range of NLP tasks, fostering a community ecosystem for feature attention in NLP.
























named


