abstract: Personalized chatbots focus on endowing chatbots with a consistent personality to behave like real users, give more informative responses, and further act as personal assistants. Existing personalized approaches tried to incorporate several text descriptions as explicit user profiles. However, the acquisition of such explicit profiles is expensive and time-consuming, thus being impractical for large-scale real-world applications. Moreover, the restricted predefined profile neglects the language behavior of a real user and cannot be automatically updated together with the change of user interests. In this paper, we propose to learn implicit user profiles automatically from large-scale user dialogue history for building personalized chatbots. Specifically, leveraging the benefits of Transformer on language understanding, we train a personalized language model to construct a general user profile from the user's historical responses. To highlight the relevant historical responses to the input post, we further establish a key-value memory network of historical post-response pairs, and build a dynamic post-aware user profile. The dynamic profile mainly describes what and how the user has responded to similar posts in history. To explicitly utilize users' frequently used words, we design a personalized decoder to fuse two decoding strategies, including generating a word from the generic vocabulary and copying one word from the user's personalized vocabulary. Experiments on two real-world datasets show the significant improvement of our model compared with existing methods. Our code is available at <https://github.com/zhengyima/DHAP>
author: Zhengyi Ma$^{2,3}$, Zhicheng Dou$^{1,3}$, Yutao Zhu$^{5}$, Hanxun Zhong$^{2,3}$, and Ji-Rong Wen$^{3,4}$
title: One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles

# Introduction

Faced with extensive information available on the Internet, it is very appealing to have an intelligent assistant that can provide the most relevant information , collaborate with us on day-to-day problems , or even act as our agent for some specific tasks . Towards this ultimate goal, in the dialogue system area, building digital agents has attracted more and more attention and had some preliminary applications in our daily life . In the future, some chit-chat between humans will inevitably be completed by digital agents. In this paper, we carry out a preliminary study toward this goal and focus on the problem of developing personalized chatbots. A personalized chatbot aims at leveraging personalized information (*e.g.*, a predefined persona ) to provide personalized responses when communicating with others. Such personalized information can help chatbots generate more consistent and informative replies. More importantly, if the personalized information can be well captured, the chatbots can perform similar behaviors like real users (*e.g.*, serving as an agent of the user and give similar responses to others when the user is busy, as shown in Figure ), thus having the potential to be an intelligent agent with that specific personality .

Many models have been proposed for improving the chatbot's capability to generate personalized responses. Early studies tried to integrate the user ID embeddings to a sequence-to-sequence (Seq2Seq) model for identifying the user and generating user-related responses . Recently, some studies proposed to assign predefined personas to chatbots so as to generate more personalized responses . They assumed that personality can be described in several sentences or attributes and help chatbots generate more persona-related responses. Different from existing studies, in this work, we propose letting the chatbot **learn the implicit user profile automatically from the user dialogue history** and **generate personalized responses based on the learned user profile**. In this way, the chatbot can be personalized by user's (*e.g.*, user A) historical data, behave like this user, act as this user's agent, and chat with any other users (*e.g.*, user B, C, etc.).

Our idea is motivated by: (1) Contrary to explicit persona descriptions, user dialogue history is easier to be gathered on the user's client devices. It is evident that obtaining explicit descriptions for massive users is impractical in real applications : First, users may be lazy to set their profile before using the chatbot . Second, manually collecting user profiles is costly and time-consuming. Furthermore, even if the user profile is collected, it cannot be updated with the change of user interests, thus may be ineffective over time. Finally, a fixed set of properties is not suitable for describing all users. (2) The user dialogue history contains massive personalized information, which is suitable for learning user profiles automatically. As shown in Figure , the dialogue history of a user includes their historical responses, and the corresponding posts issued by other users. Intuitively, users' historical responses can often reflect their language style, background knowledge, frequently used words, and even their interests. For example, there may be many electronic device names that appeared in the historical responses of an electronic hobbyist (*e.g.*, *MacBook* in Figure ). Besides, the interaction content and style between a specific user and others can be captured from the historical post-response pairs. When faced with a new input post, the chatbot can look for the historical data, check how the user has responded to a similar post before, and apply similar interactions to generate a suitable response. In addition, user profiles learned from historical data can be gradually updated with more data being collected. In summary, the user dialogue history is easy to obtain and appropriate for building user profile.

To achieve our idea, we propose a model **DHAP** for personalized chatbots, which focuses on learning implicit user profile from user **D**ialogue **H**istory **A**utomatically and generating **P**ersonalized responses. In our model, a general user profile representation is **firstly** constructed from the user's historical responses to capture the general information, including user interest, background knowledge, and speaking style. This is implemented by a personalized language model based on Transformer. **Then**, we design a personalized post encoder to construct the personalized post representation. The general user profile is utilized in the post encoder to better capture the semantic information of the input post. **Next**, we build a key-value memory network to store the user's historical post-response pairs. Based on this history memory, the dynamic post-aware user profile is built by highlighting the historical responses relevant to the current input post. **Finally**, we design a personalized decoder to fuse the learned user profile into the response generation process. The personalized decoder can switch between generating a word from a generic vocabulary and copying a word from the user's personalized vocabulary. Experimental results on two large-scale datasets show that our proposed DHAP significantly outperforms existing response generation models in various evaluation metrics.

Our contributions are three-fold: (1) We learn the implicit user profile from user's dialogue history automatically for generating personalized responses. By this means, our method can be applied without additional annotations on user profiles, and create personalized chatbots as user's digital agents. (2) We build two kinds of user profiles from the dialogue history, including the general user profile reflecting the user's general information and the dynamic post-aware user profile to apply similar interactions in the historical data for the current input. (3) We design a personalized decoder to coordinate two personalized decoding strategies, including generating a word from the generic vocabulary and copying a word from the personalized vocabulary to leverage the user's word preference.

# Related Work

#### Open-domain Chatbots and Response Generation

Open-domain chatbots have attracted more and more attention, due to their broad application in real applications, such as Microsoft XiaoIce . Typical methods can be categorized into two groups: retrieval-based and generation-based. Retrieval-based methods aim to select a suitable response from a large repository , while generation-based methods aim at generating a response from scratch . In this study, we focus on the response generation problem.

Some early studies treat the response generation task as a statistical machine translation problem because of its end-to-end and data-driven features . More recently, with the progress of deep learning, Seq2Seq methods have been applied to response generation and achieve great performance . Many Seq2Seq-based extensions have been applied to tackle the "safe response" problem ; to incorporate external knowledge ; to generate responses with emotions or personas ; and to model the hierarchical structure of the dialogue context .

#### Personalized Chatbots

Endowing chatbots with a coherent personality is a challenging but necessary step to achieve the ultimate goal of building an intelligent assistant. With personality, a chatbot can generate more informative and user-specific responses , and has the potential to perform similar behaviors as real humans.

Traditional personalized chatbots focus on modeling the user's psychological behavior such as the "Big Five" of speakers . In recent years, deep-learning based methods were proposed to learn the persona information directly from large-scale dialogue datasets via end-to-end neural networks . Some researchers first tried to input the user ID embeddings into the decoder of a Seq2Seq model to generate more personalized responses . Despite users can be identified by their IDs, the personalized performance is limited because no user-related information is used in the model. Therefore, another group of researchers proposed assigning explicit profiles (personas) for chatbots to generate personalized responses. For example, published the PERSONA-CHAT dataset, in which each user is assigned with several persona description sentences and the conversations between users are collected. With explicit profile descriptions, chatbots can learn to generate more consistent and informative dialogues. On this dataset, many methods have achieved encouraging performance, such as variational autoencoders , pre-trained language models , and multi-task modeling . These explicit persona-based methods enjoy the high quality of predefined personas, but the acquisition of these persona data is expensive and even impossible when applied to real-world systems .

In this work, we propose using the *implicit user profile* to drive the response generation. Such user profile is implicitly learned with neural models from user dialogue history rather than predefined explicitly. Due to the fact that the user dialogue history contains abundant user-specific information and is easily accessible in real-world applications, our method is more applicable in practice.

# Methodology

In this section, we first provide an overview of our proposed model DHAP. The details of each component are provided later, and the model training is introduced finally.

## The Overview of DHAP

Suppose that for a user $u$, we have their dialogue history $H$ including a series of responses issued by $u$ and the corresponding posts: $H = ((P_1,R_1),\cdots,(P_n,R_n))$, where $n$ is the number of historical post-response pairs. Note that the posts $(P_1,\cdots,P_n)$ here can be issued by different users, but the responses $(R_1,\cdots,R_n)$ are all issued by the same user $u$. We call them *historical posts* and *historical responses* respectively in the following sections. Under the single-turn setting, given an input post $X=(x_1,\cdots,x_{L_X})$ and the user dialogue history $H$, with sequence-to-sequence modeling, our task is to generate a personalized response $Y=(y_1,\cdots,y_{L_Y})$ as: $$\begin{aligned}
    p\left(Y|X, u \right) =  p\left(Y|X,H \right) = \prod_{t=1}^{L_Y} p\left( y_t| y_{<t}, X, H  \right),
\end{aligned}$$ where $y_t$ denotes the word generated at the $t$-th step, and $y_{<t}$ denotes the previous generated words $(y_1,\cdots,y_{t-1})$. It is worth noting that the dialogue history $H$ here includes dialogues between the user $u$ and several other users, thus the history is not a multi-turn dialogue between two fixed interlocutors.

To compute the probability $p(Y|X,H)$, we design a model called DHAP, which stands for learning user profiles from Dialogue History Automatically for Personalized chatbots. The structure of DHAP is shown in Figure . We briefly introduce the key components of DHAP as follows. The number of the modules corresponds to the mark in the figure. In general, DHAP considers personalized information in both the encoder and the decoder side.

### Encoder

DHAP has two different encoders, which encode the input post and user's historical responses, respectively:

**Part (1): History encoder and general user profile**. Since abundant personalized information (*e.g.*, background knowledge and speaking style) is often hidden in user's historical responses, DHAP firstly establishes a Transformer-based personalized language model to encode historical responses $(R_1,\cdots,R_n)$, then summarizes the general user profile $\mathbf{e}^G$ based on the historical responses. As $\mathbf{e}^G$ does not depend on the input $X$, we call it *general user profile*. "General" does not mean it is general among all users or cannot be updated together with time. In contrast, every user has their own profile, which can be updated once they issue a new response.

**Part (2): Personalized post encoder**. DHAP also has an encoder for the input post, which is implemented by a bidirectional GRU (BiGRU). To make the post encoder aware of the user's personalized information, we use the general user profile $\mathbf{e}^G$ for initialization. Consequently, the post $X=(x_1,\cdots,x_{L_X})$ is represented by hidden state sequence $\mathbf{h}^P=(\mathbf{h}^P_1,\cdots,\mathbf{h}^P_{L_X})$. These representations will be dynamically aggregated as a personalized post representation $\mathbf{c}_t$ by an attention mechanism at the decoding step $t$.

### Decoder

In the decoder side, DHAP incorporates the personalized information in two perspectives:

**Part (3): User history memory and dynamic post-aware user profile**. In the general user profile, all historical responses are considered in a general view. However, for a specific input post $X$, the historical responses may play different roles. Intuitively, the historical responses relevant to $X$ are more valuable than the irrelevant ones. To highlight the historical responses relevant to the input post, DHAP builds a key-value memory of historical post-response pairs $((P_1,R_1), \cdots,(P_n,R_n))$. Given $\mathbf{c}_t$ as a personalized representation of $X$, DHAP builds a dynamic post-aware user profile $\mathbf{e}^D_t$ by selecting and aggregating the historical responses from the user history memory. As $\mathbf{e}^D_t$ is dynamic with different input posts $X$ and summarizes related user history information, we call it *dynamic post-aware user profile*.

**Part (4): Personalized decoder**. Finally, the personalized post representation $\mathbf{c}_t$, general user profile $\mathbf{e}^G$, and dynamic post-aware user profile $\mathbf{e}^D_t$ are fused together to decode the response sequentially. Inspired by CopyNet , DHAP can switch between generating a word from a generic vocabulary ($p(y_t|m_g)$) and copying a word from a user's personalized vocabulary ($p(y_t|m_c)$) as: $$\begin{aligned}
    p( y_t| y_{<t}, X, H  ) &= p(m_g)p(y_t|m_g) + p(m_c)p(y_t|m_c),\label{eq_decode_prob}
\end{aligned}$$ where $p(m_g)$ and $p(m_c)$ are computed by our designed decoding switcher. $p(y_t|m_g)$ and $p(y_t|m_c)$ are calculated based on same inputs(*e.g.*, $\mathbf{c}_t$, $\mathbf{e}^G$ and $\mathbf{e}^D_t$) with different functions.

In the remaining part of this section, we will introduce these four components of DHAP in detail.

## History Encoder and General User Profile

Based on our observation, there is a large amount of personalized information in the user's historical responses. For example, a fan of cricket may talk a lot about cricket topics with others. Furthermore, different users can hold different speaking styles, such as enjoying speaking slang. Therefore, our first idea is to devise a model for learning such personalized language information in a general view.

Inspired by the strong ability of Transformer  to aggregate context and model sequences, we use a Transformer encoder to learn contextual representations of historical responses. In particular, we first add special tokens and concatenates all $n$ historical responses as $R^{w} = [\langle{\rm CLS}\rangle;R_1;\langle{\rm SEP}\rangle;\cdots;R_n;\langle{\rm SEP}\rangle]$, where each response $R_i=(r^i_1,\cdots,r^i_{L_{R_i}})$ contains $L_{R_i}$ words, and \[;\] is the concatenation operation. A "$\langle{\rm SEP}\rangle$" token is added at the tail of each response for segmentation, while a "$\langle{\rm CLS}\rangle$" token is added at the sequence head for summary. Then we map all words and special tokens into embeddings, and learn their contextual representations: $$\begin{aligned}
    \left[\mathbf{e}^G;\mathbf{E}^R\right] &= {\rm Transformer}_{N}\left( \left[\mathbf{e}_{{\rm CLS}};\mathbf{R}_1;\mathbf{e}_{{\rm SEP}};\cdots;\mathbf{R}_n;\mathbf{e}_{{\rm SEP}}\right]\right) \label{transformer_response}, \\
    \mathbf{R}_i &= [\mathbf{e}^{r_i,{\rm fuse}}_1; \cdots; \mathbf{e}^{r_i,{\rm fuse}}_{L_{R_i}}],
    % \be^{r_i,{\rm fuse}}_j &= \be^{r_i,{\rm tok}}_j + \be^{r_i,{\rm seg}}_j + \be^{r_i,{\rm pos}}_j,
    % \left[E^{R}; e^{M}\right] = {\rm Trm}\left( \left[R^{w}, \left[{\rm MASK}\right]\right] +  {\left[R^{w}, \left[{\rm MASK}\right]\right]}_{p} \right)\label{transformer_response}
\end{aligned}$$ where $\mathbf{e}^{r_i,{\rm fuse}}_j$ is the sum of the word embedding, segment embedding, and position embedding of the $j$-th word. These three embeddings are used in a similar way like BERT . The representation of "$\langle {\rm CLS}\rangle$" token ($\mathbf{e}^G$) summarizes the information in the entire historical responses, thus we call it the general user profile. $\mathbf{E}^R$ contains contextual representations of words in historical responses. For $R_i$, we denote its contextual representations as $(\mathbf{e}^{r_i}_1,\cdots,\mathbf{e}^{r_i}_{L_{R_i}})$. ${\rm Transformer}_{N}$($\cdot$) is a $N$-layer bidirectional Transformer encoder identical to the original implementation described in .

With the history encoder, we obtain: (1) a general user profile $\mathbf{e}^G$, which summarizes all historical responses and contains personalized information of the user; and (2) the contextual representations of words in the historical responses $\mathbf{E}^R$. These representations will be used to build the dynamic post-aware user profile in Section .

## Personalized Post Encoder

In real-world applications, the posts are often very short, even ambiguous. Thus, building accurate encoding of the input post is difficult, which further leads to poor quality of the generated responses . Fortunately, with personalized background knowledge, the chatbot can get more input information and is promising to better capture the semantic information of the post. Let us use an example to explain this: given a post "The new MAC is so beautiful", different users may have different understandings. For a programmer, "MAC" may refer to the Apple's laptop; but for a fashion girl, she may associate "MAC" with the lipstick. Therefore, the user's history can help distinguish the word "MAC" and provide more background knowledge for the input post, so that the post representation can be significantly enhanced. We call this encoder "personalized post encoder" - the encoded representations of the same post can be different for various users, because these users may have different profiles and different understandings of the same post.

Specifically, DHAP employs a BiGRU to encode the input post. Here we choose RNN-based architectures because they are better at capturing local correlations and encoding positional information than Transformers for short texts . To facilitate the encoder with personalized information, DHAP uses the general user profile $\mathbf{e}^G$ to initialize hidden states of BiGRU. Given the post $X=(x_1,\cdots,x_{L_X})$, its representations $(\mathbf{h}^P_1,\cdots,\mathbf{h}^P_{L_X})$ are built as: $$\begin{gathered}
    \mathbf{h}_i^P = {\rm BiGRU}(\mathbf{h}_{i-1}^P, \mathbf{x}_i),\quad \mathbf{h}_0^P = {\rm ReLU}({\rm MLP}(\mathbf{e}^G)),
     \label{postencoder}
\end{gathered}$$ where $\mathbf{x}_i$ is the embedding obtained by the embedding table.

These hidden states draw personalized information from the general user profile, and in the decoding phase, they are aggregated by the decoding hidden state $\mathbf{h}_t^R$ through an attention mechanism: $$\begin{gathered}
    \mathbf{c}_t = \sum_{i=1}^{L_X}\alpha_{t,i}\mathbf{h}_i^{p}\label{attention}, \\
    \alpha_{t,i} = \frac{{\exp}\left(s_{t,i}^{rp}\right)}{\sum_{j=1}^{L_X} {\exp}(s_{t,j}^{rp})},\quad s_{t,i}^{rp} = \mathbf{v}^{\top}_{rp} {\tanh}\left({\rm MLP}\left(\left[\mathbf{h}_t^R;\mathbf{h}_i^P\right]\right)\right). \label{atten}
\end{gathered}$$ The detailed calculation and updating scheme of the decoding state $\mathbf{h}_t^R$ at time step $t$ will be described in Section . Based on the attention mechanism, the personalized post representation $\mathbf{c}_t$ can dynamically focus on some important words of the post according to the current decoding state. In the next section, DHAP will use $\mathbf{c}_t$ to build the dynamic post-aware user profile.

## User History Memory and Dynamic Post-aware User Profile

With the general user profile, DHAP can capture personalized information of a user in a general view. However, when faced with a new input post, the historical data may play different roles. For example, when a crazy fan of cricket meets a post about cricket, they will be talkative and post a lot of things. But when they face other daily topics, they may behave much more gently. Hence, it is valuable to dynamically select the information that is most relevant to the input, and the chatbot can behave differently as the input varies. Following this idea, we propose to dynamically aggregate historical responses that are highly relevant to the current input post, and leverage them as a reference to drive the response generation.

To achieve this, DHAP uses a key-value memory network  to store the user's historical post-response pairs. Then the personalized post representation is used as the query to select highly related keys (historical posts) from the user history memory, and their corresponding values (historical responses) are aggregated as the dynamic post-aware user profile.

### User History Memory

We firstly transform the historical post-response pairs into key-value pairs and build the memory.

As we discussed earlier, the historical posts are usually issued by different users. Thus the language style and topic of them may be various. Under this circumstance, it is more reasonable to treat them independently, so DHAP applies a BiGRU to represent each historical post, respectively. In our implementation, this BiGRU shares parameters with the personalized post encoder (introduced in Section ). Consider the $i$-th historical post $P_i$, its representation is computed by a summing pooling over the word dimension as $\mathbf{p}_i^c = \sum_{j=1}^{L_{P_i}} \mathbf{h}^{i}_j$, where $\mathbf{h}^{i}_j$ is the hidden state of the BiGRU for the $j$-th word in $P_i$. For all historical posts $(P_1,\cdots,P_n)$, their representations are denoted as $(\mathbf{p}_1^c,\cdots,\mathbf{p}_n^c)$. Similarly, the representation of historical responses is also computed by the same pooling strategy as $\mathbf{r}_i^c = \sum_{j=1}^{L_{R_i}} \mathbf{e}^{r_i}_j$, where $\mathbf{e}^{r_i}_j$ is the contextual representation of the $j$-th word in $R_i$. Different from historical posts, the contextual word representations of historical responses are obtained by the history encoder in Equation (). Thus, all historical responses are represented as $(\mathbf{r}_1^c, \cdots,\mathbf{r}_n^c)$.

Finally, we build the user history memory by using the post representations as key and the corresponding response representations as value, *i.e.*, $\left\{\mathbf{k}_1^{M}:\mathbf{v}_1^{M},\cdots,\mathbf{k}_n^{M}:\mathbf{v}_n^{M} \right\} = \left\{\mathbf{p}_1^c:\mathbf{r}_1^c,\cdots,\mathbf{p}_n^{c}:\mathbf{r}_n^{c} \right\}$.

### Dynamic Post-aware User Profile

After building the user history memory, DHAP can select and aggregate the most relevant historical responses and build the dynamic post-aware user profile based on the input post. Specifically, the personalized representation of the input post obtained in Equation () is used as the query and attends to the memory keys to find the most relevant historical posts. The relevance is measured by the attention weights. Then the corresponding historical responses are summed up based on the normalized weights to construct the dynamic profile $\mathbf{e}^D_t$: $$\begin{gathered}
    \mathbf{e}^D_t = \sum_{i=1}^{n} \beta_{t,i}\mathbf{v}^{M}_i,
\end{gathered}$$ where $\beta_{t,i}$ is the attention weight of the $i$-th historical response based on the personalized post representation $\mathbf{c}_t$ in a similar way like Equation (). Note that the dynamic post-aware user profile is computed at each decoding time step $t$. The most relevant information hidden in historical responses can thus be selected to help response generation.

## Personalized Decoder

Finally, the generation probability of responses can be calculated by the personalized post representation $\mathbf{c}_t$, the general user profile $\mathbf{e}^G$, and the dynamic post-aware user profile $\mathbf{e}^D_t$. Inspired by CopyNet , in addition to leveraging the personalized information captured by the implicit user profile, we construct a personalized vocabulary so that the model is allowed to directly select personalized words that the user frequently used in history.

Specifically, the probability of the word $y_t$ generated in the personalized decoder is computed as Equation (), where $p\left(m_g\right)$ is the probability of general decoding mode and $p\left(m_c\right)$ is the probability of copy decoding mode. They are computed by our proposed decoding switcher. $p\left(y_t|m_g\right)$ and $p\left(y_t|m_c\right)$ are the probabilities of generating $y_t$ under two modes, respectively.

It is worth noting that: (1) The switching probability $p\left(m_g\right)$ and $p\left(m_c\right)$ are both in $[0,1]$. Thus, it is a "soft" decoding switcher. (2) The generic vocabulary also contains words in the personalized vocabulary. The generation probability of $y_t$ is obtained by the sum of probabilities under two decoding modes. Therefore, DHAP is just biased to the personalized words rather than lost in it.

### Decoding switcher

The decoding switcher determines the probability of the two decoding modes, *i.e.*, predicting a word from the generic vocabulary to maintain sentence fluency, or copying a word directly from the personalized vocabulary to make the response more informative and personalized. Specifically, DHAP computes the switching probability based on the matching degree between the decoder state and the concatenation of the personalized post representation, general user profile, and dynamic post-aware user profile, and further calculates two decoding mode probabilities. $$\begin{aligned}
    [p(m_g),p(m_c)] &= {{\rm Softmax}}(\mathbf{d}_t), \\
    \mathbf{d}_t &= {\rm MLP}([\mathbf{h}_t^R;\mathbf{c}_t;\mathbf{e}^G;\mathbf{e}^D_t]),
\end{aligned}$$ where $\mathbf{h}_t^R$ is the decoding hidden state at step $t$, and $\mathbf{d}_t \in \mathbb{R}^{2}$ is the matching degree vector to estimate the two mode probabilities. The softmax function guarantees that $p(m_g)+p(m_c)=1$.

### Personalized general decoding

While general decoding, the decoder should predict a word $y_t$ from the generic vocabulary: $$\begin{aligned}
    p(y_t|m_g) &= {\rm Softmax}({\rm MLP}([ \mathbf{h}_t^R;\mathbf{c}_t;\mathbf{e}^G;\mathbf{e}^D_t ])).
\end{aligned}$$

### Personalized copy decoding

The personalized vocabulary of user $u$ is composed of the words that appear in their historical responses. DHAP can directly select a word from this vocabulary to generate a more personalized response. Inspired by copy mechanism , the probability of selecting a word $y_t$ is computed as: $$\begin{aligned}
    p\left(y_t|m_c\right) = \sum_{i:r_i=y_t} \gamma_{t,i},
\end{aligned}$$ where $\gamma_{t,i}$ is the attention weight calculated by the personalized post representation $\mathbf{c}_t$ attentively reading the representation of historical responses $\mathbf{E}^R$ with the same attention process in Equation ().

### Decoder state updating

DHAP applies a GRU as the decoder. The hidden state at decoding step $t$ is calculated as: $$\begin{aligned}
    \mathbf{h}_t^R = {{\rm GRU}}(\mathbf{h}_{t-1}^R, [\mathbf{y}_{t-1};\mathbf{c}_t;\mathbf{e}^G;\mathbf{e}_t^D]),
\end{aligned}$$ where $\mathbf{y}_{t-1}$ is the embedding vector of the last generated word. The decoding states are initialized by the last hidden state of the personalized post encoder: $$\begin{aligned}
    \mathbf{h}_0^R = {{\rm ReLU}}({\rm MLP}(\mathbf{h}^P_{L_X})).
\end{aligned}$$

## Training and Optimization

Our goal is to maximize the generation probability of the target response given the input post and user's dialogue history. A length penalty is applied as  to alleviate the generation of meaningless responses. As a result, the loss function of DHAP is defined as: $$\begin{aligned}
    \mathcal{L} = - \sum_{t=1}^{L_Y} {\rm log}\left[p\left(y_t| y_{<t}, X, H\right)\right] - \eta L_Y,\label{loss}
\end{aligned}$$ where $\eta$ is a hyper-parameter to control the associated length penalty weight. $p\left(y_t| y_{\textless t}, X, H\right)$ is the generation probability of word $y_t$ based on the given input post and user's history, which is computed in Equation (). All parameters are optimized by the loss function and the whole model is trained in an end-to-end manner.

# Experiments

## Datasets

Although there are many public datasets for response generation, as far as we know, none of them contain user identification. To collect each user's dialogue history and evaluate the effectiveness of our model, we use two datasets extracted from two open online chatting forums, *i.e.*, Weibo and Reddit. The two datasets contain massive dialogue utterances (*i.e.*, post-response pairs) and user identification information, thus we can sample the data by users. To guarantee enough personalized information, we retrieve users with more than ten utterances to maintain an effective dialogue history. Each utterance is used as the target response for generation, while its former responses and the corresponding posts are treated as the dialogue history. We divide the utterances by users into 8:1:1 as training, validation, and test set respectively in time order. Besides, given a user, we ensure that the time of its records in the validation set and test set are behind the records in the training set.

**Weibo dataset** is a subset of PChatbotW , which is collected from Weibo for the one-year period beginning from Sept. 10, 2018. On Weibo, a user can post short messages visible to the public, which will be referred to as posts. Other users can make comments on a published post, which will be referred to as response. For data cleaning, we remove hashtags, URLs, emoticons, and duplicate text as . We also remove the utterances whose length is less than five words or more than 100 words. We use comparable scales of samples with  to conduct our experiments. It comprises 300,000 users and 8,618,374 utterances, in total 31M words.

**Reddit dataset** is extracted from comment chains scraped from Reddit from Dec. 1, 2015 to Oct. 30, 2018 . Since the Reddit discussions can be naturally expanded as tree-structured reply chains, we pair the parent node with all its child nodes respectively, and construct multiple post-response pairs. we treat the parent node and the child node as the post and response, respectively. As a result, a parent node can be a submission or a comment, while a child node only refers to a comment. For each submission, we use its title as the post text. We clean the raw data by removing instances containing word repetitions, offensive words, or multi-language sentences. It contains 315,340 users and 24,162,464 utterances, in total 55M words.

## Baselines

We evaluate the performance of our approach by comparing it with four groups of highly related and strong baseline methods:

\(1\) *Non-personalized response generation models*. **Seq2SeqWA**  is a standard GRU-based Seq2Seq model with attention mechanism. **MMI**  is a Seq2SeqWA using Maximum Mutual Information as loss function to improve diversity.

\(2\) *Personalized models using user ID embeddings*. **Speaker**  is also based on Seq2SeqWA but using user ID embeddings as additional input to the decoder. **PersonaWAE**  is built on an augmented Wasserstein autoencoder. It utilizes user ID embeddings for building a personalization Gaussian mixture distribution, and fuses personalization in the decoder.

\(3\) *Personalized models using explicit user profiles*. Since no explicit user profiles are given in our datasets, we use the historical responses of users as their persona texts. **GPMN**  enhances the Seq2SeqWA with a memory module, which encodes each piece of persona description as an individual memory representation. It uses the input message as the query to aggregate and incorporate the memory representations for response generation. **PerCVAE**  uses the user profile descriptions as conditions and applies a conditional variational autoencoder to generate diverse responses.

\(4\) *Personalized models using implicit user profiles*. Since no existing methods consider mining user profiles from dialogue history implicitly, we adapt several state-of-the-art multi-turn response generation models to personalized response generation. We replace the dialogue context in the original models by the user's historical post-response pairs. **VHRED-P**  extends the hierarchical recurrent encoder-decoder with a latent variable to model the complex dependencies among multiple utterances in the context. **ReCoSa-P**  applies a self-attention mechanism to measure the relevance between the response and each utterance in the context.

```latex
\begin{table*}[t!]\centering
    \small
    \caption{Automatic evaluation results of all models. All models are categorized into four groups: (1) non-personalized; (2) using user ID; (3) using explicit user profile; and (4) using dialogue history.
    % ``$\dagger$'' indicates the model outperforms all baseline models significantly with paired t-test at  $p \textless 0.05$ level.
    ``$\dagger$'' denotes the result is significantly worse than our method DHAP in t-test with $p \textless 0.05$ level.
    The best results are in  \textbf{bold} and the second best results are \underline{underlined}.}
    \label{table_a}
    \begin{tabular}{llllllllllll}
    \toprule
        \multirow{2}{*}{Dataset} & \multirow{2}{*}{Model}  & \multicolumn{3}{c}{Word Overlap} & \multicolumn{2}{c}{Diversity} & \multicolumn{3}{c}{Embedding Similarity} & \multicolumn{2}{c}{Personalization}  \\
    \cmidrule(lr){3-5} \cmidrule(lr){6-7}  \cmidrule(lr){8-10} \cmidrule(lr){11-12}
        & & BLEU-1 & BLEU-2 & ROUGE-L & Dist-1 & Dist-2 & Average & Extrema & Greedy  & P-F1(\%) & P-Cover \\
    \midrule
    % \multirow{9}{*}{Weibo} & (1) Seq2SeqWA & 3.335 & 0.294 & 8.740 & 0.935 & 2.184 & 0.321 & 0.266 & 0.254 & 1.824 & 1.698 \\
    % & (1) MMI & 3.632 & 0.095 & 5.317  & 10.714 & 43.479 & 0.477 & 0.695 & 0.305 & 2.633 & 1.874 \\
    % & (2) Speaker & 4.997 & 0.113 & 7.993  & 6.035 & 19.017 & 0.492 & 0.712 & 0.311 & 2.877 & 2.119 \\
    % & (2) PersonaWAE & 3.503 & 0.155 & 11.305  & 2.493 & 19.716 & \underline{0.513} & \underline{0.724} & 0.307 & \underline{3.240} & 5.108\\
    % & (3) GPMN & 4.901 & 0.696 & 8.090  & 11.726 & 32.734 & 0.353 & 0.391 & 0.301 & 2.254 & 4.512 \\
    % & (3) PerCVAE & 5.115 & 0.299 & 7.952  & \underline{14.095} & \underline{49.739} & 0.469 & 0.659 & 0.299 & 1.987 & 3.817 \\
    % & (4) VHRED-P & 6.992 & 0.709 & 10.695  & 2.122 & 7.874 & 0.437 & 0.560 & 0.307 & 2.783 & 5.459 \\
    % & (4) ReCoSa-P & \underline{7.266} & \underline{0.844} & \underline{11.469}  & 1.271 & 4.442 & 0.419 & 0.510 & \underline{0.312} & 3.107 & \underline{5.717}\\
    % & (4) DHAP & $\bm{9.324}^\dagger$ & $\bm{0.894}^\dagger$ & $\bm{14.122}^\dagger$ & $\bm{15.175}^\dagger$ & $\bm{58.806}^\dagger$ & $\bm{0.523}^\dagger$ &$\bm{0.747}^\dagger$ & $\bm{0.313}$  & $\bm{3.624}^\dagger$ & $\bm{7.013}^\dagger$ \\
    %  \midrule
    % \multirow{9}{*}{Reddit} &  {(1) Seq2SeqWA} & 1.819 & 0.023 & 4.069 & 5.203 & 19.485 & 0.545 & 0.554 & 0.472 & 0.951 & 0.516 \\
    % & (1) MMI & 2.065 & 0.011 & 3.784  & 5.914 & 31.093 & 0.543 & 0.607 & 0.454  & 2.233 & 0.828\\
    % & (2) Speaker & 2.642 & 0.054 & 4.469  & 8.951 & 34.187 & 0.538 & 0.606 & 0.457  &  2.482 & 1.455\\
    % & (2) PersonaWAE & 2.637 & 0.113 & 8.199  & 1.758 & 25.915 & 0.629 & \underline{0.685} & 0.442  & \underline{3.320} & 3.392\\
    % & (3) GPMN &  2.686 & 0.376 & 4.776  & \underline{12.325} & 35.762 & 0.406 & 0.331 & 0.358 & 2.211 & 3.026\\
    % & (3) PerCVAE & 5.933 & 0.576 & 8.112  & 9.631 & \underline{40.213} & \underline{0.637} & 0.649 & 0.499  & 2.595 & 3.456 \\
    % & (4) VHRED-P & 5.802 & 0.648 & 8.345  & 2.750  & 30.756 & 0.558 & 0.635 & 0.472 & 2.857 & 3.772\\
    % & (4) ReCoSa-P & \underline{6.113} & \underline{0.686}  & \underline{8.899} &2.593 & 25.767 & 0.574 & 0.632  & \underline{0.510} & 3.068 & \underline{3.998}\\
    % & (4) DHAP & $\bm{6.858}^\dagger$ & $\bm{0.737}^\dagger$ & $\bm{11.720}^\dagger$  & $\bm{18.707}^\dagger$ & $\bm{66.932}^\dagger$ & $\bm{0.709}^\dagger$ &$\bm{0.721}^\dagger$ & $\bm{0.539}^\dagger$  & $\bm{4.244}^\dagger$ & $\bm{4.639}^\dagger$\\
    \multirow{9}{*}{Weibo} & (1) Seq2SeqWA & $3.335^\dagger$ & $0.294^\dagger$ & $8.740^\dagger$ & $0.935^\dagger$ & $2.184^\dagger$ & $0.321^\dagger$ & $0.266^\dagger$ & $0.254^\dagger$ &  $1.698^\dagger  $&  $0.041^\dagger  $ \\
    & (1) MMI & $3.632^\dagger$ & $0.095^\dagger$ & $5.317^\dagger$  & $10.714^\dagger$ & $43.479^\dagger$ & $0.477^\dagger$ & $0.695^\dagger$ & $0.305$ &  $1.874^\dagger$ & $0.054^\dagger$ \\
    & (2) Speaker & $4.997^\dagger$ & $0.113^\dagger$ & $7.993^\dagger$  & $6.035^\dagger$ & $19.017^\dagger$ & $0.492^\dagger$ & $0.712^\dagger$ & $0.311$ &  $2.119^\dagger$ & $0.082^\dagger$  \\
    & (2) PersonaWAE & $3.503^\dagger$ & $0.155^\dagger$ & $11.305^\dagger$  & $2.493^\dagger$ & $19.716^\dagger$ & $\underline{0.513}^\dagger$ & $\underline{0.724}^\dagger$ & $0.307$ &  $5.108^\dagger$ & $\underline{0.093}^\dagger$ \\
    & (3) GPMN & $4.901^\dagger$ & $0.696^\dagger$ & $8.090^\dagger$  & $11.726^\dagger$ & $32.734^\dagger$ & $0.353^\dagger$ & $0.391^\dagger$ & $0.301^\dagger$ &  $4.512^\dagger$&  $0.084^\dagger$  \\
    & (3) PerCVAE & $5.115^\dagger$ & $0.299^\dagger$ & $7.952^\dagger$  & $\underline{14.095}^\dagger$ & $\underline{49.739}^\dagger$ & $0.469^\dagger$ & $0.659^\dagger$ & $0.299^\dagger$  & $3.817^\dagger$ & $0.086^\dagger$   \\
    & (4) VHRED-P & $6.992^\dagger$ & $0.709^\dagger$ & $10.695^\dagger$  & $2.122^\dagger$ & $7.874^\dagger$ & $0.437^\dagger$ & $0.560^\dagger$ & $0.307$  & $5.459^\dagger$ & $0.065^\dagger$  \\
    & (4) ReCoSa-P & $\underline{7.266}^\dagger$ & $\underline{0.844}^\dagger$ & $\underline{11.469}^\dagger$  & $1.271^\dagger$ & $4.442^\dagger$ & $0.419^\dagger$ & $0.510^\dagger$ & $\underline{0.312}$ & $\underline{5.717}^\dagger$ & $0.061^\dagger$ \\
    & (4) DHAP (ours) & $\bm{9.324}$ & $\bm{0.894}$ & $\bm{14.122}$ & $\bm{15.175}$ & $\bm{58.806}$ & $\bm{0.523}$ &$\bm{0.747}$ & $\bm{0.313}$  & $\bm{7.013}$ & $\bm{0.144}$  \\
     \midrule
    \multirow{9}{*}{Reddit} &  {(1) Seq2SeqWA} & $1.819^\dagger$ & $0.023^\dagger$ & $4.069^\dagger$ & $5.203^\dagger$ & $19.485^\dagger$ & $0.545^\dagger$ & $0.554^\dagger$ & $0.472^\dagger$  & $0.516^\dagger$ & $0.029^\dagger$ \\
    & (1) MMI & $2.065^\dagger$ & $0.011^\dagger$ & $3.784^\dagger$  & $5.914^\dagger$ & $31.093^\dagger$ & $0.543^\dagger$ & $0.607^\dagger$ & $0.454^\dagger$  &  $0.828^\dagger$ & $0.038^\dagger$ \\
    & (2) Speaker & $2.642^\dagger$ & $0.054^\dagger$ & $4.469^\dagger$  & $8.951^\dagger$ & $34.187^\dagger$ & $0.538^\dagger$ & $0.606^\dagger$ & $0.457^\dagger$  &  $1.455^\dagger$ & $0.031^\dagger$ \\
    & (2) PersonaWAE & $2.637^\dagger$ & $0.113^\dagger$ & $8.199^\dagger$  & $1.758^\dagger$ & $25.915^\dagger$ & $0.629^\dagger$ & $\underline{0.685}^\dagger$ & $0.442^\dagger$  &  $3.392^\dagger$ & $0.032^\dagger$ \\
    & (3) GPMN &  $2.686^\dagger$ & $0.376^\dagger$ & $4.776^\dagger$  & $\underline{12.325}^\dagger$ & $35.762^\dagger$ & $0.406^\dagger$ & $0.331^\dagger$ & $0.358^\dagger$  & $3.026^\dagger$ & $0.037^\dagger$   \\
    & (3) PerCVAE & $5.933^\dagger$ & $0.576^\dagger$ & $8.112^\dagger$  & $9.631^\dagger$ & $\underline{40.213}^\dagger$ & $\underline{0.637}^\dagger$ & $0.649^\dagger$ & $0.499^\dagger$  &  $3.456^\dagger$ & $0.040^\dagger$ \\
    & (4) VHRED-P & $5.802^\dagger$ & $0.648^\dagger$ & $8.345^\dagger$  & $2.750^\dagger$  & $30.756^\dagger$ & $0.558^\dagger$ & $0.635^\dagger$ & $0.472^\dagger$  & $3.772^\dagger$ & $\underline{0.047}^\dagger$ \\
    & (4) ReCoSa-P & $\underline{6.113}^\dagger$ & $\underline{0.686}^\dagger$  & $\underline{8.899}^\dagger$ &$2.593^\dagger$ & $25.767^\dagger$ & $0.574^\dagger$ & $0.632^\dagger$  & $\underline{0.510}^\dagger$  & $\underline{3.998}^\dagger$ & $0.044^\dagger$ \\
    & (4) DHAP (ours) & $\bm{6.858}$ & $\bm{0.737}$ & $\bm{11.720}$  & $\bm{18.707}$ & $\bm{66.932}$ & $\bm{0.709}$ &$\bm{0.721}$ & $\bm{0.539}$  &  $\bm{4.639}$ &  $\bm{0.111}$ \\
    \bottomrule
    \end{tabular}
    % \label{tab:my_label}
\end{table*}
```

| Model | Readability | Informativeness | Personalization |
|:---|:--:|:--:|:--:|
| \(1\) Seq2SeqWA | $2.10^\dagger$ | $1.85^\dagger$ | $0.19^\dagger$ |
| \(1\) MMI | $2.06^\dagger$ | $1.88^\dagger$ | $0.23^\dagger$ |
| \(2\) Speaker | $\underline{2.14}^\dagger$ | $1.93^\dagger$ | $0.25^\dagger$ |
| \(2\) PersonaWAE | $2.07^\dagger$ | $1.99^\dagger$ | $0.36^\dagger$ |
| \(3\) GPMN | $2.12^\dagger$ | $1.92^\dagger$ | $0.35^\dagger$ |
| \(3\) PerCVAE | $2.04^\dagger$ | $\underline{2.01}^\dagger$ | $0.39^\dagger$ |
| \(4\) VHRED-P | $2.09^\dagger$ | $1.96^\dagger$ | $\underline{0.47}^\dagger$ |
| \(4\) ReCoSa-P | $2.12^\dagger$ | $1.93^\dagger$ | $0.44^\dagger$ |
| \(4\) DHAP (ours) | $\bm{2.26}$ | $\bm{2.09}$ | $\bm{0.56}$ |
| Ground-truth | 2.69 | 2.35 | 0.84 |

Human evaluation results on Weibo dataset. "$\dagger$" denotes the result is significantly worse than our method in t-test with $p \textless 0.05$ level. The best results are in **bold** and the second best results are <u>underlined</u>. The Fleiss Kappa is 0.42.

## Evaluation Metrics

**Automatic Evaluation**: We consider several automatic metrics in different perspectives to jointly evaluate the generated responses. (1) We use BLEU-1, BLEU-2 , and ROUGE-L  to measure word overlaps between the generated response and ground truth. A higher value of these metrics indicates a higher word-level similarity between the generated response and the golden response. (2) Following , we employ Dist-1 and Dist-2 to evaluate the diversity of the generated response. Responses with more distinct unigrams/bigrams will have higher Dist-1/Dist-2. (3) As suggested by , we use three embedding-based metrics to measure the semantic relevance between the generated response and the ground-truth response. Concretely, we use the bag-of-words embeddings to represent both the generated and ground-truth response, and calculate their average similarity (Ave.), greedy similarity (Gre.), and extrema similarity (Ext.). The pre-trained word embeddings for Weibo and Reddit corpus are offered by and , respectively. (4) Furthermore, since the goal of our model is to leverage user history for personalization, we evaluate the personalized performance by measuring how much information in the dialogue history is reflected in the generated response. Following , we use Persona F1 (P-F1) to measure the unigram F1 between the generated response and user's historical responses. Thus, the more historical words the generated response contains, the higher P-F1 we will get. Since the importance of the shared words can be different, following , we further use Persona Coverage (P-Cover) to measure the IDF-weighted word overlap between generated response and dialogue history. Specifically, for $n$ historical responses $\left\{R_1,\cdots,R_n\right\}$ and the generated response $Y$, P-Cover is defined as: $$\begin{aligned}
  {\rm P\text{-}Cover} = {{\rm max}}_{j \in \left[1,n\right] } \frac{ \sum_{w_k \in W_j}{\rm IDF}(w_k)}{|Y|},
\end{aligned}$$ where $W_j$ is the set of shared words between $R_j$ and $Y$.

```latex
\begin{table*}[t!]\centering
    \small
    \caption{Performance of ablation models on Weibo dataset. ``$\dagger$'' denotes the result is significantly worse than our method in t-test with $p \textless 0.05$ level.  The best results are denoted in \textbf{bold} font.}
    \label{table_ablation}
    \begin{tabular}{p{0.1\textwidth}llllllllll}
    \toprule
         \multirow{2}{*}{Model}  & \multicolumn{3}{c}{Word Overlap} & \multicolumn{2}{c}{Diversity} & \multicolumn{3}{c}{Embedding Similarity}  & \multicolumn{2}{c}{Personalization} \\
    \cmidrule(lr){2-4} \cmidrule(lr){5-6}  \cmidrule(lr){7-9} \cmidrule(lr){10-11}
         & BLEU-1 & BLEU-2 & ROUGE-L & Dist-1 & Dist-2 & Average & Extrema & Greedy  & P-F1(\%) & P-Cover \\
    \midrule
     DHAP & $\bm{9.324}$ & $\bm{0.894}$ & $\bm{14.122}$ & $\bm{15.175}$ & $\bm{58.806}$ & $\bm{0.523}$ &$\bm{0.747}$ & $\bm{0.313}$   & 7.013 & 0.144 \\
     \midrule
     \quad \textit{w/o} G & $7.726^\dagger$ & $0.801^\dagger$ & $11.815^\dagger$ & $12.176^\dagger$ & $49.808^\dagger$ &  $0.495^\dagger$ &  $0.707^\dagger$ & $0.294^\dagger$  & $6.179^\dagger$ & $0.107^\dagger$   \\
     \quad \textit{w/o} D & $8.503^\dagger$ & $0.855^\dagger$  & $12.610^\dagger$ & $13.699^\dagger$ & $54.623^\dagger$ & $0.499^\dagger$ & $0.713^\dagger$ & $0.303^\dagger$  &   $6.286^\dagger$ & $0.109^\dagger$   \\
     \quad \textit{w/o} PC & 8.830 & $0.868^\dagger$ & 13.981 & 14.457 & $56.263^\dagger$ & $0.503^\dagger$ & $0.728^\dagger$ & $0.301^\dagger$  & 6.884   & $0.120^\dagger$\\
          \midrule
    \quad \textit{w/o} GEN & $4.982^\dagger$ & $0.328^\dagger$  & $9.571^\dagger$ & $9.051^\dagger$ & $32.566^\dagger$ & $0.478\dagger$ & $0.571^\dagger$ & $0.276^\dagger$ & \textbf{9.331} & $\textbf{0.165}$  \\
    \quad \textit{w/o} COP & $8.347^\dagger$ &  $0.837^\dagger$ & $12.585^\dagger$ & $13.487^\dagger$ & $52.087^\dagger$ & $0.499^\dagger$ &$0.717^\dagger$   & $0.298^\dagger$  & $6.234^\dagger$ & $0.110^\dagger$  \\
    \quad \textit{w} FIX & $8.549^\dagger$ & $0.855^\dagger$ & $12.871^\dagger$ & $13.904^\dagger$ & $54.539^\dagger$ & $0.496^\dagger$ & $0.716^\dagger$  & $0.301^\dagger$  & $6.326^\dagger$ & $0.113^\dagger$  \\
    %   \midrule
    \bottomrule
    \end{tabular}
    % \label{tab:my_label}
\end{table*}
```

**Human Evaluation**: The automatic evaluation metrics can measure the quality of the generated response with respect to the ground-truth. However, due to the diversity of human dialogues, a response different from the ground-truth may also be acceptable. Thus, we randomly sample 100 test samples to conduct human evaluations. We present the generated responses, the corresponding post, and the user's historical post-response pairs to three well-educated annotators. The annotators will evaluate the quality of the generated responses in a double-blind fashion. Following , the evaluation criterion includes: (1) Readability, which measures the grammatical correctness and smoothness of generated responses; (2) Informativeness, which measures whether the responses are informative or trivial; and (3) Personalization, which measures if the response can reflect personalized information (sharing some information with the history of the user). For the former two perspectives, we use a score 1/2/3 for bad/normal/good quality. For personalization, we use the score 0/1 to judge whether a response reflects personalized information or not. The Fleiss Kappa is 0.42 that indicates the annotators achieve a substantial agreement.

## Implement Details

To determine the parameters of the model, we conducted multiple sets of experiments. The final parameters are selected as follows. For all datasets, we use 512 as the hidden size of GRU, 0.001 as the learning rate. The hidden size and number of heads of Transformer are 256 and 8. The number of Transformer layers $N=6$. The history length is set to 25. The vocabulary size is limited to 40,000. The word embedding dimension is 300/100 for Weibo/Reddit datasets, respectively. We use the Adam optimizer with a batch size of 256. We train all models for 10 epochs and select the best model based on the validation results on BLEU-1.

## Experimental Results

### Automatic Evaluation

All evaluation results under automatic metrics are reported in Table . We can observe that:

\(1\) Among all models, **DHAP achieves the best results in terms of all evaluation metrics.** DHAP improves performance with a large margin over two strongest baselines VHRED-P and ReCoSa-P, which can also learn implicit user profile. Concretely, DHAP significantly outperforms ReCoSa-P by $28.4\%/12.2\%$ improvements in BLEU-1 on Weibo/Reddit dataset. The reason for the improvement reduction on Reddit set is that it has a larger scale and more varied conversations, which leads to more noise. Besides, for the embedding similarity metrics, DHAP also outperforms the best baselines. These results demonstrate that DHAP can generate more semantically relevant responses to the ground-truth by leveraging user's history. Furthermore, DHAP has dramatic improvements of Dist-1/2, indicating DHAP can generate more informative and diverse responses based on the personalized information. All these results prove that learning implicit user profiles from user's dialogue history can improve the quality of generated responses.

\(2\) All personalized methods outperform non-personalized methods, indicating that **personalization is helpful for generating more informative and relevant responses**. Seq2SeqWA generally has the lowest performance, reflecting that the semantic information in the post is insufficient for generating an informative response. MMI improves the diversity performance significantly, but loses some ability on modeling semantic relevance, as it changes the training objective. Speakers and PersonaWAE use user ID embeddings to identify different users for personalization, and outperform the non-personalized methods. The explicit persona-based model GPMN and PerCVAE show comparable performance to the user ID embedding based baselines. A potential reason is that they are designed for leveraging explicit user profile, which is usually of high quality. In our case, they are only provided with the user's historical responses, which are much noisy. Therefore, existing personalized methods for explicit user profile is not appropriate for dealing with the implicit user profile contained in the user history.

\(3\) Among all personalized methods, **the ones using implicit user profile perform better.** VHRED-P and ReCoSa-P show better performance on most metrics, confirming that the dialogue history can be used to mine implicit user profile for a specific user. However, these two methods are originally proposed for multi-turn dialogue generation. Their performance in personalized tasks is limited because the dialogue history covers far more aspects than the context in multi-turn dialogue. On the contrary, our DHAP models the personalized information in both encoder and decoder side, and consider the implicit user profile in both general and dynamic style. Hence, DHAP can achieve significant improvements compared with existing personalized baselines.

### Human Evaluation

We also conduct a human evaluation for all models on Weibo dataset. The results are shown in Table . Generally, DHAP achieves significant improvements in terms of all perspectives, which is consistent with the results on automatic metrics. In particular, we find that DHAP is much better than ReCoSa-P in terms of personalization. This is because DHAP learns the implicit user profile more comprehensively and enhances the influence of personalized words directly in the decoder. DHAP also performs better than other baselines in terms of readability, which shows that DHAP is better at language understanding with the help of user history. Besides, the informativeness of responses generated by DHAP is also improved. This demonstrates that leveraging personalized information is effective to generate more meaningful responses.

In summary, the automatic and human evaluation results strongly verify that **dialogue history is suitable to build the user profile implicitly, and leveraging implicit user profiles is effective to generate meaningful and personalized responses, further achieving a personalized chatbot**.

## Further Analysis

We further analyze the influence of different modules (Section ) and the performance over different history lengths (Section ). Both of these experiments are performed on Weibo dataset.

### Ablation Study

DHAP learns several personalized user profiles based on the dialogue history and designs a decoding switcher and two decoding strategies in the personalized decoder. We remove one of them once a time to analyze its contribution. The experiment results on Weibo dataset are shown in Table .

**The ablation on personalized representations.** Three settings are considered: (1) **without G**: the general user profile $\mathbf{e}^G$ is not used; (2) **without D**: the dynamic post-aware user profile $\mathbf{e}^D_t$ is not used; and (3) **without PC**: the post encoder is non-personalized, namely initializing the post encoder with random states rather than the general user profile.

The results show that all of the personalized representations are useful. Specifically, removing the general user profile causes the most decline in all metrics, which confirms the necessity and contribution of it on summarizing personalized information in a general view. The performance degradation caused by removing the dynamic post-aware user profile shows that selecting historical responses relevant to the input post contributes to the further enhancement of user modeling. The influence of removing personalization in post encoder is relatively smaller. It proves that using the user profile to enhance the understanding of the current post is effective but limited, since such information is only provided at the beginning step and decreases with the hidden state update.

**The ablation of components in the personalized decoder.** We test the following variants of our model: (1) **without GEN**: the general decoding is banned; (2) **without COP**: the copying mode is banned; (3) **with FIX**: the probability of two modes are fixed. Specifically, the general decoding probability is set as 0.8 and the copy decoding probability is 0.2, these probabilities are set according to the best results of DHAP in our preliminary experiments.

It can be seen that the results of three variants all underperform the whole framework. Without general decoding, the performance of DHAP drops sharply in terms of all metrics except personalization metrics. Specifically, it drops 46.55% in terms of BLEU-1. This indicates that only copying words from personalized vocabulary is unable to generate a suitable response, because there are lots of noises irrelevant to the current post and some general words may not be contained in the vocabulary. The reason for its improvement on personalized metrics is that all of the generated words are copied from the history, regardless of the significant hurt on relevance and diversity. Thus, general decoding considering both the post, decoding states, and personalized information is necessary. However, only using general decoding also hurts the performance, which indicates the words reflecting user personalized information is also very valuable. To combine the two decoding strategies, DHAP calculates the possibilities of two decoding strategies dynamically. It works well in DHAP yet using fixed probabilities has lower performance.

### Performance across Various Lengths of Dialogue History

As we leverage user's dialogue history for personalization, the length of history may affect the model's performance. To investigate the influence of history length, we test the performance of DHAP by using different numbers of historical post-response pairs. The results of BLEU-1 on Weibo set are illustrated in Figure . We find:

\(1\) In general, DHAP performs better when a user has a longer dialogue history. This is consistent with our speculation as a longer dialogue history can provide more personalized information. DHAP achieves the best performance with the history length around 25. Unfortunately, when more than 30 historical pairs are used, the performance of DHAP becomes unstable. The potential reason is that more historical data may bring more noise and increase the difficulty of building an effective user profile. (2) When the history is less than 5, ReCoSa-P performs better than DHAP without general user profile. This is because the persona information is extremely limited. Under this circumstance, the more complex architecture of ReCoSa shows its superiority. Nevertheless, our DHAP still performs best, showing its scalability for various history lengths.

# Conclusion

In this work, we implemented response generation of personalized chatbots in an alternative way. Different from existing personalized methods, we propose the personalized model DHAP, which learns the implicit user profile automatically from large-scale user dialogue history. We design a personalized language model to capture the user's general interest from their historical responses and summarize the general user profile. To further highlight the historical responses which are relevant and valuable to the current input post, we build a history memory and construct the dynamic post-aware user profile. We build a personalized decoder to coordinate two personalized decoding strategies. Experimental results confirm the effectiveness of our model on generating informativeness and personalized responses.


abstract: With the widespread use of mobile phones and scanners to photograph and upload documents, the need for extracting the information trapped in unstructured document images such as retail receipts, insurance claim forms and financial invoices is becoming more acute. A major hurdle to this objective is that these images often contain information in the form of tables and extracting data from tabular sub-images presents a unique set of challenges. This includes accurate detection of the tabular region within an image, and subsequently detecting and extracting information from the rows and columns of the detected table. While some progress has been made in table detection, extracting the table contents is still a challenge since this involves more fine grained table structure(rows & columns) recognition. Prior approaches have attempted to solve the table detection and structure recognition problems independently using two separate models. In this paper, we propose TableNet: a novel end-to-end deep learning model for both table detection and structure recognition. The model exploits the interdependence between the twin tasks of table detection and table structure recognition to segment out the table and column regions. This is followed by semantic rule-based row extraction from the identified tabular sub-regions. The proposed model and extraction approach was evaluated on the publicly available ICDAR 2013 and Marmot Table datasets obtaining state of the art results. Additionally, we demonstrate that feeding additional semantic features further improves model performance and that the model exhibits transfer learning across datasets. Another contribution of this paper is to provide additional table structure annotations for the Marmot data, which currently only has annotations for table detection.
author:
bibliography: file.bib
title: TableNet: Deep Learning model for end-to-end Table detection and Tabular data extraction from Scanned Document Images

# Introduction

With the proliferation of mobile devices equipped with cameras, an increasing number of customers are uploading documents via these devices, making the need for information extraction from these images more pressing. Currently, these document images are often manually processed resulting in high labour costs and inefficient data processing times. Frequently, these documents contain data stored in tables with multiple variations in layout and visual appearance. A key component of information extraction from these documents therefore involves digitizing the data present in these tabular sub-images. The variation in the table structure, and in the graphical elements used to visually separate the tabular components make extraction from these images a very challenging problem. Most existing approaches to tabular information extraction divide the problem into the two separate sub-problems of 1) table detection and 2) table structure recognition, and attempt to solve each sub-problem independently. While table detection involves detection of the image pixel coordinates containing the tabular sub-image, tabular structure recognition involves segmentation of the individual rows and columns in the detected table.

In this paper, we propose TableNet, a novel end-to-end deep learning model that exploits the inherent interdependence between the twin tasks of table detection and table structure identification. The model utilizes a base network that is initialized with pre-trained VGG-19 features. This is followed by two decoder branches for 1) Segmentation of the table region and 2) Segmentation of the columns within a table region. Subsequently, rule based row extraction is employed to extract data in individual table cells.

A multi-task approach is used for the training of the deep model. The model takes a single input image and produces two different semantically labelled output images for tables and columns. The model shares the encoding layer of VGG-19 for both the table and column detectors, while the decoders for the two tasks are separate. The shared common layers are repeatedly trained from the gradients received from both the table and column detectors while the decoders are trained independently. Semantic information about elementary data types is then utilized to further boost model performance. The utilization of the VGG-19 as a base network, which is pre-trained on the ImageNet dataset allows for exploitation of prior knowledge in the form of low level features learnt via training over ImageNet.

We have evaluated TableNet's performance on the ICDAR-2013 dataset, demonstrating that our approach marginally outperforms other deep models as well as other state-of-the-art methods in detecting and extracting tabular information in image documents. We further demonstrate that the model can generalize to other datasets with minimal fine tuning, thereby enabling transfer learning. Furthermore, the Marmot dataset which has previously been annotated for table detection was also manually annotated for column detection, and these new annotations will be publicly released to the community for future research.

In summary, the primary contributions made in this paper are as follows:

1.  We propose TableNet: a novel end-to-end deep multi-task architecture for both table detection and structure recognition yielding state of the art performance on the public benchmark ICDAR and Marmot datasets.

2.  We demonstrate that adding additional spatial semantic features to TableNet during training further boosts model performance.

3.  We show that using a pre-trained TableNet model and fine tuning it on an another new dataset will boost the performance of the model on the new dataset, thereby allowing for transfer learning.

4.  We have manually annotated the Marmot dataset for table data extraction and will release the annotations to the community.

The rest of the paper is organized as follows: Section provides an overview of the related work on tabular information extraction. Section provides a detailed description of the TableNet model. Section outlines the extraction process with TableNet. Section provides details about the datasets, preprocessing steps and training. Section outlines the experiment details and results. Finally, the conclusions and future work are presented in Section .

# Related Work

There is significant prior work on identifying and extracting the tabular data inside a document. Most of these have reported results on table detection and data extraction separately

Before the advent of deep learning, most of the work on table detection was based on heuristics or metadata. TINTIN exploited structural information to identify tables and their component fields. used hierarchical representations based on the MXY tree for table detection and was the first attempt at using Machine Learning techniques for this problem. T Kasar et al. identified intersecting horizontal, vertical lines and low-level features and used an SVM classifier to classify an image region as a table region or not.

Probabilistic graphical models were also used to detect tables; Silva et al. modelled the joint probability distribution over sequential observations of visual page elements and the hidden state of a line (HMM) to merge potential table lines into tables resulted in a high degree of completeness. Jing Fang et al. used the table header as a starting point to detect the table region and decompose its elements. Raskovic et al. made an attempt to detect borderless tables. They utilized whitespaces as a heuristic rather than content for detection.

Recently, DeepDeSRT was proposed which uses deep learning for both table detection and table structure recognition, i.e. identifying rows, columns, and cell positions in the detected tables. This work achieves state-of-the-art performance on the ICDAR 2013 table competition dataset. After this, combined deep convolutional neural networks, graphical models and saliency concepts for localizing tables and charts in documents. This technique was applied on an extended version of ICDAR 2013 table competition dataset and outperforms existing models. locates the text components and extracts text blocks. After that, the height of each text block is compared with the average height and if satisfies a series of rules, the ROI is regarded as a table.

T-Recs was one of the earliest works to extract tabular data based on clustering of given word segments and overlap of the text inside the table. Y. Wang et al. estimates probabilities from geometric measurements made on the various entities in a given document.

Ashwin et al. exploit the formatting cues from semi-structured HTML tables to extract data from web pages. Here the cells are already demarcated by tags since they are in HTML tables. Singh et al. use object detection techniques for Document Layout understanding.

# TableNet: Deep Model for Table and Column Detection

In all prior deep learning based approaches, table detection and column detection are considered separately as two different problems, which can be solved independently. However, intuitively if all the columns present in a document are known apriori, the table region can be determined easily. But by definition, columns are vertically aligned word/numerical blocks. Thus, independently searching for columns can produce a lot of false positives and knowledge of the tabular region can greatly improve results for column detection (since both tables and columns have common regions). Therefore, if convolutional filters utilized to detect tables, can be reinforced by column detecting filters, this should significantly improve the performance of the model. Our proposed model, exploits this intuition and is based on the Long et al., encoder-decoder model for semantic segmentation. The encoder of the model is common across both tasks, but the decoder emerges as two different branches for tables and columns. Concretely, we enforced the encoding layers to use the ground truth of both tables and columns of document for training. However, the decoding layers are separated for table and column branches. Thus, there are two computational graphs to train.

The input image for the model, is first transformed into an RGB image and then, resized to 1024 \* 1024 resolution. This modified image is processed using tesseract OCR as described in the previous section. Since a single model produces both the output masks for the table and column regions, these two independent outputs have binary target pixel values, depending on whether the pixel region belongs to the table/column region or background respectively.

The problem of detecting tables in documents is similar to the problem of detecting objects in real world images. Similar to the generic object detection problem, visual features of the tables can be used to detect tables and columns. The difference is that the tolerance for noise in table/column detection is much smaller than in object detection. Therefore, instead of regressing for the boundaries of tables and columns, we employed a method to predict table and column regions pixel-wise. Recent work on semantic segmentation based on pixel wise prediction, has been very successful. FCN architecture, proposed by Long et al., has demonstrated the accuracy of encoder-decoder network architectures for semantic segmentation. The FCN architecture uses the skip-pooling technique to combine the low-resolution feature maps of the decoder network with the high-resolution features of encoder networks. VGG-16 is used as the base layer in their model and fractionally-strided convolution layers are used to upscale the found low-resolution semantic map which is then combined with high resolution encoding layers.

Our model uses the same intuition for the encoder/decoder network as the FCN architecture. Our proposed model as shown in Figure , uses a pre-trained VGG-19 layer as the base network. The fully connected layers (layers after pool5) of VGG-19 are replaced with two (1x1) convolution layers. Each of these convolution layers (conv6) uses the ReLU activation followed by a dropout layer having probability of 0.8 (conv6 + dropout as shown in Figure ). Following this layer, two different branches of the decoder network are appended. This is according to the intuition that the column region is a subset of the table region. Thus, the single encoding network can filter out the active regions with better accuracy using features of both table and column regions. The output of the (conv6 + dropout) layer is distributed to both decoder branches. In each branch, additional layers are appended to filter out the respective active regions. In the table branch of the decoder network, an additional (1x1) convolution layer, conv7_table is used, before using a series of fractionally strided convolution layers for upscaling the image. The output of the conv7_table layer is also up-scaled using fractionally strided convolutions, and is appended with the pool4 pooling layer of the same dimension. Similarly, the combined feature map is again up-scaled and the pool3 pooling is appended to it. Finally, the final feature map is upscaled to meet the original image dimension. In the other branch for detecting columns, there is an additional convolution layer (conv7_column) with a ReLU activation function and a dropout layer with the same dropout probability. The feature maps are up-sampled using fractionally strided convolutions after a (1x1) convolution (conv8_column) layer. The up-sampled feature maps are combined with the pool4 pooling layer and the combined feature map is up-sampled and combined with the pool3 pooling layer of the same dimension. After this layer, the feature map is up-scaled to the original image. In both branches, multiple (1x1) convolution layers are used before the transposed layers. The intuition behind using (1x1) convolution is to reduce the dimensions of feature maps (channels) which is used in class prediction of pixels, since the output layers (output of encoder network) must have channels equal to the number of classes (channel with max probability is assigned to corresponding pixels) which is later up-sampled. Therefore, the outputs of the two branches of computational graphs yield the mask for the table and column regions.

# Table Row Extraction

After processing the documents using TableNet, masks for table and column regions are generated. These masks are used to filter out the table and its column regions from the image. Since, all word positions of the document are already known (using Tesseract OCR), only the word patches lying inside table and column regions are filtered out. Now, using these filtered words, a row can be defined as the collection of words from multiple columns, which are at the similar horizontal level. However, a row is not necessarily confined to a single line, and depending upon the content of a column or line demarcations, a row can span multiple lines. Therefore, to cover the different possibilities, we formulate three rules for row segmentation:

-   In most tables for which line demarcations are present, the lines segment the rows in each column. To detect the possible line demarcation (for rows), every space between two vertically placed words in a column is tested for the presence of lines via a Radon transform. The presence of horizontal line demarcation clearly segments out the row.

-   In case a row spans multiple lines, the rows of the table which have maximum non-blank entries is marked as the starting point for a new row. For example in a multi-column table, some of the columns can have entries spanning just one line (like quantity, etc), while others can have multi-line entries (like description, etc). Thus, each new row begins when all the entities in each column are filled.

-   In tables, where all the columns are completely filled and there are no line demarcations, each line (level) can be seen as a unique row.

# Dataset Preparation

Deep-learning based approaches are data-intensive and require large volumes of training data for learning effective representations. Unfortunately, there are very few datasets like Marmot, UW3, etc for table detection and even these contain only a few hundred images. There are even fewer datasets for table structure identification such as the ICDAR 2013 table competition dataset for both table detection and its structural analysis . This creates a constraint for deep learning models to solve both table detection and table structural analysis.

For training our model, we have used the Marmot table recognition dataset. This is the largest publicly available dataset for table detection but unfortunately did not have annotations for table columns or rows. We manually annotated the dataset for table structure recognition since the dataset had ground truth only for table detection. The dataset was manually annotated by labeling the bounding boxes around each of the columns inside the tabular region. The manually annotated modified dataset is publicly released with the name Marmot Extended for table structure recognition [^1].

## Providing Semantic Information

Intuitively, any table has common data types in the same row/column depending on whether the table is in row major or column major form. For example, a name column will contain strings, while, a quantity header will contain numbers. To provide this semantic information to the deep model, text regions with similar data types are color coded. This modified image is then fed to the network resulting in improved performance.

We have included spatial semantic features by highlighting the words with patches as shown in Figure (a) and this dataset is also made publicly available. The document images are first processed with histogram equalization. After pre-processing, the word blocks are extracted using tesseract OCR. These word patches are colored depending upon their basic data type. The resulting modified images are used as input to the network. The TableNet model takes the input image and generates the binary mask image of both table and columns separately. The achieved result is filtered using rules outlined on the basis of the detected table and column masks. An example of the generated output is shown in Figure .

## Training Data Preparation for TableNet

To provide the basic semantic type information to the model, the word patches are color coded. The image is first processed with tesseract OCR, to get all the word patches in the image document. Then the words are processed via regular expressions to determine their data-type. The intuition is to color the word bounding boxes to impart both the semantic and spatial information to the network. Each data type is given a unique color and, bounding-boxes of words with similar data-types are shaded in the same color. Word bounding boxes are filtered out to remove spurious detections. However, since word detection and extraction from the OCR will produce some noise, the model needs to learn to be resilient to these cases. Therefore to simulate the case of incomplete word detection in the training image, a few randomly selected word patches are dropped deliberately. The formed color coded image can be used for training, but many visual features like line demarcations, corners, color highlights, etc are lost, while using only the word annotated document image. Therefore, to retain those important visual features in the training data, the word highlighted image is pixel-wise added to the original image. These modified document images are then used for training.

```latex
\begin{tabular}{|M{3.7cm}||M{1cm}|M{1cm}|M{1.3cm}|M{1.3cm}|  }
 %\hline
 %\multicolumn{4}{|c|}{Results on Table Detection} \\
 \hline
Model &  Recall & Precision & F1-Score\\
\hline
\textbf{TableNet + Semantic Features (fine-tuned on ICDAR)}  &\textbf{0.9628} & \textbf{0.9697} & \textbf{0.9662}\\
\hline
\textbf{TableNet + Semantic Features} & \textbf{0.9621} & \textbf{0.9547} & \textbf{0.9583}\\
\hline
\textbf{TableNet} & \textbf{0.9501} & \textbf{0.9547} & \textbf{0.9547}\\
\hline
DeepDeSRT \cite{schreiber} & 0.9615 & 0.9740 & 0.9677 \\
\hline
Tran et al \cite{tran2015table} & 0.9636 & 0.9521 & 0.9578 \\
 \hline
\end{tabular}
```
```latex
\begin{tabular}{|M{3.7cm}||M{1cm}|M{1cm}|M{1.3cm}|M{1.3cm}|  }
%\hline
 %\multicolumn{4}{|c|}{Results on Table Detection} \\
 \hline
Model & Recall & Precision & F1-Score\\
\hline
\textbf{TableNet + Semantic Features (fine-tuned on ICDAR)} & \textbf{0.9001} & \textbf{0.9307} & \textbf{0.9151}\\
\hline
\textbf{TableNet + Semantic Features} & \textbf{0.8994} & \textbf{0.9255} & \textbf{0.9122}\\
\hline
\textbf{TableNet} & \textbf{0.8987} & \textbf{0.9215} & \textbf{0.9098}\\
\hline
DeepDeSRT \cite{schreiber} & 0.8736 & 0.9593 & 0.9144\\
\hline
\end{tabular}
```

# EXPERIMENTS AND RESULTS

This section describes the different experiments performed on the ICDAR 2013 table competition dataset . The model performance is evaluated based on the Recall, Precision & F1-score. These measures are computed for each document and their average is taken across all the documents.

1.  Table Detection: Completeness and Purity are the two standard measures used in page segmentation . A region is complete if it includes all sub-objects present in the ground-truth. A region is pure if it does not include any sub-objects which are not in the ground-truth. Sub-objects are created by dividing the given region into meaningful parts like heading of a table, body of a table etc. But these measures do not discriminate between minor and major errors. So, individual characters in each region are treated as sub-objects. Precision and recall measures are calculated on these sub-objects in each region and the average is taken across all the regions in a given document.

2.  Table Data Extraction: Each entry inside a table is defined as a cell. For each cell, adjacency relations are generated with the nearest horizontal and vertical neighbours. Thus, adjacency relations of a given cell is a 1D-tuple containing the text information of its neighboring cells. The content in each cell was normalized; white spaces were removed, special characters were replaced with underscores and lowercase letters with uppercase. This 1D-tuple can then be compared with the ground truth by using precision and recall.

TableNet requires both table and structure annotated data for training. We used the Marmot table detection data and manually annotated the structure information. There are a total of 1016 documents containing tables including both Chinese and English documents, out of which 509 English documents are annotated and used for training. The proposed deep model has been implemented in Tensorflow and implemented on a system with Intel(R) Xeon(R) Silver CPU having 32 cores and RAM of 128 GB Tesla V100-PCIE-1 GPU with 6GB of GPU memory. In Experiment 1, we trained our model on all positive samples of Marmot and tested on the ICDAR 2013 table competition dataset for both table and structure detection. There are two computation graphs which require training. Each training sample is a tuple of a document image, table mask and column mask. With each training tuple, the two graphs are computed at-least twice. In the initial phase of training, the table branch and column branch are computed in the ratio of 2:1. With each training tuple, the table branch of the computational graph is computed twice, and then the column branch of the model is computed. It is worth noting that, although the table branch and column branch are different, the encoder is the same for both. During initial iterations of training, the learning is more focused on training the model to generate big active tabular regions which on subsequent training specializes to column regions. After around 500 iterations with a batch size of 2, when train loss of both table and column detectors are comparable, this training scheme is modified. However, it should be noted that the table classifier at this stage must exhibit a converging trend (otherwise, training is extended with the same 2:1 scheme). The model is then trained in the ratio of 1:1 for both branches until convergence. Using the same training pattern, the model is trained for 5000 iterations with a batch size of 2 and learning rate of 0.0001. The Adam optimizer is used for improving and optimizing training with parameters beta1=0.9, beta2=0.999 and epsilon=1e-08. The convergence and over-fitting behavior was monitored by observing the performance over the validation set (taken from ICDAR 2013 data-set). During testing, 0.99 is taken as the threshold probability for pixel-wise prediction. The results are compiled in Table and Table .

Similarly, in Experiment 2, we used the modified Marmot data-set where, the words in each document were highlighted to provide semantic context as described in Section . All the parameters were identical to Experiment 1. There was slight improvement in the results, when these spatial, semantic information are appended to the images (see table for comparison). Output of the model is shown in Figure  . Additionally, Experiment 3 was carried out to compare TableNet with the closest deep-learning based solution, DeepDSert . In DeepDSert, separate models are made for Table detection and structure recognition, which were trained on different datasets such as Marmot for table detection, and the ICDAR 2013 table dataset for table structure recognition. To generate comparable results, we fine-tuned our Marmot trained TableNet model, on ICDAR train and test data splits. As done in DeepDSert, we also randomly chose 34 images for testing and used the rest of the data images for fine-tuning our model. Our model was fine-tuned, with the same parameters, in the ratio of 1:1 for both branches for 3000 iterations with a batch size of 2. The performance of our model further improved after the fine-tuning, possibly due to the introduction to the domain of ICDAR documents. The results of this experiments are also compiled in tables and . Average time taken for our system for each document image is 0.3765 seconds, however this could not be compared with DeepDSert as their model was not publicly available. While the results are not conclusively better than DeepDSert, they are certainly comparable and the fact that our model is end-to-end means further improvements can be made with richer semantic knowledge, and additional branches for learning row based segmentation.

# CONCLUSION

This paper presents TableNet, a novel deep learning model trained on dual tasks of table detection and structure recognition in an end-to-end fashion. Existing approaches to information extraction treat detection and structure recognition as two distinct problems to be solved independently. TableNet is the first model to jointly address both tasks simultaneously, by exploiting the inherent interdependence between table detection and table structure identification. TableNet utilizes the knowledge from previously learned tasks and can transfer that knowledge to newer, related ones demonstrating transfer learning. This is particularly useful when the training data is sparse. We also show that highlighting the text to provide data-type information improves the performance of the model. In the future we intend to train TableNet with a third branch to identify rows, however this will require a manual annotation exercise as currently datasets do not provide for row based annotations. Another question that arises from these experiments is what other semantic knowledge could be provided for better model performance, perhaps the use of more abstract data types such as currency, country or city, might be useful.

[^1]: https://drive.google.com/drive/folders/1QZiv5RKe3xlOBdTzuTVuYRxixemVIOD