{'fraction_non_alphanumeric': 0.03620341472370112, 'fraction_numerical': 0.01020745364420782, 'mean_word_length': 4.619145863420672, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Activity Modeling in Email

We introduce a latent activity model for workplace emails, positing that communication at work is purposeful and organized by activities. We pose the problem as probabilistic inference in graphical models that jointly capture the interplay between latent activities and the email contexts they govern, such as the recipients, subject and body. The model parameters are learned using maximum likelihood estimation with an expectation maximization algorithm. We present three variants of the model that incorporate the recipients, co-occurrence of the recipients, and email body and subject. We demonstrate the model’s effectiveness in an email recipient recommendation task and show that it outperforms a state-of-the-art generative model. Additionally, we show that the activity model can be used to identify email senders who engage in similar activities, resulting in further improvements in recipient recommendation.


Introduction
Activities are a prominent characteristic of a workplace, typically governed by people's job roles and work responsibilities. Examples of workplace activities can include organizing a conference, purchasing equipment, managing candidate interviews, etc. Activities can be viewed as a collaborative work practice involving a set of people each playing a different role in the activity (Dredze et al., 2006).
Although emails are an integral part of workplace communication, current email clients offer little support for the activity oriented use of email (Khoussainov and Kushmerick, 2005). Discussions can get split across long email threads and communications about all activities can get intermixed, making activity management difficult (Balakrishnan et al., 2010).
In this work, we model activities as latent probability distributions personalized to the email sender. We present three variants of the activity model, incorporating: (1) email recipients, (2) email recipient pairs which account for co-occurrence of the email recipients, and (3) email body and subject tokens along with email recipient pairs. Additionally, we experiment with lexical (bag of words), syntactic (nouns and verb phrases), and semantic (things of interest in an email) representations of the body and subject tokens of an email. The parameters of the generative model are learned using an expectation maximization (EM) algorithm.
For evaluation, we formulate a real world task setting for email recipient recommendation, where we assume that all but the last recipient of an email has been entered by the sender, and we test the effectiveness of our activity model in recommending the last recipient. Such a system has practical applications, such as reminding an email sender about a potentially forgotten recipient or recommending the next recipient as the sender enters each recipient.
The main contributions of our research are: • We introduce a latent activity model for emails where the email contexts are governed by workplace activities; • We present probabilistic modeling that incorporates co-occurring recipients with lexical, syntactic and semantic contexts of an email; • We identify senders engaging in similar activities using the activity model, and show improvements in recipient recommendation.

Related Work
Prior research related to our work can be divided into the following three major areas presented below.

Activity in Emails
Prior research has treated emails as a communication tool for workplace activities (Moran, 2005) or a task management resource (Bellotti et al., 2003). Kushmerick and Lau (2005) formalized e-commerce activities as finite-state automata, where transitions among states represent messages sent between participants. Dredze et al. (2006) used user generated activity labels and classified emails into activities using overlapping participants and content similarity. Minkov et al. (2008) modeled user created folders and TO-DO items as activities, and created a heterogeneous graph to perform activity-centric search. Shen et al. (2006) predicted tasks associated with an incoming email by leveraging email sender, recipients and distinct subject words. They found the body words to not provide additional prediction value. Although they used similar information as we do, they used a combination of generative and discriminative models toward task classification, and did not do recipient recommendation. Our activity model designs are closer to the model introduced by , who presented a Dirichlet process mixture model combined with author and thread information. Our designs differ as we use cooccurring recipients in the generative process, and use various linguistic representations of content.

Generative models for Emails
Latent Dirichlet Allocation (LDA) (Blei et al., 2003) is a frequently used generative topic model. Assuming a Dirichlet prior, LDA models learn probability distributions of words as latent topics in a corpus. In emails, LDA models have been used for learn-ing summary keywords , analyzing how topics change over time (Wang and Mc-Callum, 2006), understanding entity relations (Balasubramanyan and Cohen, 2011), analyzing communication networks (Nguyen et al., 2013), for authorship attribution (Seroussi et al., 2012), and discovering topics associated with authors (McCallum et al., 2005).
Other generative models have also been used for analyzing email communication behavior (Navaroli et al., 2012), identifying links between an email sender and a recipient to detect potential anomalous communication (Huang and Zeng, 2006), and resolving personal names in emails (Elsayed et al., 2008). Representing workplace activities of the emails with probabilistic inference in graphical models where observed information is personalized to the email senders is what sets our work apart from previous research in computational models for emails.

Email Recipient Recommendation
For recommending email recipients, interactions among email participants and content similarity are the signals that have been explored most. Carvalho and Cohen (2007) leveraged content similarity by creating tf-idf centroid vectors and determining k-nearest neighbors of a target email. Pal et al. (2007) presented a discriminative author recipient topic model that uses transfer learning. Desai and Dash (2014) used reverse chronologically arranged implicit groups determined from sent emails. Sofershtein and Cohen (2015) created a ranking function combining temporal and textual features.
Among the generative modeling based approaches, Pal and McCallum (2006) learned probability distributions of recipients, and words in body and subject to predict recipients in email cc lists.  evaluated the impact of summary keywords generated using LDA, for email recipient prediction. More recently, Graus et al. (2014) predicted email recipients by estimating sender and recipient likelihood using a communication graph, email likelihood using content words, and evaluated performance on the Avocado email corpus. In our work, we use latent activity distributions, and identify senders who engage in similar activities. We compare our recipient prediction results against the generative model of Graus et al. (2014).

Activity Modeling in Emails
Our motivation for activity modeling in email stems from the assumption that in the workplace, people primarily use emails as a communication tool for their ongoing activities, and an email's recipient list, content, and other context are governed by a given activity. For example, an employee attending a conference may write emails to the conference organizers regarding registration or scheduling, or emails to a hotel for booking confirmation. The communication may span multiple emails, involving many parties, but all under the same activity.
We model the activities as a latent probabilistic variable over the email recipients and content, personalized to the email sender. Let D be the set of all emails in a corpus containing N emails, generated by S = {s i | 1 ≤ i ≤ S D } senders, and sent to represent the body and subject vocabulary of the emails respectively. Let K be the number of latent activities for each sender. We model the activities as probability distributions over email components S, R, B and T .

Corpus Description and Data Sets
For our experiments, we use the Avocado email corpus, available from the LDC catalog. 1 The corpus contains emails from a defunct IT company referred to as "Avocado". For learning the activity model, we extract emails from 7/1/2000 -5/1/2001 to create a training data set, and from 5/1/2001 -6/30/2001 for development/tuning. In this work, we did not consider threaded conversations, only retained the first email in a thread and discarded the rest.
As additional filtering steps, we only kept emails written by the Avocado employees, allowing us to confine the scope of the activities within the company. To control sparsity and noise, for each email, we enforced a a minimum of two recipients, and a maximum of ten recipients. 2 In a practical system, we consider it reasonable that a model only activates after some history of email is sent by a user. We therefore removed emails by the senders having fewer than 25 emails in the training data. From email bodies and subjects, we removed stopwords, and words appearing fewer than 5 times or more than 100 times. 3 When a recipient's same email alias was present multiple times, we took it only once, as well as removed a sender's email alias from the recipients list if it was present there. In this work, we focused on recipients from only the "TO" field, and did not include recipients from the "CC" or "BCC" field. Table 1 presents statistics of the data sets.

Activity Models
Our key assumption in modeling the activities in email is that different components of an email contain specific types of information that can help to characterize the activities that drive user behavior. In our generative process of the activity model, for an email d ∈ D, a sender s ∈ S is first generated from a multinomial distribution with probability vector σ, then an activity a is generated from a sender personalized multinomial distribution with probability vector θ s . Let R d ⊆ R, B d ⊆ B and T d ⊆ T be the set 4 of recipients, body and subject tokens of d respectively. The generation of the email contexts (recipients and body/subject tokens) varies based on the specific design of each variant of our model. In a first simplistic model, we assume that recipient r ∈ R d , body token b ∈ B d and subject token t ∈ T d for an email can be generated from the multinomial distributions with probability vectors λ s,a , φ s,a , and τ s,a respectively, that are conditioned s and a. Point estimates for σ can be directly less directly relevant to an employee's activities. Some emails in the Avocado dataset have more than 500 recipients. 3 A heuristic we used to remove words that are too general. 4 Multiset for body and subject tokens. calculated from a training corpus, whereas θ, λ, φ, and τ are the unknown parameters of the model.

Model 1: Rec
In our first model Rec, we assume that the latent activities can be learned as a probability distribution over the recipients alone. The generative process is: Figure 1 presents the plate diagram of the model. The joint probability of the Rec model is the product of the conditional distributions: The probability of a sender s, an activity a given s, and a recipient r given s and a are defined below 5 : We apply Bayes' rule to find the posterior distribution over the activities P n (a|d), which is directly proportional to the joint distribution P n (a, d).
We can exactly compute this distribution by evaluating the joint distribution for every value of a and the observed document d n .
Learning: Point estimates for σ can be directly obtained from the training corpus. We estimate the 5 I is an indicator variable parameters θ and λ by maximizing the (log) probability of observing D. We write the log(D) as: We use the Expectation-Maximization (EM) algorithm to set the parameters. Starting with a random initialization of the parameters (with Gaussian noise), EM iterates between the E-step in which P n (a|s, R d ) is computed for each email with fixed parameter values computed in the previous M-step, and the M-step in which the parameters are updated with fixed P n (a|s, R d ) values computed in the Estep. The parameter updates are obtained by taking the derivative of log P (D) with respect to each parameter, and setting the resultant to 0, providing us with the following parameter updates: a P n (a|d) We run EM until the change in log P (D) is less than our convergence threshold 10 −5 .

Model 2: CoRec
Using co-occurring recipients in generative models for emails has been rarely explored in previous work. Pal and McCallum (2006) modeled corecipient information as a probability distribution of recipients conditioned on the other recipients, and noted that this information improved their email cc prediction performance. In our CoRec model, we model co-recipients as pairs of recipients generated from a probability distribution conditioned on the sender and the activity.
having L D pairs of recipients in the corpus. For an email d, L d ⊆ L is the set of recipient pairs in d. The CoRec model first generates a sender s from the probability distribution σ, then an activity a from a distribution over latent activities θ s personalized to s, and finally recipient pairs r p ∈ L d from a distribution over recipient pairs ω s,a conditioned on s and a. The generative process is summarized below:  The joint probability of the CoRec model is: P (s, a, r p |σ, θ, ω) = P (s|σ)P (a|s, θ) rp∈L d P (r p |s, a, ω) This adds over the Rec model the probability of a recipient pair r p given s and a, defined below: The EM algorithm is applied in the same way as in the Rec model. During the M-step, update for θ remains the same. The update for ω is given below: a P n (a|d)

Model 3: CoRecBT
Finally, in the CoRecBT model, we further incorporate body and subject of emails. The generative process of the model is: The joint probability of the CoRecBT model: P (s, a, r p , b, t|σ, θ, ω, φ, τ ) = P (s|σ)P (a|s, θ) where the probability of a body token b and subject token t given s and a defined as: During the M-step of the EM algorithm, updates for θ and ω remain the same as the CoRec model. The updates for φ and τ are given below: a P n (a|d)

Subject and Body Token Representations
Previous work in modeling email content mostly explored bag of words (e.g., (Graus et al., 2014)) or tf-idf vectors (e.g., (Carvalho and Cohen, 2007)) as the content representation of an email. For modeling activities in emails, we experiment with different linguistic representations of the email content. They are: • Lexical: as the lexical representation, we use the bag of words (BOW) from email body and subject, after Penn Tree Bank (PTB) style tokenization.
• Syntactic: using heuristics on the output of a PTB constituent parser (Quirk et al., 2012), we identify Nouns (N) and Verb Phrases (VP) in email body and subject.
• Semantic: we identify phrases in emails that represent topics, concept and entities discussed in the emails. We refer to them as Thing of Interest (TOI). To extract these key phrases, we use Web search queries as a source of information. Using queries as a dictionary of possible key phrases is useful but has limited coverage since many topics/concepts are discussed in emails but absent or not widely available in Web search queries. Instead of using queries directly, we use them to construct a training set of key phrases and their content and we train a discriminative model to identify the key phrases. We treat each query as a key phrase and the surrounding text from Web results as context. We use a sample of hundreds of thousands of search queries from the usage logs of a commercial Web search engine. Only queries tagged as English and from the United States locale were retained to remove geographic or linguistic variations. Queries were kept only if they have been submitted by at least 100 different users in one month. For each query, the text from the Web page that is most relevant to the query and that contains the exact query text is collected as the context for the query. Relevance is estimated by the percentage of time the page has received a long dwell time click (greater than 30 seconds) for the query. If no relevant pages exist, the query is ignored. To generate negative examples, random n-grams were extracted from web pages. We experimented with a large number of features including: first word of the phrase, last word of the phrase, n-gram features (n=1 to 3), the word right before/after the phrase, the part-of-speech tag of the first word in the phrase, the part-ofspeech tag of the last word in the phrase, ngram features (n ranges from 1 to 3) over the sequence of part-of-speech tags representing the phrase and the part-of-speech tags of the word right before/after the phrase, phrase length, how many times it appeared in the body/title, and the relative location of the first occurrence of the phrase in the body. We trained a logistic regression classifier using these features and the data described above. The trained classifier is then applied to our email data to identify the Thing of Interest (TOI) phrases.

Discussion
Full Bayesian Treatment: In the above models, we learn point estimates for the parameters (σ, θ, ω, φ, τ ). One can take a Bayesian approach and treat these parameters as variables (for instance, with Dirichlet prior distributions), and perform Bayesian inference. However, exact inference will become intractable and we would need to resort to methods such as variational inference or sampling. We found this extension unnecessary, as we had a sufficient amount of training data to estimate all parameters reliably. In addition, our approach enabled us to learn (and perform inference in) the model with large amounts of data with reasonable computing time.

Recipient Recommendation
To evaluate the effectiveness of our activity model, we formulate a recipient recommendation task.
Task Definition: For a test email document d containing the list of recipients R d , a modified list of recipients R * d is created by removing the last recipient r * ∈ R d . Given d with R * d , the task objective is to recommend r * as the next recipient for d.

Our Methods
To recommend a recipient for a test email document d written by sender s d , we first create a candidate recipient list by combining recipients who received an email from s d , and recipients who co-occurred with an observed recipient r ∈ R * d in the training corpus. Sender s d and any r ∈ R * d are excluded from the candidate list. Next, we determine the probability distribution of the activities in d using: a P (s, a, d|σ, θ, ω, φ, τ ) Each candidate recipient r * is then ranked by a score using two different methods defined below. The ranked list is used as our final recommended recipients. The two scoring methods are: Reg Method: In the Reg method, we score using the chain rule 6 :

r|s, a)
We smooth the above function using the following linear interpolation: P (r * , r|a, s) = α 1 × P (r * , r|a, s) + (1 − α 1 )× (α 2 × P (r * , r) + (1 − α 2 ) × P (r rare )) Here, P (r rare ) is the lowest probability of any recipient in the training data. We calculate α i with a sigmoid logistic function, allowing us to determine when to rely more on the learned probabilities: x is the pointwise mutual information (PMI) between s and r in training data, with steepness parameter k = 50. For α 2 , x is the frequency of r in training data, with k = .5. Sigmoid's midpoint 6 Scoring function for the Rec model uses P (r * |s, a) x 0 is the first quartile (Q1) of the PMI and recipient frequency distributions respectively. The above values for k have been determined from the shape of the sigmoid curves in the training data.
Sim Method: In the Sim method, we explore the idea that the activity model can be used to identify other senders with similar activities as s d , who we refer to as similar senders, S * d . To identify the similar senders, we evaluate senders who maximize the log likelihood of the test document d by calculating log P (s, d) for all s ∈ S, and identify the top 5 with the highest scores to add to S * d . The observed sender s d is not included in S * d . We then calculate P s (r * |d) for each s ∈ S * d using the Reg method, along with a weight w s : The final scoring function for the Sim method is: Here, α is determined with the frequency of s d in training data, using the sigmoid function with k = 0.5 and x 0 as the Q1 of the frequency distribution.

Baseline Systems
As simple baseline systems to compare with our methods, we use 1) a random recipient baseline; 2) ranked recipients by P (r = r * ); and 3) ranked recipients by P (r = r * |s = s d ), where the probabilities are calculated from the training data. We evaluate two additional generative baselines using 4) P (r = r * |R * d ), and 5) P (r = r * |s = s d , R * d ) by applying Bayes' theorem, and assuming conditional independence among r ∈ R * d . For these methods, we used similar interpolation smoothing as before.
We additionally implemented the generative model presented by Graus et al. (2014) for recipient recommendation, which for test email d uses: Graus estimated P (d|r * , s d ) by P (b|r * , s d ) where b is an observed term in the email. The evaluation task was different from ours as they predicted all recipients of an email. In our evalua-  tion task, we recommend the last recipient, allowing us to use the already observed recipients R * d for estimating P (d|r * , s d ). Consequently, we present 3 additional baselines adopting Graus' method: 6) GrausB(BOW ) method uses body words, 7) GrausB(V P − T OI) uses the verb phrases and things of interest, and finally 8) GrausR method uses R * d for estimating P (d|r * , s d ). GrausR is equivalent to how we calculate our fifth baseline, P (r = r * |s = s d , R * d ), with the only difference of the smoothing function.
6 Experimental Results

Experimental Setup
To evaluate recipient recommendation, we create a test data set by extracting emails from 7/1/2001 -8/31/2001 from the Avocado data set. First we train our activity model with the training data and determine the optimum number of activities for each method by evaluating recipient recommendation on the development data. The number of activities per model is shown in Table 3. We then combine training and tuning data to create a new training data set in order to minimize the time difference between training and test emails. From the test data, we removed emails that had a sender or recipient never appearing in the training data. Although this lim-its the scope of the recipient recommendation evaluation task, predicting a recipient for a sender who never appeared in the training data is beyond our current modeling scope and practical settings.  With the ranked lists of recipients generated by each method, we calculate precision@X (X= 1, 2, 5, 10), and MRR (Mean Reciprocal Rank). Preci-sion@X is defined as percentage of emails having the actual recipient in the top X ranked recipients. Table 2 presents the recipient recommendation results for different methods. The first 5 rows show that the generative baselines from row (4) and (5) performed much better than the simple baselines (row (1)-(3)), yielding up to .2871 M RR. Comparatively, the GrausB(BOW) baseline in row (6) that uses body words, did not perform well, which is consistent with the finding by Shen et al. (2006) about body words not providing additional value in their task classification work. However, the GrausB(VP-TOI) in row (7) shows that using body terms more selectively has the potential for improving performance. Comparatively, the use of observed recipients (GrausR in row (8)) substantially improved recommendation results, yielding the highest M RR, precision@1, and precision@2 scores, while the generative baseline in row (5) retained the highest precision@5 and precision@10 scores.

Recipient Recommendation Results
Next, rows (9) to (13) show results for the activity models that use the Reg scoring. First, the Rec model outperformed the simple baselines from rows (1) to (3) as well as the GrausB methods from rows (6) and (7), but did not perform better than the generative baselines from rows (4) and (5) or GrausR. All the CoRec models performed better at recommending a recipient at top of the ranked lists with higher precision@1 and precision@2 scores, which are more practically useful for recommendation purposes, and also resulted in higher M RR scores.
Finally, the rows (14) and (15) show the results with the Sim scoring, and we observe a substantial improvement across the board, with verb phrase and thing of interest as the body context yielding our best results. This model achieved 3.31% additional improvements in MRR, and 3.38% additional improvements in precision@1 over the best baseline results. This demonstrates that the learned activity model can be used to identify senders who are likely to engage in similar activities, improving recipient recommendation performance further. Figure 2 shows examples of activity tokens from the emails of a sender in our training corpus, using word clouds. This is meant to serve as a case analysis, but it is not straightforward to interpret word clouds. When inspecting, we found that the names of potential customers (Nokia, Siemens, SAP) in the first example are prominent in some of the emails of the sender in the raw data. The recipients in these emails form a small cluster of people who are mainly involved in discussions around a particular event (Mobile Business Forum) where these companies are amongst the sponsors.
The second example, from the same sender, shows a coherent set of recipients. But in this case, the model seemed to have conflated multiple topics (such as the Palm VII device and support issues). We suspect that the cause for this confusion lies in the strong and coherent cluster of recipients which forces divergent topics to coalesce. While the combined signals of co-recipients and topic words improve the overall activity model, in some of the individual cases it leads to one signal improperly dominating the other.

Conclusion and Future Work
We presented a latent activity model for workplace emails where the activities are modeled as probability distributions over email recipients and other contexts, personalized to the email sender. Our model incorporates co-occurring recipients as part of the generative process, and can be used to identify senders who participate in similar activities, resulting in improved performance in email recipient recommendation. Our experiments suggest that syntactic and semantic knowledge such as verb phrases and thing of interests in emails can model the activities much better than bag-of-words, as demonstrated by the recipient recommendation results. Learning topics and sub-activities under workplace activities is a promising research direction which we will explore in future work.