title: Email spam classification extracting features from header
body: <p>I am trying to build a spam classifier. I have been reading some of the research papers and along with adding content based features I am also trying to add header field features for e.g. number of BCC recipients, subject, sender etc, however I am stuck at one particular place:</p>
  
  <ul>
  <li>I need to check the legitimacy of the sender domain address. I am
  writing all my code in R and I am not very sure how can I check that
  using R.</li>
  <li>I am also trying to extract the X-Mailer field, which is not a
  difficult task. However, the problem with X-mailer is that if its not
  present then it is a good indication that the email is a spam,
  however the problem arises when spammers try to obfuscate the
  X-mailer and fill it with garbled text, how can I differentiate
  between these two types of data - garbled X-mailer content and legitimate X-mailer.</li>
  <li>Similarly I am trying to create features like: "domain_legality"
  legality of the domain   of the sender,"date_time_legality" the
  legality of the date and time the message was created and received,
  "IP_legality" IP of receiver, and "sender_legality", which are some what
  self explanatory.</li>
  </ul>
  
  <p>I thank you for your time and consideration.</p>
  
  <p>So here is a sample of my code what I am trying to do:</p>
  
  <pre><code>extract_header &lt;- function(email.data){
    header.features &lt;- data.frame(matrix(ncol = 13))
    email.regex &lt;- "[[:alnum:].-]+@[[:alnum:].-]+" #regular expression to extract from email address
    colnames(header.features) &lt;- c("rec_field_num_of_hops", "span_time", "domain_legality", "date_time_legality", "IP_legality", "sender_legality", "num_of_To_receivers", "num_of_CC_receivers", "num_of_BCC_receivers", "mail_agent", "email_subject", "date_received")
    for(i in 1:length(email.data)){
      #extracting the email address of the sender
      header.features$sender_legality[i] = str_match(email.data[[i]]$meta$author, email.regex)
  
      #the subject of the email
      header.features$email_subject[i] = email.data$meta$heading
  
      #number of To receipients of the email
      posToField = which(!is.na(str_match(email.data[[i]]$meta$header, ignore.case("^To:"))))
      if(length(posToField) &gt; 0)
        header.features$num_of_To_receivers[i]  = sum(str_count(email.data[[i]]$meta$header[posToField], email.regex))
      else
        header.features$num_of_To_receivers[i]  = 0
  
      #number of people CC in the email
      posCCField = which(!is.na(str_match(email_corpus[[i]]$meta$header, ignore.case("^Cc:"))))
      if(length(posCCField) &gt; 0)
        header.features$num_of_CC_receivers[i] = sum(str_count(email.data[[i]]$meta$header[posCCField], email.regex))
      else
        header.features$num_of_CC_receivers[i] = 0
  
      #number of the Bcc people in the email
      posBccField = which(!is.na(str_match(email_corpus[[i]]$meta$header, ignore.case("^Bcc:"))))
      if(length(posBccField) &gt; 0)
        header.features$num_of_BCC_receivers[i] = sum(str_count(email.data[[i]]$meta$header[posBccField], email.regex))
      else
        header.features$num_of_BCC_receivers[i] = 0
  
      #number of email servers hopped by
      header.features$rec_field_num_of_hops[i] &lt;- sum(str_count(email_corpus[[i]]$meta$header, "^Received: from"))
  
    }
  }
  </code></pre>
  
  <p>I am following the approach laid out in the research papers:</p>
  
  <ul>
  <li>A scalable intelligent non-content-based spam-filtering framework</li>
  <li>Identifying Potentially Useful Email Header Features for Email Spam Filtering</li>
  </ul>
  
  <p>I need to check if the sender of the email was a legit sender, the rationale behind doing this is that most of the times spammers spoof their email address, and this particular feature helps in identifying whether the email is a spam or not.</p>
  
  <p>Header:</p>
  
  <pre><code>From rpm-list-admin@freshrpms.net  Tue Oct  8 10:56:20 2002
  Return-Path: &lt;rpm-zzzlist-admin@freshrpms.net&gt;
  Delivered-To: zzzz@localhost.example.com
  Received: from localhost (jalapeno [127.0.0.1])
      by example.com (Postfix) with ESMTP id 79DB116F16
      for &lt;zzzz@localhost&gt;; Tue,  8 Oct 2002 10:56:20 +0100 (IST)
  Received: from jalapeno [127.0.0.1]
      by localhost with IMAP (fetchmail-5.9.0)
      for zzzz@localhost (single-drop); Tue, 08 Oct 2002 10:56:20 +0100 (IST)
  Received: from egwn.net (ns2.egwn.net [193.172.5.4]) by
      dogma.slashnull.org (8.11.6/8.11.6) with ESMTP id g988mPK07565 for
      &lt;zzzz-rpm@example.com&gt;; Tue, 8 Oct 2002 09:48:25 +0100
  Received: from auth02.nl.egwn.net (localhost [127.0.0.1]) by egwn.net
      (8.11.6/8.11.6/EGWN) with ESMTP id g988i1f16827; Tue, 8 Oct 2002 10:44:02
      +0200
  Received: from chip.ath.cx (cs146114.pp.htv.fi [213.243.146.114]) by
      egwn.net (8.11.6/8.11.6/EGWN) with ESMTP id g988hGf13093 for
      &lt;rpm-list@freshrpms.net&gt;; Tue, 8 Oct 2002 10:43:16 +0200
  Received: from chip.ath.cx (localhost [127.0.0.1]) by chip.ath.cx
      (8.12.5/8.12.2) with ESMTP id g988hASA018848 for &lt;rpm-list@freshrpms.net&gt;;
      Tue, 8 Oct 2002 11:43:10 +0300
  Received: from localhost (pmatilai@localhost) by chip.ath.cx
      (8.12.5/8.12.5/Submit) with ESMTP id g988h9j2018844 for
      &lt;rpm-list@freshrpms.net&gt;; Tue, 8 Oct 2002 11:43:10 +0300
  X-Authentication-Warning: chip.ath.cx: pmatilai owned process doing -bs
  From: Panu Matilainen &lt;pmatilai@welho.com&gt;
  X-X-Sender: pmatilai@chip.ath.cx
  To: rpm-zzzlist@freshrpms.net
  Subject: Re: a problem with apt-get
  In-Reply-To: &lt;Pine.LNX.4.44.0210071231560.4199-100000@urgent.rug.ac.be&gt;
  Message-Id: &lt;Pine.LNX.4.44.0210081140130.18762-100000@chip.ath.cx&gt;
  MIME-Version: 1.0
  Content-Type: TEXT/PLAIN; charset=US-ASCII
  X-Mailscanner: Found to be clean, Found to be clean
  Sender: rpm-zzzlist-admin@freshrpms.net
  Errors-To: rpm-zzzlist-admin@freshrpms.net
  X-Beenthere: rpm-zzzlist@freshrpms.net
  X-Mailman-Version: 2.0.11
  Precedence: bulk
  Reply-To: rpm-zzzlist@freshrpms.net
  List-Help: &lt;mailto:rpm-zzzlist-request@freshrpms.net?subject=help&gt;
  List-Post: &lt;mailto:rpm-zzzlist@freshrpms.net&gt;
  List-Subscribe: &lt;http://lists.freshrpms.net/mailman/listinfo/rpm-zzzlist&gt;,
      &lt;mailto:rpm-list-request@freshrpms.net?subject=subscribe&gt;
  List-Id: Freshrpms RPM discussion list &lt;rpm-zzzlist.freshrpms.net&gt;
  List-Unsubscribe: &lt;http://lists.freshrpms.net/mailman/listinfo/rpm-zzzlist&gt;,
      &lt;mailto:rpm-list-request@freshrpms.net?subject=unsubscribe&gt;
  List-Archive: &lt;http://lists.freshrpms.net/pipermail/rpm-zzzlist/&gt;
  X-Original-Date: Tue, 8 Oct 2002 11:43:09 +0300 (EEST)
  Date: Tue, 8 Oct 2002 11:43:09 +0300 (EEST)
  </code></pre>
  
  <p>I hope these additional details help out. Thank you for help :).</p>