title: multiple including and excluding condition in regular expressions in r
body: <p>I have a data with over 10 million observations, with a string variable <code>dat_text</code>. I am trying to give an indicator variable (IV) for each observation. </p>
  
  <p>The IV will be 1 when the string <code>dat_text</code> <strong>includes</strong> any form that contains that any of the string in <code>pat_text</code> <strong>AND</strong> <strong>excludes</strong> any form that contains that any of the string in <code>ex_text</code> (Please refer to the EDIT for the meaning). I am trying to implement this in R.</p>
  
  <pre><code>dat_text &lt;- c("dbhgfadgdfgc", "sdfdsfsdgdfxgfydz", "fqdfsbfdjhdhts","dbhgfghfadgdfgc", "sdfdghsfsdgdfxgfydz", "fqdfsbfdjhfghdhts", "fdsafgdjfx", "dfdoslfspd")
  ex_text &lt;- c("fgh", "opl")
  pat_text &lt;- c("abc", "xyz", "jbq")
  </code></pre>
  
  <p>I have created a function with loop built in:</p>
  
  <pre><code>myfunction &lt;- function(pat_text, ex_text, dat_text){
  # =========PART 1: pat_text========================
  logic_tem &lt;- list()
  for(i in 1:length(pat_text)){# for each phrase in "pat_text"
    temp &lt;- list()
    for(t in 1:nchar(pat_text[i])){# for each character in the phrase
      temp[[t]] &lt;- grepl(substring(pat_text[i], t, t), dat_text) 
    }
    # Use "AND" to connect multiple logic vectors
    temp &lt;- do.call(cbind, temp)
    logic_tem[[i]] &lt;- (rowSums(temp) == dim(temp)[2L])
  }
  
  logic_tem &lt;- do.call(cbind, logic_tem)
  logic_pattext &lt;- rowSums(logic_tem) &gt; 0
  
  # =========PART 2: ex_text========================
  logic_tem &lt;- list()
  for(i in 1:length(ex_text)){# for each phrase in "ex_text"
    temp &lt;- list()
    for(t in 1:nchar(ex_text[i])){# for each character in the phrase
      temp[[t]] &lt;- grepl(substring(ex_text[i], t, t), dat_text) 
    }
    temp &lt;- do.call(cbind, temp)
    logic_tem[[i]] &lt;- (rowSums(temp) == dim(temp)[2L])
  }
  # Use "OR" to connect multiple logic vectors
  logic_tem &lt;- do.call(cbind, logic_tem)
  logic_extext &lt;- rowSums(logic_tem) &gt; 0
  
  # =========PART 3: combine the two parts=========
  return(logic_pattext &amp; !logic_extext)
  }
  </code></pre>
  
  <p>This function works well:</p>
  
  <pre><code>&gt; myfunction(pat_text, ex_text, dat_text)
  [1] FALSE  TRUE  TRUE FALSE FALSE FALSE FALSE FALSE
  </code></pre>
  
  <p>But  turns out to be rather slow and inefficient when I implement this on my original data. I am asking if anybody could provide any clue on improving the performance of this function in R? Thanks</p>
  
  <p><strong>EDIT</strong> It was my bad that I did not clear state "any permutation of any string". By that phrase, I actually means that the observation contains all letters in that string. If the <code>pat_text</code> is <code>fgh</code>, then the following phrases all meet the requirement:</p>
  
  <pre><code>"fgh", "00000f00000g00000h", "00000g00000h00000f", "000000h00000f00000g"
  </code></pre>
  
  <p>Actually the <code>0</code>s in the above pattern can be any characters or numbers. I realised that this question actually become a totally different question after this edition. I really didn't realized that it was a totally different question when I wrote that. I am so sorry.</p>