title: PHP input filtering - checking ascii vs checking utf8
body: <p>I need to insure that all my strings are utf8. Would it be better to check that input coming from a user is ascii-like or that it is utf8-like?</p>
  
  <pre><code>//KohanaPHP
  function is_ascii($str) {
      return ! preg_match('/[^\x00-\x7F]/S', $str);
  }
  
  //Wordpress
  function seems_utf8($Str) {
      for ($i=0; $i&lt;strlen($Str); $i++) {
          if (ord($Str[$i]) &lt; 0x80) continue; # 0bbbbbbb
          elseif ((ord($Str[$i]) &amp; 0xE0) == 0xC0) $n=1; # 110bbbbb
          elseif ((ord($Str[$i]) &amp; 0xF0) == 0xE0) $n=2; # 1110bbbb
          elseif ((ord($Str[$i]) &amp; 0xF8) == 0xF0) $n=3; # 11110bbb
          elseif ((ord($Str[$i]) &amp; 0xFC) == 0xF8) $n=4; # 111110bb
          elseif ((ord($Str[$i]) &amp; 0xFE) == 0xFC) $n=5; # 1111110b
          else return false; # Does not match any model
          for ($j=0; $j&lt;$n; $j++) { # n bytes matching 10bbbbbb follow ?
              if ((++$i == strlen($Str)) || ((ord($Str[$i]) &amp; 0xC0) != 0x80))
              return false;
          }
      }
      return true;
  }
  </code></pre>
  
  <p>I did some benchmarking on 100 strings (half valid utf8/ascii and half not) and found that seems_utf8() tasks 0.011 while is_ascii only takes 0.001. But my gut is telling me that you get what you pay for and the utf8 checking would be a better choice.</p>
  
  <p>I'm planning on then doing something like this convert.</p>
  
  <pre><code>&lt;?php
  
  /* Example data */
  $string[] = 'hello';
  $string[] = 'asdfghjkl;qwertyuiop[]\zxcvbnm,./]12345657890-=+_)(*&amp;^%$#@!';
  $string[] = '';
  $string[] = 'accentué';
  $string[] = '»á½µÎ½Ï‰Î½ Ï„á½° ';
  $string[] = '???R??=8 ????? ++++¦??? ???2??????';
  $string[] = 'hello¦ùó 5/5¡45-52ZÜ¿»'. "0x93". octdec('77'). decbin(26). "F???pp?? ??? ". '»á½µÎ½Ï‰Î½ Ï„á½° ';
  
  
  $time = microtime(true);
  
  //Count the successes
  $true = array(1 =&gt; 0, 0 =&gt; 0);
  
  foreach($string as $s) {
      $r = seems_utf8($s);    //0.011
  
      print_pre(mb_substr($s, 0, 30). ' is '. ($r ? 'UTF-8' : 'non-UTF-8'));
  
  
      if( ! $r ) {
  
          $e = mb_detect_encoding($s, "auto");
  
          print_pre('Encoding: '. $e);
  
          //Convert
          $s = iconv($e, 'UTF-8//TRANSLIT', $s);
  
          print_pre(mb_substr($s, 0, 30). ' is now '. (seems_utf8($s) ? 'valid' : 'not'). ' UTF-8');
      }
  
  }
  
  print_pre($true);
  print_pre((microtime(TRUE) - $time). ' seconds');
  
  function print_pre() { print '&lt;pre&gt;'; print_r(func_get_args()); print '&lt;/pre&gt;'; }
  </code></pre>