title: /usr/xpg4/bin/grep -q [^0-9] does not always work as expected
body: <p>I have a Unix ksh script that has been in daily use for years (kicked off at night by the crontab).  Recently one function in the script is behaving erratically as never happened before.  I tried various ways to find out why, but have no success.</p>
  <p>The function validates an input string, which is supposed to be a string of 10 numeric characters.  The function checks if the string length is 10, and whether it contains any non-numeric characters:</p>
  <pre class="lang-sh prettyprint-override"><code>#! /bin/ksh
  
  # The function:
  is_valid_id () {
     # Takes one argument, which is the ID being tested.
     if [[ $(print ${#1}) -ne 10 ]] || print &quot;$1&quot; | /usr/xpg4/bin/grep -q [^0-9] ; then
        return 1
     else
        return 0
     fi
  }
  
  cat $input_file | while read line ; do
     id=$(print $line | awk -F: '{print $5}')
     # Calling the function:
     is_valid_id $id
     stat=$?
     if [[ $stat -eq 1 ]] ; then
        print &quot;The ID $id is invalid.  Request rejected.\n&quot; &gt;&gt; $ERRLOG
        continue
     else
        ...
     fi
  done
  </code></pre>
  <p>The problem with the function is that, every night, out of scores or hundreds of requests, it finds the IDs in several requests as invalid.  I visually inspected the input data and found that all the &quot;invalid&quot; IDs are actually strings of 10 numeric characters as should be.  This error seems to be random, because it happens with only some of the requests.  However, while the rejected requests persistently come back, it is consistently the same IDs that are picked out as invalid day after day.</p>
  <p>I did the following:</p>
  <ol>
  <li>The Unix machine has been running for almost a year, therefore might need to be refreshed.  The system admin to reboot the machine at my request.  But the problem persists after the reboot.</li>
  <li>I manually ran exactly the same two tests in the function, at command prompt, and the IDs that have been found invalid at night are all valid.</li>
  <li>I know the same commands may behave differently invoked manually or in a script.  To see how the function behaves in script, the above code excerpt is the small script I ran to reproduce the problem.  And indeed, some (though not all) of the IDs found to be invalid at night are also found invalid by the small trouble-shooting script.</li>
  <li>I then modified that troubleshooting script to run the two tests one at a time, and found it is the <code>/usr/xpg4/bin/grep -q [^0-9]</code> test that erroneously finds some of the ID as containing non-numeric character(s).  Well, the IDs are all numeric characters, at least visually.</li>
  <li>I checked if there is any problem with the xpg4 grep command file (<code>ls -l /usr/xpg4/bin/grep</code>),  to see if it is put there recently.  But its timestamp is year 2005 (this machine runs Solaris 10).</li>
  <li>Knowing that the data comes from a central ERP system, to which data entry is performed from different locations using all kinds of various terminal machines running all kinds of possible operating systems that support various character sets and encodings.  The ERP system simply allows them.  But can characters from other encodings visually appear as numeric characters but the encoded values are not as the <code>/usr/xpg4/bin/grep</code> command expects to be on our Unix machine?  I tried the <code>od</code> (octal dump) command but it does not help me much as I am not familiar with it.  Maybe I need to know more about <code>od</code> for solving this problem.</li>
  </ol>
  <p>My temporary work-around is omitting the <code>/usr/xpg4/bin/grep -q [^0-9]</code> test.  But the problem has not been solved.  What can I try next?</p>