title: "Argument list too long" for every command
body: <p>Occasionally, when I have a program that generates large arrays I get this bug where every command throws the error 
  "Argument list too long"
  even if I just type:</p>
  
  <pre><code>$ cp
  -bash: /bin/cp: Argument list too long
  $
  </code></pre>
  
  <p>I can't use <code>ls</code>, or even open a new file with <code>vim</code>:</p>
  
  <pre><code>$ vim test.txt
  -bash: /usr/bin/vim: Argument list too long
  $
  </code></pre>
  
  <p>I tried using "wait" for all bg processes to finish, but no change.  It seems to happen inconsistently, but when it does, the only fix is to restart the shell.</p>
  
  <p>Any ideas what might be going on?</p>
  
  <p>Update:  I did some further testing and i got the error to be repeatable.  It happens when a recursively defined array reaches 85 elements in length.  The first command which throws the error is a <code>bc</code> that doesnt even depend on the array! and then from there on out, almost every other command throws the same error.</p>
  
  <p>Update:  The program I'm using has many bash scripts working together, but I've determined the problem always arises in this one: </p>
  
  <pre><code>function MPMDrun_prop()
  {
  PARDIR=$1
  COMPDIR=$2
  runSTR=$3
  NUMNODES=$4
  ForceRun=$5
  
  if [ $# -le 3 ] ; then
  echo "USAGE: MPMDrun_prop \$PARDIR \$COMPDIR \$runSTR \$NUMNODES \$ForceRun"
  fi
  echo "in MPMDrun_Prop"
  . $PARDIR/ParameterScan.inp
  . $MCTDHBDIR/Scripts/get_NumberOfJobs.sh
  
  if [ "$MPMD" != "T" ]; then
    MPMDnodes=1
  fi
  
  ## If no runscripts in the $PARDIR, copy one and strip of the line which runs the program
  if [ -z "$(ls $PARDIR/run*.sh 2&gt; /dev/null)"  ] ; then
  
   if [ "$forhost" == "maia" ]; then
     cp $MCTDHBDIR/../PBS_Scripts/run-example-maia.sh $PARDIR/run.tmp
     sed 's|mpirun.*||' &lt; $PARDIR/run.tmp &gt; $PARDIR/run.sh
     jobtime=86400
   elif [ "$forhost" == "hermit" ]; then
     cp $MCTDHBDIR/../PBS_Scripts/run-example-hermit.sh $PARDIR/run.tmp
     sed 's|aprun.*||' &lt; $PARDIR/run.tmp &gt; $PARDIR/run.sh
     jobtime=86400
   elif [ "$forhost" == "hornet" ]; then
     cp $MCTDHBDIR/../PBS_Scripts/run-example-hornet.sh $PARDIR/run.tmp
     sed 's|aprun.*||' &lt; $PARDIR/run.tmp &gt; $PARDIR/run.sh
     jobtime=86400
   elif [ "$forhost" == "bwgrid" ]; then
     cp $MCTDHBDIR/../PBS_Scripts/run-example-BWGRID.sh $PARDIR/run.tmp
     sed 's|mpirun.*||' &lt; $PARDIR/run.tmp &gt; $PARDIR/run.sh
     jobtime=86400
   fi
   sed 's|nodes=[0-9]*|nodes=0|' &lt; $PARDIR/run.sh &gt; $PARDIR/run.tmp
   sed 's|#PBS -N.*|#PBS -N MONSTER_'$MonsterName'|' &lt; $PARDIR/run.tmp &gt; $PARDIR/run.sh_
  
   rm $PARDIR/run.sh
   rm $PARDIR/run.tmp
   chmod 755 $PARDIR/run.sh_
   echo ". $MCTDHBDIR/Scripts/RunFlagSleeper.sh" &gt;&gt; $PARDIR/run.sh_
   ## Include check_convergence.sh for mixed relax/prop compatibility
   echo ". $MCTDHBDIR/Scripts/check_convergence.sh" &gt;&gt; $PARDIR/run.sh_
   echo "RunFlagSleeper $jobtime " &gt;&gt; $PARDIR/run.sh_
   echo "(" &gt;&gt; $PARDIR/run.sh_
   cp $PARDIR/run.sh_ $PARDIR/run1.sh
  fi
  ### Add $runSTR to the most recent runscript
  ### find runscript$N.sh (run1.sh, run 2.sh, etc)  that has numnodes less than $MPMDnodes
  for qq in $(ls $PARDIR/run[0-9]*.sh | sort -g ); do
    NodesInRun=$(cat $qq | grep -o "nodes *= *[0-9]*" | grep -o "[0-9]*")
    if [ "$NodesInRun" -lt "$MPMDnodes"  ]; then
     ## The number of nodes already specified in the runscript doesnt exceed the maximum, so add on another job
     NewNodes=$(echo "$NodesInRun+$NUMNODES" | bc)
     ## Start each aprun command in its own subshell
     ## wait for 24 hrs after aprun, to guarantee that no subshell finishes before the job is done
     sed 's|nodes=[0-9]*|nodes='$NewNodes'|' &lt; $qq &gt; $qq-1
     sed 's|\(RunFlagSleeper .*\)|\1 '$COMPDIR'|' &lt;$qq-1 &gt;$qq
     rm $qq-1
       echo "  (" &gt;&gt; $qq
       ## Sleeps for $jobtime - 5 mins, then removes runflag. in case aprun doesnt finish in $jobtime 
       echo "    cd $COMPDIR" &gt;&gt; $qq
       echo "    $runSTR" &gt;&gt; $qq
       ## remove runflag after aprun command has finished
       echo "    rm $COMPDIR/RunFlag" &gt;&gt; $qq
  #     echo "sleep $jobtime" &gt;&gt; $qq-1
       echo "  ) &amp;" &gt;&gt; $qq
  #   mv $qq-1 $qq
     ## put a flag in the computation directory so it isnt computed multiple times
     touch $COMPDIR/RunFlag
  
     if [[ "$NewNodes" -ge "$MPMDnodes" || "$ForceRun" == "T"  ]]; then
      ## This last process made the nodecount exceed the maximum, or there is a ForceRun flag passed
      ## So now, exceute the runscript and start another
      echo "  wait" &gt;&gt; $qq
      echo ") &amp;" &gt;&gt; $qq
      echo "PID=\$!" &gt;&gt; $qq
      echo "wait \$PID" &gt;&gt; $qq
      ## Ensure the queue has room for the next job, if not, wait for it
     Njobs=$(get_NumberOfJobs $runhost)
     while [ "$Njobs" -ge "$maxjobs" ]; do
      echo "Njobs=$Njobs and maxjobs=$maxjobs"
      echo "Waiting 30 minutes for que to clear"
      sleep 1800
     done
      echo "qsub $qq"
  #    qsub $qq
      RunCount=$(echo $qq | grep -o 'run[0-9]*.sh' | grep -o '[0-9]*')
      let "RunCount++"
      cp $PARDIR/run.sh_ $PARDIR/run$RunCount.sh
     fi
    fi
  done
  } 
  </code></pre>
  
  <p>The error typically starts at the 80-90'th call of this function at the first <code>cp</code> or <code>bc</code>.  I've commented ALL array manipulations, so there is zero chance this is caused by the array being too large.  The environment stays at ~100-200 Kb so that isn't the problem either.</p>