I'm pretty new to Perl (and multithreading), so I was wondering if someone would be willing to take a look at my script and give me any tips on increasing performance.MotivationThe script is designed to build several hundred C++ components, using multiple threads (one per component)DesignAt a high level, the script should follow this flow:Get parametersDefault the number of compilation and linking threads to useRead in a list of componentsThe format here would be something like, if you pass in 'SERVER', we have a file that maps that component name to ~40 directories that need builtLaunch 16 threads in 'listening' stateQue up the work to be performedDependent on the actions passed inCodeCommented throughout to clarify what I expect to be happeninguse warnings;use strict;use POSIX qw(ceil floor strftime);use Perl::OSType ':all';use Getopt::Long;use Pod::Usage;use List::Util qw(min max);use Log::Message::Simple qw(msg error debug);use File;use threads;use threads::shared;use Thread::Semaphore;use Thread::Queue 3.01 qw( );my $q = Thread::Queue->new();my @procs;my @comps;my @compsCopy;my @retCodes;share(@retCodes);my @failedComponents;share(@failedComponents);my $maxThreadsSem = Thread::Semaphore->new(0);my $compStr = ;my $MAKE_INVOCATION_PATH;my $MAKE;## Parse options and print usage if there is a syntax error,## or if usage was explicitly requested.GetOptions('help|?' => \my $help, man => \my $man, 'verbose' => \my $verbose, 'comps:s{,}' => \my @compList, 'COMPLIST:s' => \my $legacyCOMPLIST, 'action:s{1,3}' => \my @actionList,           'clthreads:i' => \my $clthreads, 'linkthreads:i' => \my $linkthreads) or pod2usage(2);pod2usage(1) if $help;pod2usage(-verbose => 2) if $man;#set silent flag for make$ENV{SL} = '@' unless $verbose;@compList = qw(SERVER TOOLS) if !@compList;@actionList = qw(clean depend build link) if !@actionList;#default thread counts### --> The assumption here is that compile consumes less resources than link,###     so I default to the number of cores for compile, and slightly lower for linkingmy $numCores;if(is_os_type('Windows')){  $numCores = `grep -c '^processor' \/proc\/cpuinfo`;}elsif(is_os_type('Unix')){  my $numCoresText = `lsconf | grep Processors`;  $numCoresText =~ m/(\d+)/;  $numCores = $1;}$clthreads = $numCores if !$clthreads;$linkthreads = ceil($numCores * .75) if !$linkthreads;SetMake();my $startTime = (strftime %m-%d-%Y %H:%M:%S, gmtime);#the workBuildInit(@ARGV);print Started At: \t$startTime\n;print Ended At: \t . (strftime %m-%d-%Y %H:%M:%S, gmtime) .\n;my $maxReturnCode = max @retCodes;print Highest Error Code: $maxReturnCode\n;print Components with failures: @failedComponents\n if @failedComponents; exit($maxReturnCode);#----------------------------------------------------------sub SetMake{  if ( $ENV{MAKECMD} ) {     $MAKE=$ENV{MAKECMD}  } else {     # -k option tells make to continue its attempt to build a component after failures.     # make no longer quits compiling an entire directory after 1 compilation error.     $MAKE=make -k -C;  }  $MAKE_INVOCATION_PATH = $ENV{PATH_CYGDRIVE} || $ENV{PATH};}sub BuildInit {   my $actionStr = ;   my $compStr = ;                                     ### --> Gets the list of directories for each 'component'    my @component_dirs;     foreach my $comp (@compList){      @component_dirs = (@component_dirs, GetDirs($comp));     }    ### --> Limit the number of threads that can work at one time    my $maxThreadsSem = Thread::Semaphore->new($clthreads);        print Action List: @actionList\n;    print Component List: @compList\n;    print Component Dir List: @component_dirs\n;    print Compile Threads: $clthreads\n;    print Link Threads: $linkthreads\n;    #---------------------------------------    #----   Setup Worker Threads  ----------     for (1..16) {      async {         while (defined(my $job = $q->dequeue())) {            worker($job);         }      };    }    #-----------------------------------    #----   Enqueue The Work  ----------     for my $action (@actionList){      my $sem = Thread::Semaphore->new(0);  ## used to lock execution between 'actions'      #manage our resources while linking      if ($action eq 'link'){        $maxThreadsSem = Thread::Semaphore->new($linkthreads);      }      elsif ($action eq 'build'){        $maxThreadsSem = Thread::Semaphore->new($clthreads);      }      if($action eq 'depend'){        $ENV{PATH} = $ENV{BUILD_PATH}/makedepend/bin; . $ENV{PATH};        system(rm dependlist) == 0 or die Failed on attempt to remove dependlist;        (system($MAKE $MAKE_INVOCATION_PATH/makedepend)) == 0 or die Could not build 'makedepend' - killing process;      }      ### --> perform 'action' on each 'component directory'      $q->enqueue([$_, $action, $sem, $maxThreadsSem]) for @component_dirs;      ### --> we don't want to move on to the next action before this action finishes      ### --> e.g., we don't want to start linking prior to compiling finishing      $sem->down(scalar @component_dirs);      print \n------>> Waiting for prior actions to finish up... <<------\n;    }    # Nothing more to do - notify the Queue that we're not adding anything else    $q->end();    for my $thread (threads->list())    { $thread->join(); }    return 0;}sub worker {    my ($job) = @_;    my ($component, $action, $sem, $maxThreadsSem) = @$job;    #used to throttle the build / link process according the to specified threads    $maxThreadsSem->down() if $action eq 'link' or $action eq 'build';    Build($component, $action);    $sem->up();    $maxThreadsSem->up() if $action eq 'link' or $action eq 'build';}sub Build {    my ($comp, $action) = @_;    my $cmd = $MAKE $MAKE_INVOCATION_PATH/$comp ;    my $retCode = -1;    #lack of reliable switch support in perl    if($action eq depend)    {$cmd .= $action >nul 2>&1} #suppress output    elsif($action eq clean)    {$cmd .= $action}    elsif($action eq build)    {$cmd .= 'l1'}    elsif($action eq link)    {$cmd .= ''} #add nothing; default is to link    else {die Action: $action is unknown to me.}    print \n\t\t*** Performing Action: \'$cmd\' on >>> $comp <<< *** if ($verbose || $action eq 'depend');    if ($action eq link){      # hack around potential race conditions -- will only be an issue during linking      my $tries = 4;      until ($retCode == 0 or $tries == 0){        last if ($retCode = system($cmd)) == 2; #compile error; stop trying        $tries--;      };    }    else{      $retCode = system($cmd);    }    return $retCode if $action eq depend; #stop here for dependlist building    lock(@retCodes);  #lock to keep thread-safe    push(@retCodes, ($retCode>>8));        #testing    if($retCode != 0){      if($verbose){        print \n\t\t*** ERROR IN $comp: $@ !! ***\n;        print \t\t*** Action: $cmd -->> Error Level:  . ($retCode >>8) . \n;      }      lock(@failedComponents);  #lock to keep thread-safe      push(@failedComponents, $comp);    }    return $retCode;}# searches the $strDataFile for the given $strComponentName, then # fills the @dir_array with the appropriate directories for that componentsub GetDirs {   # ... <SNIP> ...   # mostly irrelevant   return @dir_array;}#-----------------------------------------------------------------#----------------  Documentation / Usage / Help ------------------=head1 NAME  sample - Using GetOpt::Long and Pod::Usage=head1 SYNOPSIS  buildmom.pl [options]   options:     -help            Brief help message     -man             Full documentation     -comp s          The list of components to perform actions on (from your comps.dat) | space delimited     -COMPLIST        Legacy support for old build script     -action          A list of actions to perform on each component; executed in the order given  | space delimited                      Available actions include clean, depend, build, link     -clthreads       Number of threads used during compiling     -linkthreads     Number of threads used during linking     -verbose         Enables addition build output - may increase build time=head1 DESCRIPTIONB<buildmom.pl> is used to perform various actions on your Momentum C++ project. Parameters worth note:  -comps | [optional: defaulted to ALL] The list of components from your comps.dat file on which to have the           specified actions performed. Example usage:  buildmom.pl -comps FOUNDATION COMMON TRANSPORT SERVER  -COMPLIST | [optional: legacy support] This is the same as the `-comps` parameter, with the exception that it allows                legacy support for older parameter passing (e.g., -COMPLIST=FOUNDATION:COMMON:TRANSPORT:SERVER)  -action | [optional: defaulted to `clean depend build link`] The set of actions, in order, which you want performed             on the comps list. Available actions include:                 clean: Executes `make clean` for each component specified(excluding ffsfld)                depend: Builds the make depend list for each component specified                build: Builds each component specified (formerly 'the first pass')                link: Links each component specified (formerly 'the second pass')  -clthreads | [optional: defaulted to 100%] Specifies the number of threads to use during compilation  -linkthreads | [optional: defaulted to 75%] Specifies the number of threads to use during linking      { For certain actions, specifically clean and depend, <buildmom.pl> will assume 100% thread usage every time.        These actions are lightweight, and the performance loss from these actions is minimal. If you specify multiple actions        e.g., buildmom.pl -actions clean depend build link, the script will consume 100% CPU for a couple of minutes while performing        the `clean` and `depend` steps, and then throttle back to the specified number of threads.         Another example: buildmom.pl -actions clean depend build link -clthreads 2 -linkthreads 2                    On a 4 core non-HT system, you would see your CPU usage jump to 100% for a few minutes while                    we perform the `clean` and `depend` steps. Once compilation begins, your CPU usage will throttle                     back down to ~50% due to the number of threads specified. }=cutAssumptionsLike I said, I'm pretty new to both Perl and multithreading. The primary assumption I've made here that I'm really not sure about is the number of threads to use. I've throttled it based on the number of cores available on the machine - is that the correct way to go about it? I assumed that anything >#cores would just cause some thrashing and competition of resources; is that accurate?
Building hundreds of C++ components using multiple threads
performance;beginner;multithreading;perl