title: how to solve pgcc&openacc linker error "__pgi_uacc_multicorestart", "__pgi_uacc_multicoreend"
body: <p>I am trying to parallelize my program in C with OpenACC 2.5 on Ubuntu 16.04 LTS. After a simple modification which is just adding one line, I can compile all my .c files to .o files. In the linking step, pgcc compiler shows </p>
  
  <blockquote>
    <p>undefined reference to `__pgi_uacc_multicorestart'</p>
  </blockquote>
  
  <p>and </p>
  
  <blockquote>
    <p>undefined reference to `__pgi_uacc_multicoreend'</p>
  </blockquote>
  
  <p>. Google search shows nothing related to these error message. Please help me on this problem.</p>
  
  <p>Here is the information and source code related to my system and program. I try to post the essential part and please let me know if you need anything else.</p>
  
  <hr>
  
  <p>OS, software:</p>
  
  <pre><code>LSB Version:    core-9.20160110ubuntu0.2-amd64:core-9.20160110ubuntu0.2-noarch:printing-9.20160110ubuntu0.2-amd64:printing-9.20160110ubuntu0.2-noarch:security-9.20160110ubuntu0.2-amd64:security-9.20160110ubuntu0.2-noarch
  Distributor ID: Ubuntu
  Description:    Ubuntu 16.04.3 LTS
  Release:    16.04
  Codename:   xenial
  
  Using built-in specs.
  COLLECT_GCC=gcc
  COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/5/lto-wrapper
  Target: x86_64-linux-gnu
  Configured with: ../src/configure -v --with-pkgversion='Ubuntu 5.4.0-6ubuntu1~16.04.5' --with-bugurl=file:///usr/share/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-5-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
  Thread model: posix
  gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.5)
  
  pgcc 17.10-0 64-bit target on x86-64 Linux -tp haswell 
  PGI Compilers and Tools
  Copyright (c) 2017, NVIDIA CORPORATION.  All rights reserved.
  </code></pre>
  
  <p>.bashrc:</p>
  
  <pre><code>#CUDA
  export PATH=/usr/local/cuda/bin:$PATH;
  export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH;
  #####
  ulimit -s unlimited
  #####
  #Environment Modules
  source /usr/share/modules/init/bash
  module add /opt/pgi/modulefiles/pgi64/17.10
  module add /opt/pgi/modulefiles/openmpi/2.1.2/2017
  #####
  #intel compiler
  source /opt/intel/bin/compilervars.sh intel64
  #intel vtune
  source /opt/intel/vtune_amplifier/amplxe-vars.sh
  #intel advisor
  source /opt/intel/advisor/advixe-vars.sh
  #intel inspector
  source /opt/intel/inspector/inspxe-vars.sh
  #intel mkl
  source /opt/intel/mkl/bin/mklvars.sh intel64
  </code></pre>
  
  <p>makefile:</p>
  
  <pre><code>CC = pgcc
  CFLAGS_pgcc = -O0 -Minform=inform -Minfo -ta=multicore -g -pg -Mprof=time
  CFLAGS = $(CFLAGS_$(CC)) -c
  LFLAGS = $(LFLAGS_$(CC)) -L${MKLROOT}/lib/intel64 -lmkl_rt -lpthread -lm -ldl
  IFLAGS = $(IFLAGS_$(CC)) -I${MKLROOT}/include
  
  &lt;content is partially neglected&gt;
  
  serial: $(C_OBJ)
          $(CC)  $(IFLAGS) $(CFLAGS) -c msg_ser.c
          $(CC)  $(IFLAGS) -o dplbe $(C_OBJ) msg_ser.o $(LFLAGS)
  </code></pre>
  
  <p>error message:</p>
  
  <pre><code>lbe.o: In function `equilibrium_distrib':
  &lt;content is partially neglected&gt;lbe.c:548: undefined reference to `__pgi_uacc_multicorestart'
  &lt;content is partially neglected&gt;lbe.c:583: undefined reference to `__pgi_uacc_multicoreend'
  makefile:57: recipe for target 'serial' failed
  make: *** [serial] Error 2
  </code></pre>
  
  <p>lbe.c, to which I only add one line as a babystep to use OpenACC.</p>
  
  <pre><code>#include "header.h"
  extern int    max_x, max_y, max_z;
  extern int    num_x, x_min, x_max;
  extern int    num_proc, n_proc;
  extern double tau[2], tau_v[2];
  
  &lt;content is partially neglected&gt;
  
  void equilibrium_distrib(int xy, int z, double ***velcs_df, double dt,
       struct vector forceDen,  struct vector *correctedVel, double *f_eq)
  {
  
  &lt;content is partially neglected&gt;
  
  #pragma acc kernels
    {
    for(int q=0; q &lt; 19; q++)
    {
      double term1 = (c_x[q] * correctedVel-&gt;x + c_y[q] * correctedVel-&gt;y + 
                      c_z[q] * correctedVel-&gt;z)*3.;
      double term2 = 0.5*term1*term1;
      f_eq[q] = weight[q]*density*(1 + term1 + term2 - term3); 
    }
    }  
  }
  </code></pre>
  
  <p>compile lbe.c to lbe.o message:</p>
  
  <pre><code>pgcc-Warning--Mprof=time is not supported
  
  PGC-I-0222-Redundant definition for symbol __THROW (/usr/include/x86_64-linux-gnu/sys/cdefs.h: 74)
  PGC-I-0222-Redundant definition for symbol __extension__ (/usr/include/x86_64-linux-gnu/sys/cdefs.h: 358)
  lbe_zcol:
  
  &lt;content is partially neglected&gt;
  
  equilibrium_distrib:
      558, FMA (fused multiply-add) instruction(s) generated
      559, FMA (fused multiply-add) instruction(s) generated
      560, FMA (fused multiply-add) instruction(s) generated
      565, FMA (fused multiply-add) instruction(s) generated
      566, FMA (fused multiply-add) instruction(s) generated
      567, FMA (fused multiply-add) instruction(s) generated
      573, FMA (fused multiply-add) instruction(s) generated
      577, Loop is parallelizable
           Generating Multicore code
          577, #pragma acc loop gang
      580, FMA (fused multiply-add) instruction(s) generated
  </code></pre>