title: Benchmarktools' belapsed not working on symbol
body: <h1>Introduction</h1>
  
  <hr>
  
  <p>I have a directory with the following structure</p>
  
  <pre><code>--&gt; Report
  --&gt; Problems
      --&gt; PE_001
          --&gt; Julia
              PE_001.naive.jl
              PE_001.jl
          --&gt; Benchmarks
              test_001.txt
              test_002.txt
          --&gt; Results
      --&gt; PE_002
       .
       .
       .
      --&gt; PE_XXX
  --&gt; Benchmark
  </code></pre>
  
  <p>I am attempting to iterate over all the Julia files and benchmark them against the benchmarking data located under the top directory <code>Benchmark</code>. I do not want to have to <code>cd</code> into each directory and run <code>@ belapsed</code> from the julia commandline to time every function individually.</p>
  
  <p>To solve this problem I wrote the following code that is supposed to be located under <code>benchmarks</code> in the hierachy above. However I made it slightly simpler for illustrative purposes.</p>
  
  <h1>Attempt at solution</h1>
  
  <hr>
  
  <p><strong>EDIT</strong>: The code below does NOT follow the hierarchy outlined above. To quickly reproduce the error, the code below has been written in such a way that all the files can be placed in the same directory. </p>
  
  <p><strong><em>benchmark.jl</em></strong></p>
  
  <pre><code>include("PE_002.jl")
  
  using BenchmarkTools
  
  function get_file_path(PE=1)
      current_folder = pwd()
      PE_folder = "/PE_" * lpad(string(PE),3,"0")
      dirname(pwd()) * "/Problems" * PE_folder * "/Julia"
  end
  
  
  function include_files(PE_dir)
      for filename in readdir(PE_dir)
          if !startswith(filename, "benchmark")
              filepath = PE_dir * "/" * filename
              @everywhere include($filepath)
          end
      end
  end
  
  
  function benchmark_files(PE_dir)
      for filename in readdir(PE_dir)
          if !startswith(filename, "benchmark")
              f = getfield(Main, Symbol(filename[1:end-3]))
              # Produces an error
              println(@belapsed f())
          end
      end
  end
  
  # Works
  println(@belapsed PE_002())
  
  PE_dir = pwd()
  include_files(PE_dir)
  benchmark_files(PE_dir)
  </code></pre>
  
  <p><strong>PE_002.jl</strong></p>
  
  <pre><code>function PE_002(limit = 4*10^6)
      a, b = 0, 2
      while b &lt; limit
          a, b = b, 4 * b + a
      end
      div(a + b - 2, 4)
  end
  </code></pre>
  
  <p><strong><em>PE_002_naive.jl</em></strong></p>
  
  <pre><code>function PE_002_naive(limit=4 * 10^6, F_1=1, F_2=2)
      total = 0
      while F_2 &lt; limit
          if F_2 % 2 == 0
              total += F_2
          end
          F_1, F_2 = F_2, F_1 + F_2
      end
      total
  end
  </code></pre>
  
  <p><strong>test_001.txt</strong></p>
  
  <pre><code>0*10**(2**0)
  4*10**(2**0) 
  4*10**(2**1)
  4*10**(2**2)
  4*10**(2**3) 
  4*10**(2**4) 
  4*10**(2**5) 
  4*10**(2**6) 
  </code></pre>
  
  <hr>
  
  <h1>Question</h1>
  
  <p>Interestingly enough including the file <code>PE_002</code> and then running <code>@ belapsed</code> works, however obtaining the filename from the directory, turning it into a symbol, then trying to time it with <code>@belapsed</code> fails. </p>
  
  <p>I know <code>@elapsed works</code>, however, due to garbage collection it is not nearly accurate enough for my needs. </p>
  
  <p><strong>Is there a simple way to benchmark all files in a remote directory using BenchmarkTools or similar tools as accurate?</strong></p>
  
  <p>All I need is a single number representing mean / average running time from each file.  </p>
  
  <p>EDIT 2: Per request I have included the full error message below</p>
  
  <pre><code>~/P/M/Julia-belaps ❯❯❯ julia benchmark.jl
  
  9.495495495495496e-9
  ERROR: LoadError: UndefVarError: f not defined
  Stacktrace:
   [1] ##core#665() at /home/oisov/.julia/v0.6/BenchmarkTools/src/execution.jl:290
   [2] ##sample#666(::BenchmarkTools.Parameters) at /home/oisov/.julia/v0.6/BenchmarkTools/src/execution.jl:296
   [3] #_run#6(::Bool, ::String, ::Array{Any,1}, ::Function, ::BenchmarkTools.Benchmark{Symbol("##benchmark#664")}, ::BenchmarkTools.Parameters) at /home/oisov/.julia/v0.6/BenchmarkTools/src/execution.jl:324
   [4] (::BenchmarkTools.#kw##_run)(::Array{Any,1}, ::BenchmarkTools.#_run, ::BenchmarkTools.Benchmark{Symbol("##benchmark#664")}, ::BenchmarkTools.Parameters) at ./&lt;missing&gt;:0
   [5] anonymous at ./&lt;missing&gt;:?
   [6] #run_result#16(::Array{Any,1}, ::Function, ::BenchmarkTools.Benchmark{Symbol("##benchmark#664")}, ::BenchmarkTools.Parameters) at /home/oisov/.julia/v0.6/BenchmarkTools/src/execution.jl:40
   [7] (::BenchmarkTools.#kw##run_result)(::Array{Any,1}, ::BenchmarkTools.#run_result, ::BenchmarkTools.Benchmark{Symbol("##benchmark#664")}, ::BenchmarkTools.Parameters) at ./&lt;missing&gt;:0
   [8] #run#17(::Array{Any,1}, ::Function, ::BenchmarkTools.Benchmark{Symbol("##benchmark#664")}, ::BenchmarkTools.Parameters) at /home/oisov/.julia/v0.6/BenchmarkTools/src/execution.jl:43
   [9] (::Base.#kw##run)(::Array{Any,1}, ::Base.#run, ::BenchmarkTools.Benchmark{Symbol("##benchmark#664")}, ::BenchmarkTools.Parameters) at ./&lt;missing&gt;:0 (repeats 2 times)
   [10] macro expansion at /home/oisov/.julia/v0.6/BenchmarkTools/src/execution.jl:208 [inlined]
   [11] benchmark_files(::String) at /home/oisov/Programming/Misc/Julia-belaps/benchmark.jl:26
   [12] include_from_node1(::String) at ./loading.jl:569
   [13] include(::String) at ./sysimg.jl:14
   [14] process_options(::Base.JLOptions) at ./client.jl:305
   [15] _start() at ./client.jl:371
  while loading /home/oisov/Programming/Misc/Julia-belaps/benchmark.jl, in expression starting on line 36
  </code></pre>