title: How to benchmark two different versions of the same non-NuGet library in BenchmarkDotNet?
body: <p>I'm trying to use BenchmarkDotNet to prepare performance regression testing for a library. This requires that I compare the same tests with an older (stable) version of the library. Now, there's an option to give <a href="https://benchmarkdotnet.org/articles/samples/IntroNuGet.html" rel="nofollow noreferrer">different versions of a NuGet package to a job</a>. There doesn't seem to be an option to reference different assemblies instead.</p>
  
  <p>I've tried custom build configurations:</p>
  
  <pre><code>  &lt;ItemGroup Condition="'$(Configuration)' == 'Baseline'"&gt;
      &lt;Reference Include="MyAssembly"&gt;
        &lt;HintPath&gt;lib\baseline\MyAssembly.dll&lt;/HintPath&gt;
      &lt;/Reference&gt;
    &lt;/ItemGroup&gt;
  
    &lt;ItemGroup Condition="'$(Configuration)' != 'Baseline'"&gt;
      &lt;Reference Include="MyAssembly"&gt;
        &lt;HintPath&gt;lib\current\MyAssembly.dll&lt;/HintPath&gt;
      &lt;/Reference&gt;
    &lt;/ItemGroup&gt;
  </code></pre>
  
  <p>but when trying to use those configurations from BenchmarkDotNet via</p>
  
  <pre><code>    public static void Main(string[] args) {
        var summary = BenchmarkRunner.Run(typeof(Program).Assembly,
          DefaultConfig.Instance
            .With(Job.Default.WithCustomBuildConfiguration("Baseline"))
            .With(Job.Default.WithCustomBuildConfiguration("Current")));
      }
  </code></pre>
  
  <p>I get build errors that indicate that the assembly is not referenced at all. BenchmarkDotNet also helpfully clears up any temporary artifacts it creates so I can't even look at the generated project file to figure out how that looked.</p>
  
  <p>Would the only workaround here be to wrap the library in a NuGet package? Or is there something I'm overlooking in the (for this case seemingly sparse) docs?</p>
  
  <p><a href="https://github.com/dotnet/BenchmarkDotNet/issues/1197" rel="nofollow noreferrer">This issue</a> seems to be vaguely related to the build errors I'm getting.</p>