title: Parametrizing multiple tests dynamically in Python
body: <p>I'm attempting to use Pytest to write a dynamic test suite, where the test data is held in a separate file, e.g. a YAML file or a .csv. I want to run multiple tests, all of which are parameterised from the same file. Let's say I have a testing file <code>test_foo.py</code>, that looks like this:</p>
  <pre><code>import pytest
  
  @pytest.mark.parametrize(&quot;num1, num2, output&quot;, ([2, 2, 4], [3, 7, 10], [48, 52, 100]))
  def test_addnums(num1, num2, output):
      assert foo.addnums(num1, num2) == output
  
  @pytest.mark.parametrize(&quot;foo, bar&quot;, ([1, 2], ['moo', 'mar'], [0.5, 3.14]))
  def test_foobar(foo, bar):
      assert type(foo) == type(bar)
  </code></pre>
  <p>Using the parametrize decorator, I can run multiple tests in pytest, and that works as expected:</p>
  <pre><code>test_foo.py::test_addnums[2-2-4] PASSED                                                                                                                                                            
  test_foo.py::test_addnums[3-7-10] PASSED                                                                                                                                                           
  test_foo.py::test_addnums[48-52-100] PASSED                                                                                                                                                        
  test_foo.py::test_foobar[1-2] PASSED                                                                                                                                                               
  test_foo.py::test_foobar[moo-mar] PASSED                                                                                                                                                           
  test_foo.py::test_foobar[0.5-3.14] PASSED
  </code></pre>
  <p>But I want to parameterise these tests dynamically. By which I mean that, I want to write the test data for <strong>all tests</strong> in a separate file so that when I run pytest, it will apply all the test data I've written to each test function. Let's say I had a YAML file that looked something like:</p>
  <pre><code>test_addnums:
    params: [num1, num2, output]
    values:
      - [2, 2, 4]
      - [3, 7, 10]
      - [48, 52, 100]
  
  test_foobar:
    params: [foo, bar]
    values:
      - [1, 2]
      - [moo, mar]
      - [0.5, 3.14]
  </code></pre>
  <p>I would then want to read this YAML file and use the data to parameterise all test functions in my test file.</p>
  <p>I'm aware of the <code>pytest_generate_tests</code> hook, and I've been trying to use this to load tests dynamically. I tried adding the same parameters and data values that I previously passed into the <code>parametrize</code> decorator into the <code>metafunc.parametrize</code> hook:</p>
  <pre><code>def pytest_generate_tests(metafunc):
      metafunc.parametrize(&quot;num1, num2, output&quot;, ([2, 2, 4], [3, 7, 10], [48, 52, 100]))
      metafunc.parametrize(&quot;foo, bar&quot;, ([1, 2], ['moo', 'mar'], [0.5, 3.14]))
  
  def test_addnums(num1, num2, output):
      assert foo.addnums(num1, num2) == output
  
  def test_foobar(foo, bar):
      assert type(foo) == type(bar)
  </code></pre>
  <p>This doesn't work, however, because pytest tries to apply the test data to every function:</p>
  <pre><code>collected 0 items / 1 error                                           
  
  =============================== ERRORS ================================
  ____________________ ERROR collecting test_foo.py _____________________
  In test_addnums: function uses no argument 'foo'
  ======================= short test summary info =======================
  ERROR test_foo.py
  !!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!
  ========================== 1 error in 0.16s ===========================
  </code></pre>
  <p>What I want to know is: how can I dynamically parameterise <strong>multiple tests</strong> using pytest? I've introspected pytest using pdb, and from what I can tell, <code>metafunc</code> is only aware of the first test you've defined in the file. In my above example, <code>test_addnums</code> is defined first, so when I print <code>vars(metafunc)</code> in the pdb debugger, it shows these values:</p>
  <pre><code>(Pdb) pp vars(metafunc)
  {'_arg2fixturedefs': {},
   '_calls': [&lt;_pytest.python.CallSpec2 object at 0x7f4330b6e860&gt;,
              &lt;_pytest.python.CallSpec2 object at 0x7f4330b6e0b8&gt;,
              &lt;_pytest.python.CallSpec2 object at 0x7f4330b6e908&gt;],
   'cls': None,
   'config': &lt;_pytest.config.Config object at 0x7f43310dbdd8&gt;,
   'definition': &lt;FunctionDefinition test_addnums&gt;,
   'fixturenames': ['num1', 'num2', 'output'],
   'function': &lt;function test_addnums at 0x7f4330b5a6a8&gt;,
   'module': &lt;module 'test_foo' from '&lt;PATH&gt;/test_foo.py'&gt;}
  </code></pre>
  <p>But if I switch around the <code>test_foobar</code> and <code>test_addnums</code> functions, and reverse the order of the <code>parametrize</code> calls, it shows information about <code>test_foobar</code> instead.</p>
  <pre><code>(Pdb) pp vars(metafunc)
  {'_arg2fixturedefs': {},
   '_calls': [&lt;_pytest.python.CallSpec2 object at 0x7f6d20d5e828&gt;,
              &lt;_pytest.python.CallSpec2 object at 0x7f6d20d5e860&gt;,
              &lt;_pytest.python.CallSpec2 object at 0x7f6d20d5e898&gt;],
   'cls': None,
   'config': &lt;_pytest.config.Config object at 0x7f6d212cbd68&gt;,
   'definition': &lt;FunctionDefinition test_foobar&gt;,
   'fixturenames': ['foo', 'bar'],
   'function': &lt;function test_foobar at 0x7f6d20d4a6a8&gt;,
   'module': &lt;module 'test_foo' from '&lt;PATH&gt;/test_foo.py'&gt;}
  </code></pre>
  <p>So it seems like metafunc doesn't actually store information about every test function in my test file. Therefore I can't use <code>fixturenames</code> or <code>function</code> properties, as they only apply to one particular function, not all of them.</p>
  <p>If that's the case, then how can I access all of the other test functions and parameterise them individually?</p>