Programmatically set Panelizer View Template on Node
It turned out the output of $node->panelizer->getValue(); was giving me a clue that I missed. 

I had forgotten that there were different panelizer templates for different view modes.

I was able to accomplish what I needed with

    $panelizer_arr = $node-&gt;panelizer-&gt;getValue();
    for ($i = 0; $i &lt; count($panelizer_arr); $i++) {
        if ($panelizer_arr[$i]['view_mode'] == 'full') {
            $panelizer_arr[$i]['default'] = 'test_display';
        }
    }
    $node-&gt;panelizer = $panelizer_arr;
    $node-&gt;save();