Call AJAX function in the beginning not just after change
Rather than calling the AJAX function at the beginning I simply built the form as it should be in buildForm. Like so

    for ($i = 0; $i &lt; $this-&gt;configuration['test']; $i += 1) {
        $columns[$i] = [
            '#type'  =&gt; 'textfield',
            '#title' =&gt; t('Column Title'),
            '#value' =&gt; $this-&gt;configuration['test'],
        ];
    }
    $form['column_wrapper'] = [
        '#type'        =&gt; 'container',
        '#attributes'  =&gt; ['id'  =&gt; 'column_wrapper'],
        'columnTitles' =&gt; $columns,

    ];