title: Fine-tune BERT HuggingFace with many text inputs
body: <p>I'm trying to fine-tune BERT / transformer model that takes tabular data as an input.</p>
  <p>Each row has many text inputs and some numeric features.</p>
  <p>I want to have one neural network on one side. Then a transformer on the other side. The transformer will take the text inputs. The neural network will take the numeric inputs. The outputs will be concatenated and a final dense 1 sigmoid layer will produce a prediction.</p>
  <pre class="lang-py prettyprint-override"><code>def bert_model():
      num_columns = 10
      inputs = []
      numeric_inputs = layers.Input(shape = (window_size, num_columns))
      categorical_inputs = layers.Input(shape = (window_size, num_columns))
      
      inputs.append(numeric_inputs)
      inputs.append(categorical_inputs)
      
      input_ids = layers.Input(shape=(512,), name='input_token', dtype='int32')
      input_masks_ids = layers.Input(shape=(512,), name='masked_token', dtype='int32')
      unpooledoutputs = tfbert_model(input_ids, input_masks_ids)
      bert_layer = layers.Dropout(0.1)(unpooledoutputs.pooler_output)
      inputs.append(input_ids)
      inputs.append(input_masks_ids)
      # textual data
      for i in range(1, window_size):
          input_ids = layers.Input(shape=(512,), name=f'input_token_{i}', dtype='int32')
          input_masks_ids = layers.Input(shape=(512,), name=f'masked_token_{i}', dtype='int32')
          inputs.append(input_ids)
          inputs.append(input_masks_ids)
      
          unpooledoutputs = tfbert_model(input_ids, input_masks_ids)
          unpooledoutputs = layers.Dropout(0.1)(unpooledoutputs.pooler_output)
          bert_layer = layers.Concatenate()([bert_layer, \
                                                   unpooledoutputs])
      ... rest of the concatenation with other model
  
  </code></pre>
  <p>Is this the correct way to do this? I would like to use the single bert model to be fine-tuned over all of the inputs. I do want want to have to use like 50 BERT models, one for every text input, but to just have one.</p>