title: Flatten a nested dictionary and convert it into columns of Dataframe
body: <p>I have a JSON which I converted into a dictionary and trying to make a dataframe out of it. the problem is that it is multiple nested and with inconsistent data</p>
  <p>For e.g.</p>
  <pre><code>d = &quot;&quot;&quot;[
        {
          &quot;id&quot;: 51,
          &quot;kits&quot;: [
              {
                  &quot;id&quot;: 57,
                  &quot;kit&quot;: &quot;KIT1182A&quot;,
                  &quot;items&quot;: [
                      {
                          &quot;id&quot;: 254,
                          &quot;product&quot;: {
                              &quot;name&quot;: &quot;Plastic Pallet&quot;,
                              &quot;short_code&quot;: &quot;PP001&quot;,
                              &quot;priceperunit&quot;: 2500,
                              &quot;volumetric_weight&quot;: 21.34
                          },
                          &quot;quantity&quot;: 5
                      },
                      {
                          &quot;id&quot;: 258,
                          &quot;product&quot;: {
                              &quot;name&quot;: &quot;Separator Sheet&quot;,
                              &quot;short_code&quot;: &quot;FSS001&quot;,
                              &quot;priceperunit&quot;: 170,
                              &quot;volumetric_weight&quot;: 0.9
                          },
                          &quot;quantity&quot;: 18
                      }
                  ],
                  &quot;quantity&quot;: 5
              },                                     #end of kit
              {
                  &quot;id&quot;: 58,
                  &quot;kit&quot;: &quot;KIT1182B&quot;,
                  &quot;items&quot;: [
                      {
                          &quot;id&quot;: 259,
                          &quot;product&quot;: {
                              &quot;name&quot;: &quot;Plastic Pallet&quot;,
                              &quot;short_code&quot;: &quot;PP001&quot;,
                              &quot;priceperunit&quot;: 2500,
                              &quot;volumetric_weight&quot;: 21.34
                          },
                          &quot;quantity&quot;: 5
                      },
                      {
                          &quot;id&quot;: 260,
                          &quot;product&quot;: {
                              &quot;name&quot;: &quot;Plastic Sidewall&quot;,
                              &quot;short_code&quot;: &quot;PS001&quot;,
                              &quot;priceperunit&quot;: 1250,
                              &quot;volumetric_weight&quot;: 16.1
                          },
                          &quot;quantity&quot;: 5
                      },
                      {
                          &quot;id&quot;: 261,
                          &quot;product&quot;: {
                              &quot;name&quot;: &quot;Plastic Lid&quot;,
                              &quot;short_code&quot;: &quot;PL001&quot;,
                              &quot;priceperunit&quot;: 1250,
                              &quot;volumetric_weight&quot;: 9.7
                          },
                          &quot;quantity&quot;: 5
                      }
                     
                  ],
                  &quot;quantity&quot;: 7
              }                                    #end of kit
          ],
          &quot;warehouse&quot;: &quot;Yantraksh Logistics Private limited_GGNPC1&quot;,
          &quot;receiver_client&quot;: &quot;Lumax Cornaglia Auto Tech Private Limited&quot;,
          &quot;transport_by&quot;: &quot;Kiran Roadways&quot;,
          &quot;transaction_type&quot;: &quot;Return&quot;,
          &quot;transaction_date&quot;: &quot;2020-08-13T04:34:11.678000Z&quot;,
          &quot;transaction_no&quot;: 1180,
          &quot;is_delivered&quot;: false,
          &quot;driver_name&quot;: &quot;__________&quot;,
          &quot;driver_number&quot;: &quot;__________&quot;,
          &quot;lr_number&quot;: 0,
          &quot;vehicle_number&quot;: &quot;__________&quot;,
          &quot;freight_charges&quot;: 0,
          &quot;vehicle_type&quot;: &quot;Part Load&quot;,
          &quot;remarks&quot;: &quot;0&quot;,
          &quot;flow&quot;: 36,
          &quot;owner&quot;: 2
      } ]&quot;&quot;&quot;
  </code></pre>
  <p>I want to convert it into a dataframe like the following:</p>
  <pre><code>transaction_no  is_delivered    flow    transaction_date    receiver_client warehouse   kits    quantity    product1    quantity1   product2    quantity2   product3    quantity3
  1180       False    36  2020-08-13T04:34:11.678000Z Lumax Cornaglia Auto Tech Private Limited   Yantraksh Logistics Private limited_GGNPC1  KIT1182A    5   PP001   5   FSS001  18  NaN NaN
  1180       False    36  2020-08-13T04:34:11.678000Z Lumax Cornaglia Auto Tech Private Limited   Yantraksh Logistics Private limited_GGNPC1  KIT1182B    7   PP001   5   PS001   5   PL001   7.0
  </code></pre>
  <p>or to show it in a better way:</p>
  <p><a href="https://i.stack.imgur.com/ZWEoE.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ZWEoE.png" alt="enter image description here" /></a></p>
  <p>What I have done:</p>
  <pre><code>data = json.loads(d)
  result_dataframe = pd.DataFrame(data)
  l = ['transaction_no', 'is_delivered','flow', 'transaction_date', 'receiver_client', 'warehouse','kits']  #fields that I need
  result_dataframe = result_dataframe[l]
  result_dataframe.to_csv(&quot;out.csv&quot;)
  </code></pre>
  <p>I tried :</p>
  <pre><code>def flatten(input_dict, separator='_', prefix=''):
  output_dict = {}
  for key, value in input_dict.items():
      if isinstance(value, dict) and value:
          deeper = flatten(value, separator, prefix+key+separator)
          output_dict.update({key2: val2 for key2, val2 in deeper.items()})
      elif isinstance(value, list) and value:
          for index, sublist in enumerate(value, start=1):
              if isinstance(sublist, dict) and sublist:
                  deeper = flatten(sublist, separator, prefix+key+separator+str(index)+separator)
                  output_dict.update({key2: val2 for key2, val2 in deeper.items()})
              else:
                  output_dict[prefix+key+separator+str(index)] = value
      else:
          output_dict[prefix+key] = value
  return output_dict
  </code></pre>
  <p>But it gives all the values in a single row, how can I seprate them on the basis of kits and get the result?</p>