title: Splitting a dataframe into smaller ones and running loops over the subsets
body: <p>I have downloaded one year worth of S&amp;P 500 stock data using the Python package <code>yfinance</code> as follows (making sure to only keep the opening price) for each of the 500 firms:</p>
  <pre><code>import pandas as pd
  import yfinance as yf
  import numpy as np
  source=pd.read_html('https://en.wikipedia.org/wiki/List_of_S%26P_500_companies')
  df = pd.DataFrame(source[0])
  tickers_symbols=df['Symbol'].values.tolist()
  GICS_sectors = df['GICS Sector'].values.tolist()
  data = pd.DataFrame()
  for t,s in zip(tickers_symbols, GICS_sectors):
      tmp = yf.download(t, period='1y', progress=False)
      tmp.reset_index(inplace=True)
      tmp['Ticker'] = t
      tmp['GICS'] = s
      data = data.append(tmp, ignore_index=True)
  ##KEEP ONLY OPENING PRICE##
  data=data.drop([&quot;Close&quot;, &quot;High&quot;, &quot;Low&quot;, &quot;Adj Close&quot;, &quot;Volume&quot;], axis=1)
  </code></pre>
  <p>Now, I need to sort this large dataaset into smaller datasets according to each company's GICS sector. In order to do this, I included the tuple <code>GICS</code> and <code>dataset</code> in a <code>dict</code> object (as suggested <a href="https://stackoverflow.com/questions/19790790/splitting-dataframe-into-multiple-dataframes">here</a>, such that I would then be able to call each smaller dataset by simply typing <code>dataset_list['GICS sector']</code>.</p>
  <pre><code>dataset_list = dict(tuple(data.groupby('GICS')))
  print(dataset_list)
  ##SPLIT DATASET BY GICS SECTOR AND REMOVE GICS COLUMN##
  for sector, dataset in dataset_list.items():
      long_dataset=data.drop(columns='GICS', axis=1)
  </code></pre>
  <p>However, I am having trouble with the subsequent steps. Indeed, as I run a loop to transform each dataset from long to wide and save it as a .csv file, it correctly creates 11 files (as many as there should be datasets), but <em>the data in each file is exactly the same</em>.</p>
  <pre><code>##CONVERT EACH DATASET FROM LONG TO WIDE##
  for sector, dataset in dataset_list.items():
      final_datasets=long_dataset.pivot_table(index=&quot;Date&quot;, columns=&quot;Ticker&quot;, values=&quot;Open&quot;)    
      final_datasets.to_csv(str(sector)+' DataFrame.csv', index=True, sep=',')  
  </code></pre>
  <p>I think there is a problem with the loop I wrote, but I am not sure how to fix it. Each loop above should modify all datasets in the <code>dataset_list</code> object, i.e. I should be able to retrieve a dataframe <code>final_datasets['GICS sector']</code>, but only one dataframe is produced.</p>
  <p>Any help is much appreciated.</p>