title: Huge sparse dataframe to scipy sparse matrix without dense transform
body: <p>Have data with more then 1 million rows and 30 columns, one of the columns is user_id (more then 1500 different users).
  I want one-hot-encode this column and to use data in ML algorithms (xgboost, FFM, scikit). But due to huge row numbers and unique user values matrix will be ~ 1 million X 1500, so need do this in sparse format (otherwise data kill all RAM).</p>
  
  <p>For me convenient way to work with data through pandas DataFrame, which also now it support sparse format:</p>
  
  <pre><code>df = pd.get_dummies(df, columns=['user_id', 'type'], sparse=True)
  </code></pre>
  
  <p>Work pretty fast and have small size in RAM. But for working with scikit algos and xgboost it's necessary transform dataframe to sparse matrix.</p>
  
  <p>Is there any way to do this rather than iterate through columns and hstack them in one scipy sparse matrix?
  I tried df.as_matrix() and df.values, but all of first transform data to dense what arise MemoryError :( </p>
  
  <p>P.S.
  Same to get DMatrix for xgboost</p>
  
  <p><strong>UPDATE:</strong></p>
  
  <p>So i release next solution (will be thankful for optimisation suggestions):</p>
  
  <pre><code> def sparse_df_to_saprse_matrix (sparse_df):
      index_list = sparse_df.index.values.tolist()
      matrix_columns = []
      sparse_matrix = None
  
      for column in sparse_df.columns:
          sps_series = sparse_df[column]
          sps_series.index = pd.MultiIndex.from_product([index_list, [column]])
          curr_sps_column, rows, cols = sps_series.to_coo()
          if sparse_matrix != None:
              sparse_matrix = sparse.hstack([sparse_matrix, curr_sps_column])
          else:
              sparse_matrix = curr_sps_column
          matrix_columns.extend(cols)
  
      return sparse_matrix, index_list, matrix_columns
  </code></pre>
  
  <p>And the following code allows to get sparse dataframe:</p>
  
  <pre><code>one_hot_df = pd.get_dummies(df, columns=['user_id', 'type'], sparse=True)
  full_sparse_df = one_hot_df.to_sparse(fill_value=0)
  </code></pre>
  
  <p>I have created sparse matrix 1,1 million rows x 1150 columns. But during creating it's still uses significant amount of RAM (~10Gb on edge with my 12Gb).</p>
  
  <p>Don't know why, because resulting sparse matrix uses only 300 Mb (after loading from HDD). Any ideas?</p>