title: Write Spark Scala udf that will read parquet files
body: <p>I want manual control of proccess that devides data read to partitions. For example, suppose I have a DF</p>
  <pre><code>+---+---------------------+
  |key|data                 |
  +---+---------------------+
  |k0 |[file0, file1, file2]|
  |k1 |[file0, file3, file4]|
  +---+---------------------+
  </code></pre>
  <p>And I have some <code>col_i, col_j</code> that I want to read from these files (in my case parquet) but I want to keep data in original partitions. So after udf call I expect to get something like</p>
  <pre><code>+---+------------------+------------------+
  |key|             col_i|             col_j|
  +---+------------------+------------------+
  | k0|val_i_0_from_file0|val_j_0_from_file0|
  | k0|val_i_1_from_file0|val_j_1_from_file0|
  | k0|val_i_0_from_file1|val_j_0_from_file1|
  | k0|val_i_0_from_file2|val_j_0_from_file2|
  | k1|val_i_0_from_file0|val_j_0_from_file0|
  | k1|val_i_1_from_file0|val_j_1_from_file0|
  | k1|val_i_0_from_file3|val_j_0_from_file3|
  | k1|val_i_0_from_file4|val_j_0_from_file4|
  +---+------------------+------------------+
  </code></pre>
  <p>with all <code>k0</code> rows on same partition (same for <code>k1</code>).</p>
  <p>Why such a question: Table with these files is not partitioned by <code>key</code> column but has it in its structure. I want to join by this column. <strong>To avoid shuffle</strong> on this key after reading I manually read metainfo about these parquet files, filter them based on other join side (small dataframe) and repartition these links by <code>key</code> before reading.</p>