title: Read JSON in pySpark with custom schema in GCP Dataproc
body: <p>In GCP Dataproc (with pySpark), I am doing a task i.e. to read JSON file as per custom schema and load it in a Dataframe.</p>
  <p>I do have following sample testing JSON:</p>
  <pre><code>{&quot;Transactions&quot;: [{&quot;schema&quot;: &quot;a&quot;,
  &quot;id&quot;: &quot;1&quot;,
  &quot;app&quot;: &quot;testing&quot;,
  &quot;description&quot;: &quot;JSON schema for testing purpose&quot;}]}
  </code></pre>
  <p>I have created following schema:</p>
  <pre><code>custom_schema = StructType([
                        StructField(&quot;Transactions&quot;,
                           StructType([
                               StructField(&quot;schema&quot;, StringType()),
                               StructField(&quot;id&quot;, StringType()),
                               StructField(&quot;app&quot;, StringType()),
                               StructField(&quot;description&quot;, StringType())
                                     ])
                              )])
  </code></pre>
  <p>Reading JSON as:
  <code>df_2 = spark.read.json(json_path, schema = custom_schema)</code></p>
  <p>Getting following results,</p>
  <p><a href="https://i.stack.imgur.com/Bqz5F.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Bqz5F.png" alt="Getting following results," /></a></p>
  <p>Now, I need to check data in Dataframe, When I try to do <code>df_2.show()</code>, it is taking too much time and show as <em><strong>kernel Busy</strong></em> for hours.</p>
  <p>I need help, that what I am missing here in code and how can I see the data in dataframe (Tabular format).</p>