title: aws glue dropping mostly null fields
body: <p>I have a dataframe df.  It has a couple columns that are mostly null.  I'm writing it to an s3 bucket using the code below.  I then crawl the s3 bucket to get the table schema in the datacatalog.  I'm finding when I crawl the data the fields that are mostly null get dropped.  I've checked the json that is output and I'm finding that some records have the field, and others don't.  Does anyone know what the issue might be?  I would like to include the fields even if they are mostly null.</p>
  
  <p>Code:</p>
  
  <pre><code># importing libraries
  
  import sys
  from awsglue.transforms import *
  from awsglue.utils import getResolvedOptions
  from pyspark.context import SparkContext
  from awsglue.context import GlueContext
  from awsglue.job import Job
  
  glueContext = GlueContext(SparkContext.getOrCreate())
  
  from pyspark.sql.functions import col
  from pyspark.sql.functions import first
  from pyspark.sql.functions  import date_format
  from pyspark.sql.functions import lit,StringType
  from pyspark.sql.types import *
  from pyspark.sql.functions import to_date,format_number,dayofmonth,hour,dayofyear,month,year,weekofyear,date_format,unix_timestamp
  from pyspark.sql.functions import *
  
  
  # write to table
  df.write.json('s3://path/table')
  </code></pre>