title: Pyspark best practices
body: <p>We are newly implementing pyspark in our company, which would be the best practice of handling code. Out Architecture is a follows</p>
  <p>S3 --&gt;EMR (Pyspark)--&gt; SNowflake . Data flow is orchestrated through airflow.
  Infra orchestration through Terraform.</p>
  <p>Currently, I am storing terraform code through Ci/CD  on github . So when Data is pushed to pyspark in dev s3  it gets copied to the prod aws account.
  My Question is do I need to store pyspark on GitHub or not ? What is the best practice to do this?</p>