title: Data science workflow with large geospatial datasets
body: <p>I am relatively new to the docker approach so please bear with me. </p>
  
  <p>The goal is to ingest large geospatial datasets to <a href="https://earthengine.google.com/" rel="nofollow noreferrer">Google Earth Engine</a> using an open source replicable approach. I got everything working on my local machine and a Google Compute Engine but would like to make the approach accessible to others as well. </p>
  
  <p>The large static geospatial files (NETCDF4) are currently stored on Amazon S3 and Google Cloud Storage (GEOTIFF). I need a couple of python based modules to convert and ingest the data into Earth Engine using a command line interface. This has to happen only once. The data conversion is not very heavy and can be done by <strong>one</strong> fat instance (32GB RAM, 16 cores takes 2 hours), there is no need for a cluster. </p>
  
  <p>My question is how I should deal with large static datasets in Docker. I thought of the following option but would like to know best practices. </p>
  
  <p>1) Use docker and mount the amazon s3 and Google Cloud Storage buckets to the docker container. </p>
  
  <p>2) Copy the large datasets to a docker image and use Amazon ECS </p>
  
  <p>3) just use the AWS CLI </p>
  
  <p>4) use Boto3 in Python</p>
  
  <p>5) A fifth option that I am not yet aware of</p>
  
  <p>The python modules that I use are a.o.: python-GDAL, pandas, earth-engine, subprocess</p>