Storing Large dataset for processing and analysis of data I am new to data engineering and wanted to know , what is the best way to store more than 3000 GB of data for further processing and analysis ? I am specifically looking for open source resources . I have explored many data formats for storage . The dataset that I want to store is a heart rate pulse data generated by a sensor.
It would depend on the use cases - read vs write vs analytics etc. Nonetheless, you may want to explore Hadoop if not done already.