Big Amount of Data on a PC?
Basically for handling large amount of data you have to use big data tool like Hadoop or Apache Spark. You can use pyspark which is combination of python and spark having high efficiency for data processing.
I suggest, if you have flat file format then used ORC file format for processing data into pyspark which improve your performance efficiency.
Hello I want to deal with a big amount of data of 1 billions rows and 23 columns. But in pandas I cannot even read the data. So how can I handle this data on my computer which is Dell XPS 9570. Can I use spark for that? Any advice to deal with it on my PC?
Thank you