Preprocessing in Data mining? I am still new to data mining but I really want (and need) to learn it so badly. I know that before I can actually process my data in softwares like WEKA, I need to do some filtering like cleaning the data, integrating, transforming, etc to actually get your data cleaned from any kind of duplicate, missing value, noise, etc. But I only know all of these theoretically. 

The problem I have now is I have a very big set of data that I need to filter first before going on to the processing part. But I don't know where to start. Fyi, my dataset is very big that the usual spreadsheet programs like Ms Excel, Libre Office, WPS,etc can't open it. I have to use Linux terminal and commands to count the number of rows, columns etc. 

What do I do in the preprocessing start? How do I 'clean' my data? I have been thinking to use Linux commands to do all of these, but I am also wondering how real data scientists clean their data. Do they do these manually or they already have some sort of software to help them? Because seriously I don't know where to start or to do. Every reference I found in the internet only explain things theoretically. ANd I need something more practical to help me understand. 

What do I do with my dataset? Help please?
The preprocessing will depend on what your data is like: textual? numerical? If in whatever analysis you want to do the data is not allowed to have duplicates, you'll have to filter those out. If not, they'll get to stay, etc. 
If you already know exactly what you want to do theoretically and just need to find some software, python pandas is very good. Load the data, clean it, group it, perform elementary analysis.