Do you apply outlier detection of numerical data in practical applications?
You'd have to think about where the outliers come from. Technical malfunction? Then they are probably safe to disregard. But when you write that your main task is 


  to draw conclusions from the training data that can be generalized to future unseen data


you have to keep in mind that your future data will contain outliers as well and deal with them appropriately. Models like trees are usually somewhat robust against those while regression can go haywire. There are several well working methods like disregarding everything that is more than 4 standard deviations (or median deviations from the median) out or just disregarding the n-th top and bottom percentiles.

But then there is the case where outliers are not just flukes and can actually tell you something about the problem domain you are working on. If you work in fraud detection, the outliers are the points that interest you. If you work on optimizing network performance, the packets that don't arrive can be the interesting ones.

So my advice is to not blindly throw away data out of principle but always think about what the data represent and why outliers may be present.