Does Box plot with many outliers effects the result?
As the comments suggest, it's not always helpful to think of points outside the whiskers as "outliers". What you are seeing in these boxplots is a strong positive skew. And yes, a strongly skewed target is typically harder to predict than a less-skewed target.

In traditional survival analysis, it's common to model this skewed data using a parametric probability distribution that naturally produces positive and positive-skewed data, e.g. the Weibull distribution.

Otherwise, you can also try a Box-Cox or inverse hyperbolic sine (IHS) transformation on the survival time to reduce the skew.

For visualizing strongly-skewed data, you can either apply one of the above skew-reducing transformations, or use an "adjusted" boxplot, which have an R implementation.