What if irrelevant features impact the outputs?
It would mean that the model trained with some irrelevant features is overfit: a good model uses only features which have some predictive power on the target variable, so if the sample is representative enough any irrelevant feature is ignored or assigned very little weight. However if the training dataset is too small or the model too complex then it uses details which happen by chance, so it could use some irrelevant features: this is overfitting.
In case the performance on the test set was high despite the model being overfit, it's likely that there is some data leakage additionally.