Correlations, p-values and features selection
The fact that a feature has low correlation with the target variable shows that it's not a good indicator on its own, but that doesn't mean that it can't be useful for the model when combined with the other features.
The only way to know if these features are useful is to use them to train a model, then evaluate on a validation set and see if it improves performance.

the p-value is less than 0.05

Is this the result of a correlation significance test? It depends on the test but in general a p-value lower than 0.05 means that there is a significant difference, i.e. in this case it probably means that the correlation is truly not zero. Anyway imho this wouldn't prove anything with respect to using these features or not.