Residual plot is tight to zero, but low R2 score
According to the plot there is a huge outlier in your residual. Plus taking into account that your cross-validation sometimes shows quite good results (0.77, 0.74) and sometimes not so great (0.42, 0.36) I would first try to remove that outlier from your data and see what you get.