Is it a good idea to overfit on a small part of your data for faster model convergence?
The first thing which you first have to understand is that does your trained model is working efficiently with both the training and testing data. If yes then its not overfitting. There is only one case where over-fitting doesn't matter and that is when your testing data is same as training data and dealing with is very important or it could be disastrous. Therefore, to avoid overfitting you can try several methods and they are followings: 


Optimization Algorithms like sgd, Adam, RMSprop, Adagard, are few gradient descents.
Loss function like Hinge loss could be useful.
Parameter initialization greatly influence the optimization process.


Overall, Large datasets are better for building ML models but there are some methods there is an interesting article that i would recommend you to read: Article Link

I Hope this Helps!!!