Feature selection by overfitting a small sample size I am using a CNN based model to do sequence classification. Since training an entire dataset is very expensive, and I have a large set of features needed to try, its impossible for me to select features by full trainings.

Usually a good sanity check before training, is try to let model overfit a small set of training samples, to make sure the model is at least capable of remember a small sample size.

Borrowed from this idea, my question is, can I train a small subset of training data, and use its loss curve as a metric, to select best features? Each training is to test how quick would the training loss converges given a subset of selected features.
Congratulations! You have suggested independently the Wrapper method for feature selection. Yes, you can use this method. 
However, consider that the wrapper method is slow since you have to train a model for each iteration. More than that, feature selection is a NP-complete problem so don't expect the optimal subset.