Lightweight binary image classifier
If I wanted to build a classifier that takes short time to be trained, I would rather simplify the classifier than the data. In this case, I would rather train a logistic regression model (probably with Ridge or Lasso regularization), than a very complex architecture with few of the data I have.

If I had to simplify the data, I would take the average of the three channels, thus having a black and white image as input. I would not simplify it further (I am very confident that 4 points won't give decent accuracies).

I don't really know your circumstances, but Lasso in logistic regression is a method that might allow you to select some pixels. As the coefficients obtained by Lasso are sparse (see this answer), Lasso will select the pixels that are relevant towards the prediction of the class. You can then train a bigger network with those pixels that allows you to capture nonlinearities and more complexity.