Is color information only extracted in the first input layer of a convolutional neural network?
The quote that you have brought is speaking about something else. Zero means that most of your linear part of neurons have a negative value and after passing through the ReLU non-linearity, they become zero. That's the explanation of that part. About your question, not really. I don't know if you've ever visualized a convolutional network or not but you've done that you may have seen that there may be layers that get activated for special colors even in deep layers. The reason is that the convolutional layer just tries to do cross-correlation operation on the input which has three channels. Next layers of convolution also do this considering e.g. the input has 64 channels. To wrap up, there may be other deep-layers that use the information of colors, Although these operations really do not care about the colors, they care about the local patterns that may happen in the input. The reason for learning colors is that similar colors have similar patterns in their RGB channels. These patterns are what convolutional networks try to learn. Take a look at the tool box that have been discussed here.