Isn't the depth of a convolutional layer, the number of colors (or colorspace size)?
Yes, the depth of an image is equal to the color channels (1 for gray-scale images, 3 for RGB). 

However, that is only the case for the input layer of the CNN. During the first convolution layer, the image can be passed through as many filters as we select. This number becomes the depth of the first layer. The subsequent layers can have any depth we want.

For example in the CNN depicted below:



The image input to the CNN initially has a depth of 1 (because it is gray-scale). The first convolutional layer passes it through 6 filters, so the depth of the first layer becomes 6. The second layer passes the output of the first through 16 filters, meaning that the second layer has a depth of 16, etc.