Is there a rule of thumb for the initial value of loss function in a CNN? I notice that most of my successfully trained CNNs initiate (i.e. first epoch) loss is in the 1-2 range. Typically, these are image classifiers, either for discrete classes (cat, dog, ship, etc) or semantic (pixelwise) segmentation. Occasionally, either because of poor design or architecture misunderstand on my part, I see an initial loss in the 60s; the other day one started at 200 (and didn't converge).

So my question is: is there a heuristic that would allow me to determine a ball park value for my initial loss value? If so, on what aspects of my trained model is it contingent?
The initial loss is purely dependent on your weight initialization and your data normalization. Random weights = random initial loss. If you're interested in the variance of the loss, this will roughly relate to the number of layers you have and number of weights in each layer. For a good example, check out this paper:

Understanding the difficulty of training deep feedforward neural networks -- Glorot, Bengio