What are interleaved layers of convolutions?
The Oxford dictionary explains "interleave" as 


  Place something between the layers of (something)


or in the context of telecommunication, as


  Mix (digital signals) by alternating between them.


In the FaceNet paper, Schroff et. al describe their first architecture as 


  multiple interleaved layers of convolutions, non-linear activations, local response normalizations, and max pooling layers


which means they create a CNN by "altering between" these layer types (e.g. Conv -&gt; Pool -&gt; Conv -&gt; Pool -&gt; ..). Or with the wording of the first definition, you "place one layer" of each type "between" the other existing layers (i.e. start with a couple of Conv layers, then place Pool layer between two Conv layers, and so on).
The final architecture would look somewhat like this:

Conv -&gt; ReLu -&gt; LRN -&gt; Pool -&gt; Conv -&gt; ReLu -&gt; LRN -&gt; Pool -&gt; Conv -&gt; ...


So, this "interleaved" CNN is a network where the different layer types are applied in series, layer-by-layer.
As a comparison, in "inception" type CNN, you apply the different layer types in parallel (the graphic below is 1 layer):

              +---&gt; Conv 3x3 ---+
              |                 |
prev layer ---+---&gt; Conv 5x5 ---+---&gt; next layer
              |                 |
              +---&gt; Pool 3x3 ---+