What is this material between our concrete and bricks (lining our pool)?
It is called "mastic". It provides a flexible buffer space between the pool and the deck while sealing that gap. It does wear out over time and need to be replaced. 

Pool mastic generally refers to the expansion seal located several inches from your in ground pool’s edge. This expansion seal is filled with mastic, a waterproof, putty-like adhesive used as a joint-sealer between concrete sections.

The pool mastic seal is vital to the care of an in-ground pool because it prevents water from seeping between the pool and your decking, which can cause the surrounding concrete to crack. Put more simply, your pool and deck both need a small margin of space to expand and contract with the weather, the pool mastic provides that space.

Pool mastic should be replaced when you notice the sealant in the expansion joint peeling or cracking. Other indicators include pool tiles falling off, cracks in your deck, or heaving (rising) in your deck. In fact, anytime you notice the concrete around your pool changing it means you should inspect your pool mastic.

Source