How to design concurrent data structures?
While I have not read deeply in this area, I found The Art of Multiprocessor Programming by Maurice Herlihy and Nir Shavit to be a helpful introduction and survey of techniques.  It explores different algorithms, reasons about how they work and examines the trade-offs, features and limitations of the different approaches.  While it has some formalism I expect it is a fairly introductory and accessible text.

For a flavor of the text, here is the introduction to the section on Atomic Registers from the 2008 edition:


  The obvious place to begin is to ask whether we can solve consensus
  using atomic registers.  Surprisingly, perhaps, the answer is no.  We
  will show that there is no binary consensus protocol for two threads. 
  We leave it as an exercise to show that if two threads cannot reach
  consensus on two values, then n threads cannot reach consensus on
  k values, where n &gt; 2 and k &gt; 2.