Does a helicopter's boundary layer have different properties that are caused by the rotation? How does this affect stability of the helicopter?
You are right, the rotation does affect the boundary layer.

Normally, as a wing approaches its stall angle of attack, the boundary layer becomes thicker and the flow starts to separate near the trailing edge. On a rotating rotor or propeller blade, the slowed-down boundary layer will experience a centrifugal acceleration, so it does not come to a standstill eventually (as it does in case of separation), but merely starts to flow tipwards. Since the speed of the rotor blade increases as the flow moves towards the tip, the boundary layer experiences an additional Coriolis acceleration. Therefore, flow separation is delayed on rotors and propellers compared to the two-dimensional case (for example, when the rotor airfoil is tested in a windtunnel).

From heli-air.net:


  The helicopter rotor experiments of Dwyer &amp; McCroskey (1971) also
  suggest favorable effects on the spanwise development of the boundary
  layer, which tend to delay the onset of flow separation to a higher
  blade section AoA and thus serve to increase the maximum thrust of the
  rotor system.


The stability is affected by other effects; here the peculiarities of a boundary layer on a rotating wing make little difference beyond the higher stall angle of attack.