What is max current of potentiometer?
I'm using 0.2W, 500ohm potentiometer ... But according to calculation, max current of this potentiometer is about 20mA. 


$$ I_{MAX} = \sqrt {\frac {P}{R}} = \sqrt {\frac {0.2}{500}} = 20~mA $$

Your calculation is correct.

You need to remember that this is the maximum current that can run through any portion of the potentiometer track. A common mistake is to think that if the pot was turned down to 20% that we could run five times the current through it and remain within the power limit. 



Figure 1. Single-turn potentiometer with metal casing removed to expose wiper contacts and resistive track. Source: Photo by Junkyardsparkle, Wikipedia Commons.

It should be obvious from the photo that we can't expect to dissipate 100% of the power rating in a fraction of the potentiometer. That portion of the track would overheat.

The power rating is a useful number for the manufacturer as they can make a range of values sharing the common design. For the user the maximum current may be a more relevant parameter.


  I'm using 3 ohm part with 300 mA.


We're all surprised the potentiometer still works. You may be fortunate that at minimum setting the metal contacts are touching the tab and giving you a few ohms resistance.

You are using the potentiometer as a "rheostat" - a variable resistor rather than varying the potential or voltage along the track. That's OK but not at the current you are putting through that potentiometer.


  This is test video. The voltage of multimeter is voltage aroung 100 ohm resistor.


Presuming you mean the "voltage across" one of the 100 Ω resistors. In the video this varies from about 13 mV to 400 mV which means your current through that leg of the circuit (one pair of LEDs) is varying (using I = V/R) from 0.13 mA to 4 mA.