flyback converter closed loop by using TL431
The TL431 is a simple, self-supplied, open-collector operational amplifier (op-amp) with a self-contained 2.5-V reference voltage. The basic electrical schematic is shown below with a French snail and a supposedly faster rabbit. I could have included a tortoise and a rabbit to recall Jean de la Fontaine also but there is no competition here : )

When the power supply regulates at the correct level, e.g. 12 V in your example and you measure the voltage across \$R_{lower}\$ you must find 2.5 V roughly. The regulation mechanism is as follows:

at power-up, the output voltage is 0 V and the TL431 is inert
because there is no current in the LED, there is no drop across \$R_{LED}\$ and the voltage at the TL431 cathode is \$V_{out}\$-\$V_f\$. \$V_f\$ is the LED voltage drop, \$\approx 1\;V\$.
when the output voltage increases cycle-by-cycle, at some point, the TL431 wakes up because there is enough voltage to supply the internal circuitry and ensure the 2.5-V reference voltage (around 2.5-3 V on the cathode).
as \$V_{out}\$ is still far from the target, the voltage at the REF pin is low because of the divider made of \$R_1\$ and \$R_{lower}\$: the TL431 remains in a high-impedance state and the voltage across \$C_2\$ in the primary side is maximum (5 V perhaps depending on the IC), asking for the maximum peak current.
before \$V_{out}\$ reaches 12 V, the voltage at the REF pin approaches 2.5 V and the TL431 starts sinking current: the voltage at the opto collector begins to fall, reducing the peak current setpoint.
if the loop is well stabilized, the TL431 will sink enough current to maintain, via the LED current, a certain voltage on the collector which, in turns, imposes the correct operating peak current in the circuit for the input and output conditions: the loop is closed and the converter regulates nicely.
if \$V_{out}\$ drops because more current is absorbed from the load, the TL431 will sink less current and the opto collector voltage will rise, asking for more peak.
if \$V_{out}\$ increases because the converter enters light-load conditions, more current flows in the LED and the collector voltage drops until the new adequate peak current is obtained.

At any operating point in regulation, the voltage at the TL431 cathode is \$V_K=V_{out}-R_{LED}I_F-V_f\$. It is important to understand that current \$I_F\$ which flows in the fast lane, in regulation, solely depends on the primary-side collector current: \$I_F=\frac{I_C}{CTR}\$. Changing \$R_{LED}\$ will not change that current but will obviously affect the drop across the resistance. That is why if you increase \$R_{LED}\$ too much, the drop can be so high that the remaining voltage at the TL431 cathode can be smaller than 2.5-3 V and regulation is lost. That is the reason why I gave the formula for the upper limit of \$R_{LED}\$:

So, for your questions specifically:

&quot;How do I design \$R_{LED}\$?&quot; You choose this resistance based on the gain you need to crossover at the selected frequency. Then, you check with the above formula that you have enough margin with the maximum value. Problems usually start to occur when you deal with regulated values of 5 V or below. If the feedback current \$I_F\$ in nominal conditions is too low compared to the minimum recommended in the data-sheet (1 mA or so for a classical TL431), then you can add an extra bias as a simple resistance in // with the LED. It provides a cheap current source and does not affect the feedback current in the LED itself.

Please note that the maximum current seen by the TL431 is during an over-voltage situation such as a transient overshoot or when the loop is broken: the TL431 would desperately try to brake by sinking the maximum current in the LED in an attempt to reduce the peak current. The TL431 would do that by reducing its operating voltage to \$V_{TL431min}\$ which is around 2.5-3 V. In that case, the maximum current absorbed by the TL431 is \$\frac{V_{out}-V_f-V_{TL431min}}{R_{LED}}\$. Assume \$R_{LED}=1\;k\Omega\$ and \$V_{out}\$ in fault is 20 V, the maximum current would be \$\frac{20-1-3}{1k}=16\;mA\$ far from the maximum value specified in the data-sheet. You can then compare this value with the maximum acceptable LED current which is 60 mA for a popular component like the SFH-615: you are safe.

&quot;How do I know the TL431 voltage?&quot;, you apply \$V_K=V_{out}-R_{LED}I_F-V_f\$ but it is irrelevant for the design procedure.

&quot;How does the TL431 work in regulation?&quot; Check my description above.