Which transistor should I use in place of a 5V relay to adjust brightness of lights (230V) with the Arduino?
Although being quite simple, this circuit is not for begginers. It is connected directly to mains! If not properly designed, dangerous voltages can reach your Arduino and PC.

You will need an optocoupler for proper isolation. Also you will not be using a transistor, you'll need a triac.

Check this schematic from wiki.dxarts.washington.edu



It is using MOC3020M which contains an optoisolator and triac driver.

Similar schematic can also be found on Arduino Playground.

Do not build a circuit without proper isolation!

Do not replace MOC3020 with other parts (optocoupler and transistor/diac) unless you know what you are doing!