Using MOSFET with Open Drain Controller
If you are going to use a BS170, use it like this (with the PCA driver configured as a totem pole output): -



At the moment you have the source connected to the positive rail and the internal body diode will conduct all the time and you won't be able to turn the LED off.

Regards the other LED outputs connected directly the the PCA device take note what its data sheet says: -


  The LED output driver is programmed to be either open-drain with a 25
  mA current sink capability at 5 V or totem pole with a 25 mA sink, 10
  mA source capability at 5 V.


In other words, if you are trying to source current into an LED you will only get 10 mA LED current.