Troubleshooting why ADuM4160's internal regulator gives 5 V instead of 3.3 V
The datasheet says that the \$V_{DD1}\$ and \$V_{DD2}\$ behaviour is the same: if \$V_{BUSx}\$ is connected to 5V, the signal at \$V_{DDx}\$ should be the 3.3V output of the LDO. If \$V_{BUSx}\$ is 3.3V, directly connect \$V_{BUSx}\$ and \$V_{DDx}\$ together.

Ceramic capacitors are by nature low-ESR, and using a linear regulator to feed the \$V_{BUSx}\$ signals should be fine.

You should scope all the \$V_{BUSx}\$ and \$V_{DDx}\$ signals with a short-ground probe, AC-coupled with bandwidth limiting turned on to see if there are any disturbances in the bus voltages (ballpark: anything in excess of 1% of the DC level may be trouble). Check multiple time scales (10s of ms, 10s of ns, etc.).

Some ceramic decoupling capacitors (100nF or so) close to the IC on the \$V_{BUSx}\$ voltage pins wouldn't hurt.