How does SuperSU check if a phone is rooted? I have a rooted custom device (Android 8.0), I can use adb and run su and do whatever I want in the shell. But SuperSU or any other app cannot find root. I have made a symlink from /system/xbin/su to /system/bin/su but it still does not work.
/system/xbin/su is a normal su binary that's shipped with userdebug or eng builds of ROMs. It gets you root access by making syscall setuid. On the other hand, with SuperSU or Magisk root, a special binary is placed at /sbin/su which works quite differently. For details see How Magisk works?.

So, with only a working /system/xbin/su executable, your device isn't essentially rooted. SuperSU won't work this binary.

RELATED: What special privileges “/system/xbin/su” does have w.r.t. root access?