iMac 2017 Catalina: camera won't work but seems recognized
The FaceTime webcam has died.
This is basically a duplicate of this question about a MacBook Pro camera but you've done a lot more diagnostics.
You've got two options:

Replace the FaceTime webcam.  I don't recommend this option unless you intend to do other work inside your iMac like upgrade your hard drive to an SSD.  This requires removing the screen which means you'll have to cut through the adhesive holding the screen to the frame; it's a very time consuming and tedious task requiring the purchase of adhesive strips for the display when reassembling

Get a USB Camera.  While my FaceTime camera works, I actually use a USB webcam (Logitech Brio) so I can get 4K resolution which is much higher than the FaceTime HD camera's resolution of only 1280x720.  The color is also much more vibrant on the Logitech making for a much better picture.


Basically, the fact that it doesn't work in Safe Mode (after a reboot) and doesn't work with Apple’s own built in software that works with it (i.e. FaceTime and PhotoBooth), pretty much says this is a physical issue.  Unless you're up for removing the glass on the iMac, go with an external USB webcam.

Future Diagnostics:
I just wanted to provide some pointers for future diagnostics for folks who may be looking to solve this (or other hardware issues).  Here are some tips to guide you in diagnosing the problem:

Hardware can be &quot;recognized&quot; but not work.  It's like having all 4 wheels removed from your car.  It will start, shift into all the gears and no engine light will turn on, but you car isn't going anywhere.
In this case, the controller (aka the USB bridge - an IC or “chip”) can still work, but the sensor (what captures the image) just dies. So, the system still see's it attached to the USB bus, but you get no image.

Plugging in another USB Webcam is the best diagnostic test.  The integrated FaceTime camera is also attached to the USB bus so if a different USB webcam works and the integrated one doesn't it's pretty much guaranteed that the issue is the camera itself.

Running Safe Mode (Hold ⇧ Shift
while booting) is an excellent troubleshooting tool as it eliminates 3rd party software and drivers (kexts).  Also, since it requires rebooting, you know everything is starting &quot;fresh.&quot;

Apple Diagnostics (Hold D while booting) is also an excellent tool, but it sometimes cannot detect things that don't generate errors (like in this case).
Like your eyes, you may lose function as you age (the failure), yet you don't experience pain (the error code).  This is why we do alternate tests.

Disabling security is not a good idea.  SIP only protects certain folders from having things written to it.  Disabling security, especially when the device was working before won't yeild much return in the way of diagnostic info.

Resetting the SMC and NVRAM is rarely a fix for anything.  You'll see these recommendations everywhere and the result is usually fruitless.  It's because the SMC handles power and cooling (fans) while the NVRAM handles the pre-boot environment like setting the boot device, and initial boot screen resolution.  There's nothing in it that addresses peripheral devices like webcams or software.

Updating your OS is always a good idea.  Apple's been known to break things with updates and new OS releases.  Now with APFS and it's ability to take snapshots of your hard drive, the converse is now true - rolling back is a good idea if you apply an update and something breaks.  APFS allows you to roll back your OS to a point where it was working - it's an &quot;undo&quot; for your Operating System.