How does macOS "detect malware downloaded with Safari"?
It's part of the XProtect / file quarantine scheme built into the OS.  macOS has a hard-coded list of file signatures in /System/Library/CoreServices/XProtect.bundle/Contents/Resources/XProtect.plist.  If a file matches one of the signatures, it gets flagged and quarantined.  The list is silently updated, outside the usual software-update mechanism.

It is effectively a very rudimentary blacklist of known malware.  XProtect does not provide active scanning or heuristic-based protection offered by typical antivirus products.