How to remove a font downloaded through Font Book?
You can replace them with themselves, but manually installed, with the shell command:

cp /System/Library/Assets/com_apple_MobileAsset_Font5/*/AssetData/* ~/Library/Fonts/


Then, from the “User” section of the sidebar, select all of the fonts you want to remove, and the system-installed version should be removed as well. An alternative way is to resolve the duplicates by keeping the manual version, and then removing that as well.