Can brew find / manage packages installed by manually binary download + install?
Assuming what you downloaded manually was with brew, just do brew list, you can see what you've installed using brew. Then just do brew uninstall packageNameHere to get rid of what you don't want.