Homebrew wants me to delete /usr/local/lib?
That's kinda the problem; Homebrew doesn't know what they are either, and randomly sourced libraries tend to break builds.  (MacPorts makes the same recommendation.  In fact, I was under the impression that was one of the things Homebrew relaxed.)

If you don't know what they are or if they're used, the simplest solution is

$ sudo mv /usr/local /usr/local.keep


and watch to see what breaks.  Chances are they're used by whatever's in /usr/local/bin, though, so look there first.