what is the brown foam and sludge in my water softener brine tank?
"Iron Reducing Bacteria" create energy by oxidizing soluble iron in the water.  In the process this creates insoluble iron which settles out and leaves iron staining.  The bacteria themselves create biofilms or foamy messes in several areas, most notably toilet tanks and water heaters.  Look in your toilet tank, if you have similar foam and iron staining you probably have iron reducing bacteria in the water.

If you do have iron reducing bacteria you have to deal with them at the source, your well.  Generally this involves pouring bleach or another sanitizer down the well, pulling and circulating that through the plumbing system and then letting it stand in the pipes and well for 12-24 hours.  This is not a permanent fix, but it can get you a few months respite.  Any well company can assist with this process or you can find information online.

The problem is your water softener resin beads will be destroyed by this process, normally the water softener is bypassed during sanitization.  You need to sanitize the plumbing system separately, and clean the water softener by hand.  I recommend replacing the resin beads and sanitizing the resin tanks and brine tank at the same time.

A more expensive but permanent solution is to install water treatment systems to deal with the bacteria.  Usually this involves chlorine injection, some contact time in a tank to kill the bacteria, and then running the water through a filter or reverse osmosis system to remove the dead bacteria and most of the chlorine.  This is very expensive.