generic load testing tool
From a quick glance, I would say you're on the right track. It might be worthwhile to bust up the work being done by Config into two components:

Config: the pure configuration elements; workers, queue, rate, burst; and
Env: the (effectful) environment state; supplier, performer, collector.

And then think about creating a few boiler-plate functions (not including the polymorphic Env bits):
new :: Config -&gt; IO Env
close :: Env -&gt; IO ()
check :: Env -&gt; IO Text

withConfig :: Config -&gt; (Config -&gt; Env -&gt; IO ()) -&gt; IO ()
withConfig c cont = bracket (new c) close (cont c)

Once you have this, you can create a new environment, play around with it, and properly close it in a robust way. As it stands, and given my workflow preferences, I would think that runLoadN is too monolithic for easy maintenance and extension.
General descriptions of these refactors include the handle pattern and the service pattern. The three layer cake is classic also though I suspect you don't need the extra complexity.
With runLoadN freed of environment setup and teardown you'll be able to see more clearly the key components of the process.
A few vague guesses about other aspect of the code:

I would work towards refactoring away the convenience functions. Do you really need the extra monadic complexity? Can it all be IO?

the collector looks like a WriterT or logger, and might be simplified by using those patterns.

There are a lot of polymorphic types being thrown around. Both m and IO, in particular, exists in Config, which might suggest some refactoring needed. I reckon you could aim for just an Env without any polymorphism.

If you're always altering pure Config types (with toInt), then it's better to change the Config types instead.

I would suggest that best practice is to define a defaultConfig :: Config explicitly, rather than run through Data.Default and use the anonymising def pattern