Using delegates to avoid duplicate creation of resources
The problem here is that you want to be able to use a large number of unrelated objects (i.e. not implementing a common interface).  One solution would be to define a usable interface and define wrapper implementations for each of these objects.  Something like:

public interface IDataSource
{
    byte[] DownloadUrl(String url);
}


Your download class would use implementations of this interface, and you could provide implementations for HttpClient, WebClient, etc. like:

public class HttpClientDataSource : IDataSource
{
    public HttpClientDataSource(HttpClient client){}
}


All that said, this doesn't seem to really matter in your case.  I'm not convinced that allocating an HttpClient for internal use is exactly wasteful of resources.  Also, by forcing this decision onto the users of your library you've made it more complicated to use.  You'll have to explain to your users how to choose between different HTTP implementations.  Part of designing a good library is trading off flexibility and simplicity, and I would say the simpler interface is better.