Is there any way to change the directory bash works out of, so as not to junk up my Home folder?
Is there any way to change the directory bash works out of.....


This is a misunderstanding of what Bash is.  Bash is a Unix shell that's a command line interface to the operating system.  

It doesn't "work out" of your Home folder.  When you start a non-login, interactive shell, it starts you out in your HOME folder as a matter of convenience.

For simplicity sake, Bash is just a "portal" (Terminal window) into the operating system; nothing more.


  ...so as not to junk up my Home folder?


Your HOME folder is becoming "junked up" because whatever you're installing is using the default location $HOME for it's support files.

Just looking at a sample of the files/folders, things like JUNK, New Folder with Items, doc, core, layers, news, etc. are either files created by applications sourced from somewhere other than the App Store or created by the user.


  ...but it’s like every application wants to store a million configuration files in my home directory. 


Apps usually store their files in ~/Library or more granular, in ~/Library/Application Support (this is the Library directory in the individual user's HOME directory).  There are exceptions...iCloud, and Adobe Create Cloud store files in the HOME directory.  I personally put my VirtualBox VM's in my HOME directory instead of my ~/Documents directory. 


  I’m actually surprised Apple allows this.


Thank goodness they do!  I wouldn't welcome any OS that restricts me from modifying MY home directory the way I want it to.

That said, it's important to note that this is not an "Apple thing."  This is a BSD and more generally a Unix thing.  Your HOME directory is a predefined variable in the OS and many applications and scripts use this directory to store it's files.

As stated before, App Store apps (the ones with oversight by Apple) conform to a standard place for their support files.  Things you download from Github...well, you're at their mercy.


  ...Where I want it to look like this with a nice, neat little folder for
  the shell to do whatever it wants.


You could do this, but bear in mind that the app you installed will be looking for it's support files in directory defined by the environment variable HOME.  Move those files somewhere else and you could break the app.  

To make this work, you would have to let the app know (by modifying it) that it's files are located in $HOME/bash.

TL;DR

You're expecting HOME to be a pristine place that doesn't get touched; that's an ill conceived expectation.  Use the HOME directory for what it's designed for - a user support directory and focus on keeping ~/Documents, ~/Downloads, ~/Music, nice and clean.