Custom database connection What is the best way to connect to a remote database through drupal? Would the correct method be to use the settings.php with the database in there, and use the built in db_select, db_query() etc functions to access the remote database?

I have a system that has a bunch of entities, which are different 'servers'. Each server may or may not have a different database for up to 4 different services. Currently, I am storing the database connection details inside of the entity fields, and calling them using entity_load() as needed. In doing this, I am having to write my own SQL library and queries to retrieve data from the external databases. 

I am wondering; if instead of writing the values to the entity fields, if I wrote them to maybe an external .php file (maybe dbconfig.php) or something, if there was a way I could still use the built in database functions and methods to access it, or if Drupal only reads from the settings.php for database connections. 

I also question if using it on a table that doesn't use the same structure as drupal would even work. (It's a completely different service with it's own tables and structures). 

What's the best way to handle this type of situation? I really would like to get away from writing my own library to use for remote connections, as being able to do pagination and all that using the built in methods is extremely helpful and already been tested and hardened.

Just trying to find the best way to get access to the drupal database methods/functions and still be able to update/edit/add new databases when they are needed without risking security, and without having to manually change permissions on the settings.php before making new changes. (The servers currently are automatically added as new entities by scanning a single remote db for new entries then using entity_create() to add them). 

Is it possible for me to put an include('databases.settings.php'); inside of settings.php, and use $databases['database_name']['default'] = $info_array; inside of that file (and keep it writable) to store the new database values?
You don't have to re-invent the database abstraction layer to access external databases. You can switch the database connection dynamically.

Using Database::addConnectionInfo() you can add your database connection information dynamically. Then use Database::setActiveConnection() and Database::getConnection() with your respective target database identifiers for doing your queries with.

Entity (field) queries won't work obviously, if the external database doesn't have entity tables. But static and dynamic queries will. The latter allow for including the core pagination, if desired.

After you queried your external database, use Database::setActiveConnection() without parameters to reset the connection to the defaults. This is important for Drupal to being able to access its own tables after your queries.

Check the Database API documentation and the database class for more in-depth information. (This will also work in Drupal 7 and its database class.)