Are there ways to restrict the number of concurrent connections to a SqlSourceSystem?
We are thinking about writing a batch job that will split large fetch into smaller requests. If we do this and there are m workers available with n threads, could we end up with m*n requests to the source system concurrently?
Basically yes but it’s a bit more complicated. On worker nodes, we use shared db connections for reads so multiple open requests (e.g. cursors) may use the same physical connection. Also, if updates are involved, each update will acquire a connection, perform it’s work and release it.
Is there a way to limit the number of concurrent connections to protect the database?
Not at the moment, at least directly. Indirectly it’s possible to limit the maximum concurrent computing threads for workers and set a max concurrency for batch/mapReduce jobs (both at the worker and job levels).