Pooled Build Server and Multibranch development to stabilize the trunk?
Attempting to run your validation on branches, before being integrated into the trunk won't get you very far. The reason is that fundamentally such validations are performed in isolation, most often they will not actually reflect the trunk picture after the branch is merged. 

The only case in which the branch validation is, well, valid, is if no other change (of any kind) makes it into the trunk between when the validation started (assuming the latest trunk changes are merged into the branch at that time) and when the branch is merged back into the trunk.

If any other change makes it into the trunk while the validation is in progress it won't be taken into consideration, leaving room for breakage when the branch is merged into the trunk. To prevent that you'd have to merge again the trunk into the branch - to pick up the latest trunk changes - and re-run the validation.

Effectively this means the branch validations (or at least the successful ones allowing the branch to be merged into the trunk) need to be entirely serialized if you want to keep the trunk stable. Which may be manageable for a while if you keep a reasonably small number of such branches (maybe multiple devs working on the same branch?). How many branches? Depends on how stable you want the trunk to be: if you want to always pass your integration tests, your serialized validations would need to run the integration tests. At 40 min each you can support max ~60 branches (assuming the merges are automated and can be performed 24x7, which isn't trivial). Without automated merges the number of branches will be a lot smaller. This won't scale, the branch lifespans will gradually increase, diverging more and more from CI and will eventually wear down your team.

The same reasoning explains why devs shouldn't actually have to run all tests if they are too lengthy - by the time they complete them the results may be invalidated by other changes committed to the trunk. Also - results tests done by developers are often not that relevant ("But it built fine in my workspace!").

There are only 2 possible approaches (that I know of) to get relevant validations:


the ones performed by the traditional CI tools - performed after the changes are committed to the trunk, reflecting the actual trunk reality, with all changes integrated. The only problem is if these validations fail - effectively blocking further development on the trunk until the culprits are found and the regressions fixed. Which may be (very) difficult in larger scale projects - as you already noticed.
the ones performed by gating CI tools - only 2 out there that I know of: Zuul and ApartCI (which is my baby). These validations are performed prior to merging changes into the trunk, but in an orchestrated manner, ensuring they remain relevant - taking into consideration all other changes in flight that will eventually be present in the trunk by the time a particular change is merged into the trunk. The actual change merging into the trunk is entirely automated, executed by the CI tool itself, immediately after the relevant validations are completed successfully. These are, IMHO, the only ones capable of guaranteeing trunk stability in larger scale projects.


Pooling resources is a good thing - give them to the CI tool - that's how you maximize their usage.