What is a gating continuous integration (CI) system?
A traditional CI system is reactive, it detects regressions, but requires human intervention for repairs. 

Traditional CI relies on automated quality checks performed on the already integrated code - after changes are committed/merged into the integration branch (typically triggered by those events). The results of these checks indicate if the integration was successful, i.e. if the branch at that point is a good candidate for any subsequent steps towards delivery/deployment or if regressions occurred. Some regressions are considered catastrophic (for example build failures) and must be repaired ASAP as they're effectively blocking development on the branch. Unfortunately in most cases the repairs require human intervention, which can be a significant bottleneck.

By contrast a gating CI system is proactive, it prevents regressions and, with proper orchestration, can be entirely automated.

Gating CI performs the quality checks on candidate changes prior to them being committed/merged into the integration branch. If the checks are successful the changes are committed/merged into the branch, ideally in an automated manner, performed by the CI system itself, without human intervention. If the checks reveal regressions considered catastrophic/blocking the candidates are rejected (i.e. their commit/merge into the branch is prevented). Hence their gating characteristic - they will prevent "bad" changes from even reaching the branch.

A simple example would be a pull request (PR) based integration with one or more automated validations (for example a build and maybe a smoketest) hooked up in the PR processing which must be successful for the PR to be merged. A fairly common practice for many development teams.

Unfortunately rejecting "bad" changes alone is not sufficient to ensure the integration branch maintains a minimum quality level at all times.

It is always possible to have multiple candidates outstanding at the same time. But having multiple quality checks (on different candidates) active at the same time means that at least some of the checks do not take into account the other candidates in flight, leaving room for regressions due to changes interfering with each-other - see a simple example in How can successfully pre-verified changes cause regressions that should have been caught?. 

It is for this reason that those using a PR-based system as mentioned above can probably recall that the branch can still see breakages once in a while.

This can be addressed with proper orchestration of the quality checks. The purpose of the orchestration is to completely eliminate the risk of interference between changes by controlling when each check is started/stopped and what candidate change(s) it covers, based on one or more inputs which can include: other candidates submissions/current states, results of other checks, actual commits/merges, availability of resources necessary for the checks, etc.

A very basic orchestration algorithm for the PR-based CI system mentioned above would be, for example, to only allow the merge if the tip of the branch at merge time is the same as it was when the automated PR validation started (i.e. no other change was committed/merged in the mean time). Effectively serializing PR processing, it could work great for small teams, but probably will not be able to keep up in a large scale project.

The orchestration algorithm (or lack thereof) is ultimately what dictates a gated CI system's scalability, performance ability to maintain the integration branch' quality level. An algorithm 100% efficient in preventing blocking regressions would no longer require human intervention for normal operation, a gated CI system using such algorithm can typically be entirely automated - a pipeline accepting change candidates at one end, filtering out regression-causing changes on a mid tap and outputting guaranteed stable branch tags at the other end.

It's probably worth mentioning that using a traditional or a gating CI only covers the integration stage and brings no fundamental restrictions on the CD (delivery/deployment) portion of a CI/CD pipeline - any traditional CI/CD tool can be plugged in behind a gated CI to perform additional (but non-gating) CI verifications (if needed) and the CD steps. Similarly any release strategy (see Should I use release branch to push my changes?) can be used with either of them.

Note: for distributed version control systems like git the context of the answer is the master/origin/central repository, not the local replicas.