How does one call civicrm_api3_system_check API
Your example of calling System.check with an empty $params array should work. For example, this is what I get if I call the API from the command line:

$ cv php:eval 'return civicrm_api3("System","check", array());'
{
    "is_error": 0,
    "version": 3,
    "count": 6,
    "values": {
        "1": {
            "name": "checkExtensionUpgrades",
            "message": "Extension upgrades are pending.  Please visit &lt;a href=\"/civicrm/admin/extensions/upgrade?reset=1\"&gt;the upgrade page&lt;/a&gt; to run them.",
            "title": "Run Extension Upgrades",
            "severity": "error",
            "severity_id": 4,
            "is_visible": 1,
            "icon": "fa-plug",
            "id": 1
        },
        ...


(For other media, you can use the API Explorer to get code snippets.)

There are a lot of checks, and it's possible that they're slow on some systems. Alternatively, maybe one of the checks has a bug, relies on an incorrect assumption about the environment or data, or has an interaction with third-party code?