What is the correct way to update existing module's rules programmatically? I want to provide an update for my custom module, which also has some rules provided via hook_default_rules_configuration. 

Now, how can I update those existing rules? Can I re-import them? Override them somehow? I cant find anything about this in the documentary or an respective hook.

There seems to be a hook to respond to rules being updated, so I guess there must be a way to provide updates for custom rules...

At first I thought it was hook_default_rules_configuration_alter but it doesnt seem to work as expected since I just want to import JSON.

For instance, using views I can simply load all .inc files and save them via views_save_view like in this question - isnt there a similar way in rules?
You can export the rules configuration via the Rules UI and then implement hook_default_rules_configuration_alter() like this:

/**
 * Implements hook_default_rules_configuration_alter().
 */
function MODULENAME_default_rules_configuration_alter(&amp;$data) {
  if (isset($data['commerce_cart_add_to_cart_message'])) {
    $data['commerce_cart_add_to_cart_message'] = entity_import('rules_config', '{ "commerce_cart_add_to_cart_message" : {
        "LABEL" : "Display an Add to Cart message",
        "PLUGIN" : "reaction rule",
        "OWNER" : "rules",
        "TAGS" : [ "Commerce Cart" ],
        "REQUIRES" : [ "rules", "commerce_cart" ],
        "ON" : { "commerce_cart_product_add" : [] },
        "IF" : [
          { "data_is" : { "data" : [ "commerce-order:commerce-nocart" ], "value" : "0" } }
        ],
        "DO" : [
          { "commerce_cart_add_to_cart_message" : { "commerce_product" : [ "commerce-product" ] } }
        ]
      }
    }');
  }
}


Just change the instances of $data['commerce_cart_add_to_cart_message'] to use your rule machine name and replace the export string with your own.

You can also use rules_import() instead of entity_import(). Either is acceptable.

Since the rules module loads default rules from code into the database and from then on loads them from the database, it won't immediately pick up your override.
To make it do so navigate to the Rules UI and revert the rule. It will revert back to the default code, which will be your override.

To programmatically revert a rule you can do this (again replacing the rule machine name):

if ($rules_config = rules_config_load('commerce_cart_add_to_cart_message')) {
  $rules_config-&gt;delete();
}


An example usage of that may be to do it during an update function for a custom module.

It may seem wring to be deleting it but delete and revert are equivalent for rules that are provided by code because delete means delete from the database. If this was a rule created via the UI then it would be gone, however a rule from code will just delete from the DB and then read in again from the code.

Unfortunately there are currently only drush commands for enable/disable of rules.
There is a feature request for a delete/revert command at https://www.drupal.org/node/2507187