Updating a custom field set for case entity Civi API
This looks like a bug to me. Looking at the code you can see in civicrm_api3_case_create that if the ID is already set it calls civicrm_api3_case_update. 

civicrm_api3_case_update has no call to the CRM_Case_BAO_Case::create method, which is the only mention I see of custom parameter creation in the BAO.

In fact api/v3/Case.php contains only one mention of the word "custom". It's surprising that something like this could have been left out and maybe I'm missing something, but when I tried creating a custom field for a Case locally and setting some custom values nothing was created.

I'd suggest mentioning this in Mattermost or just going ahead and creating a bug report for it on Jira

Edit:

This issue is about custom data being ignored by the case API but is still open.