How can I update custom order fields in the frontend
Correct, you need to use fields[customFieldName] as the input name.

The following form should do it:

&lt;form method="POST"&gt;
    &lt;input type="hidden" name="action" value="commerce/cart/updateCart"&gt;
    &lt;input type="text" name="fields[myCustomField]" value="Test123"&gt;
    ...
    &lt;input type="submit" value="Update Custom Field"&gt;
&lt;/form&gt;