Why can't I modify my comment form using hook_form_alter
It looks like the label "Comment*" is not from comment_body container wrapper, then again comment_body shouldnt be of type container.

Most likely the actual comment field is somewhere not visible of your debug screenshot. This is because drupal core comment module form doesnt use normal form api structure where containers only reference children and reverse. Instead the actual field is placed directly to container. Luckily drupal_render has no problem with neither structure.

Removing the label should not require specifying language on the actual field. It should be as easy as

$form['comment_body']['#title']='';

Instead with funky comment form render array you need to specify language of the containers sub items, hence the path transforms to

$form['comment_body'][LANGUAGE_NONE][0]['#title']=''