Stain and seal bathroom tiled floor grout - remove upper grout layer?
That layer below is likely the normal tile adhesive.

The grout is the finishing layer above that, so that broken or missing part should be regrouted.

There are tools for removing old grout. You can do as little or as much as you feel you need, but I would just repair faulty sections if there are only a few. When most are faulty then you have to do all of it.