How to delete a form element via ajax?
Adding a value using that code won't actually work since you've only defined one textfield, and thus, only one value will be stored. All the new fields returned will have the same id/name as the first one so only one form value will be sent back to the server when the form is submitted.

You need to add an extra layer to the form structure to make each field instance unique, say by including an instance number in each id/name.
Something like $form['checkboxes_fieldset'][$i]['number'] where $i is increased by one for each field instance. That goes for all the elements inside the fieldset. You generate these fields in a loop, but before that you check how many instances of the field are already present in the form state so you know how many iterations are needed in the loop. For each iteration, you also check if the form was submitted and if the current field instance matches the name of the button used to submit the form. If so, you skip adding that field instance to the form (don't increment the instance counter this iteration to keep the total count synced).

The submit handlers for adding/removing fields simply set the 'rebuild' flag in the form state and return the fieldset to be updated (all instances get re-rendered each time).
Adding/deleting field instances then become an indirect action of rebuilding the form with the correct number of field instances.

The examples module on drupal.org has an "add more" AJAX example which is fairly similar to what you want, except that it always removes the last field instance.