Problem with Boolean modifier-difference
The reason boolean difference fails on the 4 is due to non-manifold geometry. You can select non-manifold geometry with CtrlShiftAltM in edit mode. In your case, this reveals an unconnected face on the extruded version of the 4 (likely due to an extra edge before extruding):



I added an extra edge ring and merged the corresponding faces:



And then the boolean succeeds.