Isn't my ground different from the factory's ground?
Components are damaged by two or more of their pins being at a large enough potential difference. If the component has a conductive case, or pad, then that counts as a 'pin' too.

It's possible to break them by trying to charge them up to a new potential through one sensitive pin, while the voltage of the other pins is held more or less constant through capacitance to ground. That can be the situation where you, perhaps charged to 15kV with respect to ground, pick up a component that's at ground potential by (say) the gate lead.

Conductive packaging shorts all the pins together. What you do is to bring the conductive bag to your potential first. Any charging current that has to flow into the component does so through all pins, so does not damage the component.

Let's say an insulated carton of components in conductive bags charged to 100kV arrives at your workstation. You and the workstation are grounded. You open the carton, and as soon as you touch a component bag, a current flows between you and the bag to discharge it down to ground potential. Meanwhile, the bag has maintained all the component pins at the same potential, so no damaging voltage is applied across the component. Now you and component are at the same potential, you can open and touch.

Why did the component arrive at 100kV? Surely the other factory ground is not that different to yours? No, but the last bit of the trip might have been carried by a guy with nylon shoes. When stuff is properly packed, it doesn't matter if intermediate stages of the journey take it to potential way different from ground.