Is it possible to add a Child Of modifier without changing position of the child?
Yes, in the constraint, click on the Set Inverse button. By default, the child will stick to its parent, but if you click Set Inverse it will keep its original position (like a Parent > Keep Offset).