How to get rid of ants on ceiling?
You can powder a 3-6 inch border around the outlet box on the ceiling with diatomaceous earth (I'd use food grade since this is in the kitchen) to control the ants.  Then I'd use a fire-rated putty to seal the outlet box.

When I lived in an apartment, there was a large colony of ants living by the pool and they would come out of little cracks in the walls.  I ended up caulking everywhere I saw a crack or hole and powdering around all the doors windows and that stopped them.