Is there any published paper about automated planning software used in the ESA Rosetta mission?
Well, as far as I can tell, there was no specific automated planning tool for the Rosetta mission as we understand automated planning systems in ICAPS, which is the most prominent conference on Automated Planning and Scheduling.

The hardest part was to plan the journey from earth to a common location where both the spacecraft and the comet could meet and this was done mainly in two stages: one making the spacecraft orbit around some bodies to save energy (cruise phase) and this was mainly done with astronomical computations; after hibernating for a good number of years, it had to wake up and fly (now using its own power) to meet the comet. There was, additionally, another scheduling problem in the final landing operation of Philae. The last two operations were self-automated.

I asked someone with a better knowledge of Rosetta and this person told me (right now) that there were mainly scheduling systems (as they are also called in the paper you mention) and that most of the operations were self-automated (ie., the main difficulty was that Rosetta was so far that it had to be able to take some decisions on its own, but still it received orders from the ground segment chiefs in ESTEC, Germany). 

This said, I do not think we will ever see a paper about how to use PDDL (Planning Domain Definition Language) for Rosetta, or how to use LAMA, Fast-downward, FF (or any temporal PDDL planner) or anything of the kind (I mean, the kind of research we all do at ICAPS) but something instead far more domain-dependent.

On the other hand, it is rather unusual for engineers and scientists at ESA to publish papers at scientific venues such as ICAPS, ECAI (European Conference on Artificial Intelligence), AAAI (Association for the Advancement of Artificial Intelligence), IJCAI (the International Joint Conferences on Artificial Intelligence). They instead submit their papers to the SpaceOps symposium. I found a number of papers there:

Geurts K., Fantinati C., Ulamec S., Willnecker R. Rosetta lander: On-comet operations preparation and planning. 13th International Conference on Space Operations, SpaceOps 2014; Pasadena, CA; United States; 5 May 2014 through 9 May 2014

Haddow, CR., Werner D. Mission planning framework - Building the rosetta and bepicolombo planning systems. 13th International Conference on Space Operations, SpaceOps 2014; Pasadena, CA; United States; 5 May 2014 through 9 May 2014

Moussi, A. Fronton JF., Gaudon P., Hallouard D., Mangeret M., Delmas C., Lafaille, V., Durand J. PHILAE lander: A scheduling challenge. 13th International Conference on Space Operations, SpaceOps 2014; Pasadena, CA; United States; 5 May 2014 through 9 May 2014

Eiblmaier MG., Blake R., Williams A., Lodiot S., Godfrey J., Bartesaghi M. Automating ESA's planetary missions: From concept to conclusion. 13th International Conference on Space Operations, SpaceOps 2014; Pasadena, CA; United States; 5 May 2014 through 9 May 2014

Hope this helps,