I have a system that uses the at command to schedule simple commands to be executed later. The command in question uses a lock so that multiple commands started at the same time execute sequentially (mutex).At one point I've logged into a machine and saw that something caused ~250 such jobs to be queued. It shouldn't be a serious issue and after waiting for the specified time I saw that commands started running, and it looked sequential - as every command run it was removed from the list returned by atq, at a rate of about 1 per second.Eventually though, there were 2 jobs remaining on the queue - well after the time they were supposed to run. Here's the output from watch atq:Every 2.0s: atq                                    Mon Jan  4 17:55:58 2016242     Tue Jan  5 17:44:00 2016 a root241     Tue Jan  5 17:44:00 2016 a rootAlso, running $ sudo find /var/spool/cron/at*, I see this:/var/spool/cron/atjobs/var/spool/cron/atjobs/.SEQ/var/spool/cron/atjobs/a000f201714448/var/spool/cron/atjobs/a000f101714448/var/spool/cron/atspoolThe atd daemon is running, and comparing its PID to other system daemons, it looks like it hasn't crashed and respawned.I know I can atrm the jobs and just run the commands by hand, but I'm more interested in understanding why it happens and how to prevent it from happening next time. What could cause these jobs to get stuck?The system in question is an Ubuntu 12.04 (precise) fully up to date.
at jobs stuck?
ubuntu;at