datastax opscenter "SchedulesNotLoaded" error
Question: I am using Datastax OpsCenter v5.2.4 with DSE v4.8.4-1. Since few days ago,
I've been not able to retrieve the result of "Best Practice Service" both from
API and opscenter UI. When I try to get it, I get errors like below.

GUI:

> Could not retrieve best practice rules: Scheduled jobs have not been loaded
> yet. There may be a connectivity problem with Cassandra.

opscenterd.log

    
    
    2016-07-01 19:47:50+0000 [] ERROR: Problem while calling decorator (SchedulesNotLoaded): Scheduled jobs have not been loaded yet. There may be a connectivity problem with Cassandra.
      File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1020, in _inlineCallbacks
        result = g.send(result)
    
      File "/usr/lib/python2.7/dist-packages/opscenterd/WebServer.py", line 1939, in SchedulesGetController
    
      File "/usr/lib/python2.7/dist-packages/opscenterd/Schedule.py", line 213, in getAllSchedules
    
      File "/usr/lib/python2.7/dist-packages/opscenterd/Schedule.py", line 175, in _assert_loaded
    

I've tried to restart opscenterd service as well as rebooting the opscenter
machine itself but it didn't make any difference. The error says there might
be some connectivity issue, but what port/protocol is opscenter using to load
these scheduled jobs? (there is no firewall between opscenter and cassandra
nodes) There is no alert in the cluster, and agents are all connected
according to opscenter's GUI.

I couldn't find any relevant trouble shooting documentation ... how can we
recover opscenter from this situation?


Answer: Issue is around when schedules settings get in messed up state. If you
schedule a one time only run, and opscenterd is down when it is scheduled to
run, then on startup opscenterd dies loading that schedule.

If you don't have anything particularly important stored there is an easy fix.
Shutdown off opscenter, use cqlsh to `drop keyspace "OpsCenter";` and restart
opscenter.

Otherwise you have to hand clean up some of the schedules in
`OpsCenter.settings` table that got messed up. This is fixed in 6.0, so if you
upgrade it wont happen again.

