title: Need disk based priority queue library, preferably for C
body: <p>I have a queuing mechanism in C on Unix. It accepts XML transactions. Some transactions contain records to be stored. Other transactions request those transactions. The transactions get stored in a file, which is a home-grown queue. First in, first out, very simple. Header area at start of file, keeps track of next position to read from, and next position to write to. We use file locking, but not semaphores as retrieval is polled from remote systems. And there's only one program that accesses the queues. It's in C. Been working fine for years.</p>
  
  <p>Now we have to expand the system. The transactions will contain an extra XML tag. We have to selectively retrieve based on the values of that tag. We are going from a simple queue to a priority queue. There can be many different values in the tag. Say AX, BX, CX, FL and TS. Transactions get added to the queue in order received. We need to be able to retrieve them either in order they were received, or retrieve the next transaction where the tag is FL. Or TS. Or (CS or FL). Or not AX.</p>
  
  <p>How best to do this?</p>
  
  <p>Simple and fast are what we need. Several options come to mind:</p>
  
  <ol>
  <li>Use something like Berkely DB to turn the queue into a database of sorts.</li>
  <li>Tap into a PostgreSQL database, create a table that can be used as a priority queue.</li>
  <li>Find a C library that will do what we want.</li>
  <li>Write our own disk-based priority queue.</li>
  </ol>
  
  <p>We have some constraints. Time is ticking away and this needs to be done in a few weeks. C for fast insertion into the system. Maybe Python if we can tapdance fast enough to convert all the other business logic in the program that accesses the queue. Prefer not to use PostgreSQL as we have no control over the database system and the DBA has nasty habits over what he considers "his" and we have no reliability of uptime even though this is a critical system. Politics, huh!! DBA has also said that using a PostgreSQL table is not an efficient way of doing it. We prefer something that is localized so we can control it. Got to be lightning fast to handle a lot of transactions per minute.</p>
  
  <p>I'm open to any suggestions, even far-out ones. The more suggestions the better.</p>