How does Berkeley DB manage its files?
Basically the philosophy seems to be that its not worth putting much effort on compacting the DB if it will grow again. The way the BDB engine works makes it hard to really reclaim any freed space on workloads that have much insert/update activity, and I think JMS persistence quite possibly is such a workload. The gain from this philosophy, of course, is that on a burst of new messages the DB doesn't need to allocate more pages but can directly write the data on existing pages in the most efficient way. But if the hit on retrieval performance is significant it might be that BDB is indeed not right choice for you workload.

I wonder if the answers provided in these posts in the Oracle forums provide any light to the mystery (the quote comes from the second link).

Understanding compact and database file size

Berkeley DB database file size keeps growing


  There is no specific maximum size that the database must reach before
  page reuse starts. Also, you don't need to close and reopen the
  database handles in order to influence page reusing. BDB reuses pages
  when they are emptied and it does not perform any kind of automatic
  key/page balancing (as it will lead to deadlocks). The factors that
  you should be looking into are the insert, update and delete rates, if
  transactions are long-lived, the deadlock detection policy, the size
  of the database page size and the page fill factor etc.
  
  One example that might explain the behavior you're seeing is one where
  the delete process/thread(s) gets "late" to a page when trying to
  remove on key found on that page. By the time it acquires the write
  lock on that page, there are no more consecutive keys there, and at
  best the delete process/thread only manages to remove just part of
  those keys (the other keys are ones with higher values, as the insert
  and update processes/threads are far ahead), thus the page will not be
  emptied so that it can be placed onto the free list for reuse. If the
  insert and update processes are very active, this may result in rapid
  page splits (or new page allocations), hence sets of keys are moved
  over onto new pages. In addition, the new keys inserted may land on
  already populated pages, thus making things more difficult for the
  delete process. If the new keys being inserted can be distributed
  based on their values to already populated pages where there is room
  for them, then the pages on the free list, if any, will not be reused.