anchor.diff:    lastAcknowledgedIndexReplicated: 0 # <6>
   }
   ----
  -<1> the first message is meta data written in binary
  -<2> the type of header is aliased as the name SCQStore.
  -<3> the `writePosition` is the first bound value. It is the highest known byte which has been written to and is updated atomically.
  -<4> the roll cycle is daily.
  -<5> this class controls how it will be indexed on demand.  This adds meta data entries for indexed lookup.
  -<6> this is the highest message index which was acknowledged by a replica.
  -
  -For us a key feature of CQ is not just how the data structure is arranged,
  -but also how transparently this binary data structure can be inspected.
  -
  -NOTE: The `SCQStore` "bootstraps" the queue itself.  If you provided another, custom implementation the queue could behave as you wish, provided it support the same interface.  The Rolling and Indexing strategies can also be customized.
  -
  -If we look at the last message, you can see the message type, the type of the payload and the value of all the fields.
  +<1> First message is meta-data written in binary
  +<2> Type of header is aliased as the name `SCQStore`.
  +<3> `writePosition` is the first bound value. It is the highest known byte which has been written to, and is updated atomically.
  +<4> Roll cycle is daily.
  +<5> This class controls how it will be indexed on-demand.  This adds meta-data entries for indexed lookup.
  +<6> Highest message index which was acknowledged by a replica.
  +
  +For us a key feature of Chronicle Queue is not just how the data structure is arranged, but also how transparently this binary data structure can be inspected.
  +
  +NOTE: The `SCQStore` "bootstraps" the queue itself.  If you provided another, custom implementation, the queue could behave as you wish, provided it support the same interface.  The rolling and indexing strategies can also be customized.
  +
  +If we look at the last message, you can see the message type, the type of the payload, and the value of all the fields.
   
   [source, java]
   ----
  
positive.code: 
  == The Big Question
  
  How is Chronicle Queue being used for Big Data solutions in Java and how does it work under the covers to support this?
  
  == What is Chronicle Queue?
  
  Chronicle Queue (CQ) is a persisted journal of messages.
  CQ supports concurrent writers and readers even across multiple JVMs on the same machine.
  Every reader sees every message and a reader can join at any time and still see every message.
  We assume you can read/scan through messages fast enough that even if you are not interested in most messages it will still be fast enough.
  These are not consumers as such and messages don't disappear after reading them.
  
  image::http://chronicle.software/wp-content/uploads/2014/07/Chronicle-diagram_005.jpg[align=center]
  
  This retaining every message has some advantages
  
  - a message can be replayed any many times as is needed.
  - a day of production messages can be replayed in testing months later.
  - this reduces the requirement for logging almost entirely, speeding up your application.
  - greater transparency leads to optimisations you might have missed without a detailed record of every event your process.
  
  One of our clients, first implemented their trading system without CQ, and achieved an average latency of 35 micro-seconds tick to trade,
  however after using CQ the latency was reduced to 23 micro-seconds.
  
  === What makes Chronicle Queue different from similar solutions
  
  ==== Speed
  
  CQ is designed to support hundred of thousands of messages per second comfortably.
  It can also handle multi-second bursts into the millions of messages per second.
  One of our clients reported they are handling bursts of 24 million messages per second with a cluster of 6 servers.
  
  ==== Without Flow Control
  
  CQ is an unbounded queue without flow control between the producer and the consumer.
  Replication supports flow control but this is not used by default.
  A lack of flow control is what we call a "producer centric" solution.
  
  ==== Consumer Centric solution
  
  To put that in context, most messaging solutions have flow control.  They are what we call a "consumer centric" solution.
  Consumer centric solutions make sense in many application esp between servers and client GUIs.
  Any data you send to a client GUI is for display to a human.
  Desktop applications can be run on a variety of machines, over a variety of networks, and a human can only see a limited rate of data.
  Any faster than about 20 times a second and it is just appears as just a blur and counter productive.
  
  It makes a lot sense for a desktop application to only receive as much data as it can handle, as much data as the person having to view it and to push back on the server when it cannot handle this data rate.
  http://www.reactive-streams.org/[Reactive Streams] are an excellent way to model such client centric solutions.
  
  ==== Producer Centric solution
  
  CQ doesn't have flow control deliberately, as flow control isn't always possible, or even desirable.
  
  Some examples of systems where CQ is often used are
  
  - Market Data gateways, you can't slow down an exchange because you are not keeping up.
  - Compliance systems, you have to feed information to them but you never want to be slowed down by them.
  - Core trading systems, as these feed from Market Data gateways as you want to be as fast as possible all the time.
  
  In a Producer Centric solution, the producer isn't slowed down by a slow consumer and that consumer more than main memory behind the producer. The consumer might be an overnight batch job and be a whole day or week behind.
  
  This has some advantages
  
  - You can reproduce a bug even if it only occurs one in a million messages, by replaying all the messages which led to that bug triggering, but more importantly you can have confidence *the* bug, rather than *a* bug has been fixed.
  - You can test every micro-service independently as there is no flow control interaction between them. If you have flow control, say 20 services, any one of those services could slow down any producer, and its producer etc until your entire system locks up.  This means the only real performance test is a complete system.
  - You can test a micro-service replaying from the same input file repeatedly without the producer or down stream consumers running.
  - You can restart and upgrade a service by replaying its *outputs* to ensure it is committed to the decisions/outcomes it has produced in the past.  This allows you to change your service and know that even those your new version might have made different decision, it can honour those already made.
  - Flow control is designed to smooth out burst and jitter in the system, but can also hide such jitter as result.
  
  The main disadvantage is that this assumes disk space is cheap.  If you look at retail prices for Enteprise grade disks you can get TBs of disk for a few hundred dollars.
  Many Investment Banks, don't take this approach and internal charge rates for managed storage can be orders of magnitude higher.
  I have worked on systems that have more free memory than free disk space.
  
  == What technology does Chronicle Queue using?
  
  What might be surprising is that CQ is written entirely in pure Java.
  It can outperform many data storage solutions written in C.
  You might be wondering, how is this possible given that well written C is usually faster than Java.
  
  You need a degree of protection between your application and your data storage to minimise the risk of corruption.
  As Java uses a JVM, it already has an abstraction layer and a degree of protection, and if an application throws an exception this doesn't mean the data structure is corrupted.
  To get a degree of isolation in C, many data storage solutions use TCP.
  The overhead of using TCP, even over loopback can exceed the benefit of using C,
  and the throughput/latencies of CQ can be 10x greater by being able to use the data in process.
  CQ supports sharing of the data structure in memory for multiple JVMs avoiding the need to use TCP to share data.
  
  === What does the data structure look like?
  
  One of our key objectives is transparency, and we have worked hard to make sure you can see exactly
  what has been written to this data structure.
  
  We use YAML as this is a format which is designed to be readable.
  JSON and XML we see as a subset of Javascript and SGML, and not as readable. We support JSON for messaging with browsers.
  For performance reasons we use a binary form of YAML, which can be automatically translated to YAML for viewing.
  
  .Order class extends AbstractMarshallable for helper methods
  [source, java]
  
negative.code: 
  === How do we reduce garbage?
  
  For the most latency-sensitive systems, you may want to keep your allocation rate to below `300` KB/s.
  At this rate you will produce less than `24` GB of garbage a day, and
  if your `Eden space` is larger than this, you can run all day without a minor collection.  A GC is something that you can do as an overnight maintainence task.
  Reduce your garbage-per-day to less than `5` GB, and you might be able to run all week without a GC.
  
  We have a number of strategies to minimise garbage; the key one being that we translate directly between on-heap and native memory without intermediate temporary objects.
  
  We use object pools where appropriate, and we support reading into mutable objects.
  
  For text data we support both a `String` pool and reading to/from `StringBuilder`.
  
  
  '''
  <<../README.adoc#,Back to ReadMe>>
  