anchor.diff:  
   Managing the Datastore
   ~~~~~~~~~~~~~~~~~~~~~~
  -GRR currently ships with two usable datastores, MongoDB and MySQL, and an extra
  -in memory only test datastore FakeDataStore.
  -
  -By default GRR will use the MongoDataStore and expect to find it on localhost.
  -You can configure it in the config file, e.g.
  -
  ------------------------------------------
  -[MongoDataStore]
  -server = mongodb.host.example.com
  -port = 27017
  -db_name = grr
  ------------------------------------------
  -
  -We also provide a useful script [scripts/database_reset.sh] to drop the Mongo
  -database which is handy while testing things.
  -
  +GRR currently ships with a sharded SQLite datatbase that is used by default, and a MySQL Advanced datastore that may be a better choice if you have significant in-house MySQL experience and resources.
   
   Performance
   -----------
   
  -GRR is designed to scale linearly, which it mostly does. It has been run in test
  -environments with with greater than 100,000 clients reporting and hunting
  -without problems. However, this depends significantly on the datastore
  +GRR is designed to scale linearly, but performance depends significantly on the datastore
   implementation, how it is being run, and the hardware it is running on.
  -
  -As of April 2013, we don't have a clear set of metrics on performance in the
  -main release, however there is work being done to create these.
   
   Deployment Performance Factors
   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   
   The key issue with performance generally relates to how the server is deployed.
   
  -- If you are running from the prepackaged release you are likely running the
  -  grr_server.py binary. This binary runs all components in a single process,
  -  which means you are using a single core and it will perform badly. We do this
  -  to make life easy for first time users. In production you need to run each
  -  component separately (see /etc/init/grr*) in its own process (this will also
  -  help with more intelligent logging). You can switch on upstart based systems
  -  easily using the script in scripts/initctl_switch.sh.
  -- Additionally, you will probably want to run more than one worker. In a large
  +- You will probably want to run more than one worker. In a large
     deployment where you are running numerous hunts it makes sense to run 20+
     workers. As long as the datastore scales, the more workers you have the faster
     things get done. We previously had a config setting that forked worker processes off, but this turned out to play badly with datastore connection pools, the stats store, and monitoring ports so it was removed.  We're planning dockerization to make this easier, but for now you should create multiple workers using different init scripts.
  
positive.code: The cron allows for scheduling flows to run regularly on the GRR server.
  This is currently used to collect statistics and do cleanup on the database.
  The cron runs as part of the workers.
  
  
  
  
  Customizing the client
  ----------------------
  
negative.code: - As well as having a better performing http server, if you are moving a lot of
    traffic you probably want to run multiple http servers. Again, assuming your
    datastore handles it, these should scale linearly.
  - The admin UI component is usually under light load, but you can run as many as you want for redundancy.
  - Foreman check frequency. By default the foreman_check_frequency in the client
    configuration is set to 1 hr. This variable controls how often a client
    checks if there are hunts scheduled for it. Increasing this number slows down how fast a hunt ramps up, which
    normalizes the load at the cost of making the hunt slower (this is useful in
    large deployments). Decreasing this number means clients pick up hunts sooner, but each foreman check incurs a penalty on the frontend server, as it must queue up a check against the rules.
  
  Datastore Performance
  ~~~~~~~~~~~~~~~~~~~~~
  
  If you are not CPU bound on the individual components (workers, http server)
  then the key performance differentiator will be the datastore. Significant performance improvement work has been done on the MySQL Advanced and SQLite datastores but more can still be done. Improvements here will yield large gains, pull requests welcome :)
  
  Running the GRR HTTP Server In Apache
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  