anchor.diff:  In order to provide stable time synchronization with low maximum error, follow
   these best NTP configuration best practices.
   
  -*Always configure at least four time sources for NTP.* In addition to providing
  -redundancy in case one or more time sources becomes unavailable, The NTP protocol is
  -designed to increase its accuracy with a diversity of sources. Even if your organization
  -provides one or more local time servers, configuring additional remote servers is highly
  -recommended for a robust setup.
  -
  -*Pick servers in your server's local geography.* For example, if your servers are located
  -in Europe, pick servers from the European NTP pool. If your servers are running in a public
  -cloud environment, consult the cloud provider's documentation for a recommended NTP setup.
  -Many cloud providers offer highly accurate clock synchronization as a service.
  -
  -*Use the `iburst` option for faster synchronization at startup*. The `iburst` option
  -instructs `ntpd` to send an initial "burst" of time queries at startup. This typically
  -results in a faster time synchronization when a machine restarts.
  -
  -An example NTP server list may appear as follows:
  -
  -----
  -# Use my organization's internal NTP servers.
  -server ntp1.myorg.internal iburst
  -server ntp2.myorg.internal iburst
  -# Provide several public pool servers for
  -# redundancy and robustness.
  -server 0.pool.ntp.org iburst
  -server 1.pool.ntp.org iburst
  -server 2.pool.ntp.org iburst
  -server 3.pool.ntp.org iburst
  -----
  -
  -TIP: After configuring NTP, use the `ntpq` tool described above to verify that `ntpd` was
  -able to connect to a variety of peers. If no public peers appear, it is possiblbe that
  -the NTP protocol is being blocked by a firewall or other network connectivity issue.
  +*Run ntpdate prior to running NTP server.* If the offset of the local
  +clock is too far from the true time, it can take a long time before the NTP
  +server synchronizes the local clock, even if it's allowed to perform step
  +adjustments.
  +
  +*In certain public cloud environments, use the highly-available NTP server
  +accessible via link-local IP address or other dedicated NTP server provided
  +as a service.* If your cluster is running in a public cloud environment,
  +consult the cloud provider's documentation for the recommended NTP setup.
  +Both AWS and GCE clouds offer dedicated highly available NTP servers accessible
  +from within a cloud instance via link-local IP address.
  +
  +*Unless using highly-available NTP reference server accessible via link-local
  +address, always configure at least four time sources for NTP server at the
  +local machine.* In addition to providing redundancy in case one of time sources
  +becomes unavailable, this might make the configuration more robust since the
  +NTP is designed to increase its accuracy with a diversity of sources in networks
  +with higher round-trip times and jitter.
  +
  +*Use the `iburst` option for faster synchronization at startup*. The `iburst`
  +option instructs `ntpd` to send an initial "burst" of time queries at startup.
  +This results in a faster synchronization of the `ntpd` with its reference
  +servers upon startup.
  +
  +*If the maximum clock error goes beyond the default threshold set by Kudu
  +(10 seconds), consider setting lower value for the `maxpoll` option for every
  +NTP server in `ntp.conf`*. For example, consider setting the `maxpoll` to 7
  +which will cause the NTP daemon to make requests to the corresponding NTP
  +server at least every 128 seconds. The default maximum poll interval is 10
  +(1024 seconds).
  +
  +[NOTE]
  +====
  +If using custom `maxpoll` interval, don't set `maxpoll` too low (e.g., lower
  +than 6) to avoid flooding NTP servers, especially the public ones. Otherwise
  +they may blacklist the client (i.e. the `ntpd` daemon at your machine) and cease
  +providing NTP service at all. If in doubt, consult the `ntp.conf` manual page.
  +====
  +
  +A few examples of `ntpd` configuration files:
  +
  +----
  +# Use my organization's internal NTP server (server in a local network).
  +server ntp1.myorg.internal iburst maxpoll 7
  +# Add servers from the NTP public pool for redundancy and robustness.
  +server 0.pool.ntp.org iburst maxpoll 8
  +server 1.pool.ntp.org iburst maxpoll 8
  +server 2.pool.ntp.org iburst maxpoll 8
  +server 3.pool.ntp.org iburst maxpoll 8
  +----
  +
  +----
  +# AWS case: use dedicated NTP server available via link-local IP address.
  +server 169.254.169.123 iburst
  +----
  +
  +----
  +# GCE case: use dedicated NTP server available from within cloud instance.
  +server metadata.google.internal iburst
  +----
  +
  +TIP: After configuring `ntpd`, first run the `ntpdate` tool with the same set
  +of NTP servers (it's assumed that `ntpd` is not running when the `ntpdate` tool
  +is run). Make sure the tool reports success: check its exit status and output.
  +In case of issues connecting to the NTP servers, make sure NTP traffic is not
  +being blocked by a firewall (NTP generates UDP traffic on port 123 by default)
  +or other network connectivity issue. Then start the `ntpd` daemon and use the
  +`ntpq` tool described above to verify that the NTP daemon is able to connect
  +to its reference servers.
   
   ==== Troubleshooting NTP Stability Problems
   
  -As of Kudu 1.6.0, Kudu daemons are able to continue to operate during a brief loss of
  -NTP synchronization. If NTP synchronization is lost for several hours, however, daemons
  -may crash. If a daemon crashes due to NTP synchronization issues, consult the `ERROR` log
  -for a dump of related information which may help to diagnose the issue.
  +As of Kudu 1.6.0, Kudu daemons are able to continue to operate during a brief
  +loss of clock synchronization. If clock synchronization is lost for several
  +hours, daemons may crash. If a daemon crashes due to clock synchronization
  +issues, consult the `ERROR` log for a dump of related information which may
  +help to diagnose the issue.
   
   TIP: Kudu 1.5.0 and earlier versions were less resilient to brief NTP outages. In
   addition, they contained a link:https://issues.apache.org/jira/browse/KUDU-2209[bug]
   which could cause Kudu to incorrectly measure the maximum error, resulting in
   crashes. If you experience crashes related to clock synchronization on these
  -earlier versions of Kudu and it appears that the system's NTP configuration is correct,
  -consider upgrading to Kudu 1.6.0 or later.
  -
  -TIP: NTP requires a network connection and may take a few minutes to synchronize the clock
  -at startup. In some cases a spotty network connection may make NTP report the clock as unsynchronized.
  -A common, though temporary, workaround for this is to restart NTP with one of the commands above.
  +earlier versions of Kudu and it appears that the system's NTP configuration
  +is correct, consider upgrading to Kudu 1.6.0 or later.
  +
  +TIP: If using other than link-local NTP server, it may take some time for `ntpd`
  +to synchronize with one of its reference servers in case of network connectivity
  +issues. In case of a spotty network between the machine and the reference NTP
  +servers, `ntpd` may become unsynchronized with its reference NTP servers. If
  +that happens, consider finding other set of reference NTP servers: the best
  +bet is to use NTP servers in the local network or *.pool.ntp.org servers.
   
   [[disk_space_usage]]
   == Disk Space Usage
  
positive.code: used by Kudu. For example, the size listed by `ls -l` does not accurately
  reflect the disk space used by Kudu data files:
  
  [source,bash]
  ----
  $ ls -lh /data/kudu/tserver/data
  total 117M
  
negative.code: [[ntp]]
  === NTP Clock Synchronization
  
  The local clock of the machine where Kudu master or tablet server is running
  must be synchronized using the Network Time Protocol (NTP) if using the `system`
  time source. The time source is controlled by the `--time_source` flag and
  by default is set to `system`.
  
  Kudu requires the *maximum clock error* (not to be mistaken with the estimated
  error) of the NTP-synchronized clock be below a configurable threshold.
  The default threshold value is 10 seconds and it can be customized using the
  `--max_clock_sync_error_usec` flag.
  
  When running with the `system` time source, Kudu will not start and will emit
  a message such as below if the local clock is reported unsynchronized:
  
  ----
  F0924 20:24:36.336809 14550 hybrid_clock.cc:191 Couldn't get the current time: Clock unsynchronized. Status: Service unavailable: Error reading clock. Clock considered unsynchronized.
  ----
  
  If the machine's clock is synchronized, but the maximum clock error is too high,
  the user will see a message such as:
  
  ----
  