anchor.diff:  bin/elasticsearch-keystore add discovery.ec2.secret_key
   ----
   
  -The following are the available discovery settings. All should be prefixed with `discovery.ec2.`.
  -Those that must be stored in the keystore are marked as `Secure`.
  -
  -`access_key`::
  -
  -    An ec2 access key. The `secret_key` setting must also be specified. (Secure)
  -
  -`secret_key`::
  -
  -    An ec2 secret key. The `access_key` setting must also be specified. (Secure)
  -
  -`session_token`::
  -    An ec2 session token. The `access_key` and `secret_key` settings must also
  -    be specified. (Secure)
  -
  -`endpoint`::
  -
  -    The ec2 service endpoint to connect to. See
  -    http://docs.aws.amazon.com/general/latest/gr/rande.html#ec2_region. This
  -    defaults to `ec2.us-east-1.amazonaws.com`.
  -
  -`protocol`::
  -
  -    The protocol to use to connect to ec2. Valid values are either `http`
  -    or `https`. Defaults to `https`. 
  -
  -`proxy.host`::
  -
  -    The host name of a proxy to connect to ec2 through.
  -
  -`proxy.port`::
  -
  -    The port of a proxy to connect to ec2 through.
  -
  -`proxy.username`::
  -
  -    The username to connect to the `proxy.host` with. (Secure)    
  -
  -`proxy.password`::
  -
  -    The password to connect to the `proxy.host` with. (Secure)    
  -
  -`read_timeout`::
  -
  -    The socket timeout for connecting to ec2. The value should specify the unit. For example,
  -    a value of `5s` specifies a 5 second timeout. The default value is 50 seconds.
  -
  -`groups`::
  -
  -    Either a comma separated list or array based list of (security) groups.
  -    Only instances with the provided security groups will be used in the
  -    cluster discovery. (NOTE: You could provide either group NAME or group
  -    ID.)
  -
  -`host_type`::
  +The available settings for the EC2 discovery plugin are as follows.
  +
  +`discovery.ec2.access_key`::
  +
  +    An EC2 access key. If set, you must also set `discovery.ec2.secret_key`.
  +    If unset, `discovery-ec2` will instead use the instance role. This setting
  +    is sensitive and must be stored in the {ref}/secure-settings.html[{es}
  +    keystore].
  +
  +`discovery.ec2.secret_key`::
  +
  +    An EC2 secret key. If set, you must also set `discovery.ec2.access_key`.
  +    This setting is sensitive and must be stored in the
  +    {ref}/secure-settings.html[{es} keystore].
  +
  +`discovery.ec2.session_token`::
  +
  +    An EC2 session token. If set, you must also set `discovery.ec2.access_key`
  +    and `discovery.ec2.secret_key`. This setting is sensitive and must be
  +    stored in the {ref}/secure-settings.html[{es} keystore].
  +
  +`discovery.ec2.endpoint`::
  +
  +    The EC2 service endpoint to which to connect. See
  +    http://docs.aws.amazon.com/general/latest/gr/rande.html#ec2_region to find
  +    the appropriate endpoint for the region. This setting defaults to
  +    `ec2.us-east-1.amazonaws.com` which is appropriate for clusters running in
  +    the `us-east-1` region.
  +
  +`discovery.ec2.protocol`::
  +
  +    The protocol to use to connect to the EC2 service endpoint, which may be
  +    either `http` or `https`. Defaults to `https`.
  +
  +`discovery.ec2.proxy.host`::
  +
  +    The address or host name of an HTTP proxy through which to connect to EC2.
  +    If not set, no proxy is used.
  +
  +`discovery.ec2.proxy.port`::
  +
  +    When the address of an HTTP proxy is given in `discovery.ec2.proxy.host`,
  +    this setting determines the port to use to connect to the proxy. Defaults to
  +    `80`.
  +
  +`discovery.ec2.proxy.username`::
  +
  +    When the address of an HTTP proxy is given in `discovery.ec2.proxy.host`,
  +    this setting determines the username to use to connect to the proxy. When
  +    not set, no username is used. This setting is sensitive and must be stored
  +    in the {ref}/secure-settings.html[{es} keystore].
  +
  +`discovery.ec2.proxy.password`::
  +
  +    When the address of an HTTP proxy is given in `discovery.ec2.proxy.host`,
  +    this setting determines the password to use to connect to the proxy. When
  +    not set, no password is used. This setting is sensitive and must be stored
  +    in the {ref}/secure-settings.html[{es} keystore].
  +
  +`discovery.ec2.read_timeout`::
  +
  +    The socket timeout for connections to EC2,
  +    {ref}/common-options.html#time-units[including the units]. For example, a
  +    value of `60s` specifies a 60-second timeout. Defaults to 50 seconds.
  +
  +`discovery.ec2.groups`::
  +
  +    A list of the names or IDs of the security groups to use for discovery. The
  +    `discovery.ec2.any_group` setting determines the behaviour of this setting.
  +    Defaults to an empty list, meaning that security group membership is
  +    ignored by EC2 discovery.
  +
  +`discovery.ec2.any_group`::
  +
  +    Defaults to `true`, meaning that instances belonging to _any_ of the
  +    security groups specified in `discovery.ec2.groups` will be used for
  +    discovery. If set to `false`, only instances that belong to _all_ of the
  +    security groups specified in `discovery.ec2.groups` will be used for
  +    discovery.
  +
  +`discovery.ec2.host_type`::
   
   +
   --
  -The type of host type to use to communicate with other instances. Can be
  -one of `private_ip`, `public_ip`, `private_dns`, `public_dns` or `tag:TAGNAME` where
  -`TAGNAME` refers to a name of a tag configured for all EC2 instances. Instances which don't
  -have this tag set will be ignored by the discovery process.
  -
  -For example if you defined a tag `my-elasticsearch-host` in ec2 and set it to `myhostname1.mydomain.com`, then
  -setting `host_type: tag:my-elasticsearch-host` will tell Discovery Ec2 plugin to read the host name from the
  -`my-elasticsearch-host` tag. In this case, it will be resolved to `myhostname1.mydomain.com`.
  -http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html[Read more about EC2 Tags].
  -
  -Defaults to `private_ip`.
  +
  +Each EC2 instance has a number of different addresses that might be suitable
  +for discovery. This setting allows you to select which of these addresses is
  +used by the discovery process. It can be set to one of `private_ip`,
  +`public_ip`, `private_dns`, `public_dns` or `tag:TAGNAME` where `TAGNAME`
  +refers to a name of a tag. This setting defaults to `private_ip`.
  +
  +If you set `discovery.ec2.host_type` to a value of the form `tag:TAGNAME` then
  +the value of the tag `TAGNAME` attached to each instance will be used as that
  +instance's address for discovery. Instances which do not have this tag set will
  +be ignored by the discovery process.
  +
  +For example if you tag some EC2 instances with a tag named
  +`elasticsearch-host-name` and set `host_type: tag:elasticsearch-host-name` then
  +the `discovery-ec2` plugin will read each instance's host name from the value
  +of the `elasticsearch-host-name` tag.
  +http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html[Read more
  +about EC2 Tags].
  +
   --
   
  -`availability_zones`::
  -
  -    Either a comma separated list or array based list of availability zones.
  -    Only instances within the provided availability zones will be used in the
  -    cluster discovery.
  -
  -`any_group`::
  -
  -    If set to `false`, will require all security groups to be present for the
  -    instance to be used for the discovery. Defaults to `true`.
  -
  -`node_cache_time`::
  -
  -    How long the list of hosts is cached to prevent further requests to the AWS API.
  -    Defaults to `10s`.
  -
  -*All* secure settings of this plugin are {ref}/secure-settings.html#reloadable-secure-settings[reloadable].
  -After you reload the settings, an aws sdk client with the latest settings
  -from the keystore will be used.
  -
  -[IMPORTANT]
  -.Binding the network host
  -==============================================
  -
  -It's important to define `network.host` as by default it's bound to `localhost`.
  -
  -You can use {ref}/modules-network.html[core network host settings] or
  -<<discovery-ec2-network-host,ec2 specific host settings>>:
  -
  -==============================================
  -
  -[[discovery-ec2-network-host]]
  -===== EC2 Network Host
  -
  -When the `discovery-ec2` plugin is installed, the following are also allowed
  -as valid network host settings:
  -
  -[cols="<,<",options="header",]
  -|==================================================================
  -|EC2 Host Value |Description
  -|`_ec2:privateIpv4_` |The private IP address (ipv4) of the machine.
  -|`_ec2:privateDns_` |The private host of the machine.
  -|`_ec2:publicIpv4_` |The public IP address (ipv4) of the machine.
  -|`_ec2:publicDns_` |The public host of the machine.
  -|`_ec2:privateIp_` |equivalent to `_ec2:privateIpv4_`.
  -|`_ec2:publicIp_` |equivalent to `_ec2:publicIpv4_`.
  -|`_ec2_` |equivalent to `_ec2:privateIpv4_`.
  -|==================================================================
  +`discovery.ec2.availability_zones`::
  +
  +    A list of the names of the availability zones to use for discovery. The
  +    name of an availability zone is the
  +    https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html[region
  +    code followed by a letter], such as `us-east-1a`. Only instances placed in
  +    one of the given availability zones will be used for discovery.
  +
  +[[discovery-ec2-filtering]]
  +`discovery.ec2.tag.TAGNAME`::
  +
  ++
  +--
  +
  +A list of the values of a tag called `TAGNAME` to use for discovery. If set,
  +only instances that are tagged with one of the given values will be used for
  +discovery. For instance, the following settings will only use nodes with a
  +`role` tag set to `master` and an `environment` tag set to either `dev` or
  +`staging`.
  +
  +[source,yaml]
  +----
  +discovery.ec2.tags.role: master
  +discovery.ec2.tags.environment: dev,staging
  +----
  +
  +NOTE: The names of tags used for discovery may only contain ASCII letters,
  +numbers, hyphens and underscores. In particular you cannot use tags whose name
  +includes a colon.
  +
  +--
  +
  +`discovery.ec2.node_cache_time`::
  +
  +    Sets the length of time for which the collection of discovered instances is
  +    cached. {es} waits at least this long between requests for discovery
  +    information from the EC2 API. AWS may reject discovery requests if they are
  +    made too often, and this would cause discovery to fail. Defaults to `10s`.
  +
  +All **secure** settings of this plugin are
  +{ref}/secure-settings.html#reloadable-secure-settings[reloadable], allowing you
  +to update the secure settings for this plugin without needing to restart each
  +node.
  +
   
   [[discovery-ec2-permissions]]
  -===== Recommended EC2 Permissions
  -
  -EC2 discovery requires making a call to the EC2 service. You'll want to setup
  -an IAM policy to allow this. You can create a custom policy via the IAM
  -Management Console. It should look similar to this.
  +===== Recommended EC2 permissions
  +
  +The `discovery-ec2` plugin works by making a `DescribeInstances` call to the AWS
  +EC2 API. You must configure your AWS account to allow this, which is normally
  +done using an IAM policy. You can create a custom policy via the IAM Management
  +Console. It should look similar to this.
   
   [source,js]
   ----
  
positive.code: [[discovery-ec2]]
  === EC2 Discovery Plugin
  
  The EC2 discovery plugin uses the https://github.com/aws/aws-sdk-java[AWS API]
  to identify the addresses of seed hosts.
  
  *If you are looking for a hosted solution of Elasticsearch on AWS, please visit http://www.elastic.co/cloud.*
  
  :plugin_name: discovery-ec2
  include::install_remove.asciidoc[]
  
  [[discovery-ec2-usage]]
  ==== Getting started with AWS
  
  The plugin adds a seed hosts provider named `ec2`. This seed hosts provider
  finds other Elasticsearch instances in EC2 by querying the AWS metadata
  service.  Authentication is done using
  http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html[IAM
  Role] credentials by default. To enable the plugin, configure {es} to use the
  `ec2` seed hosts provider:
  
  [source,yaml]
  ----
  discovery.seed_providers: ec2
  ----
  
  ==== Settings
  
  EC2 discovery supports a number of settings.  Some settings are sensitive and
  must be stored in the {ref}/secure-settings.html[elasticsearch keystore].  For
  example, to use explicit AWS access keys:
  
  [source,sh]
  ----
  
negative.code: ----
  // NOTCONSOLE
  
  [[discovery-ec2-attributes]]
  ===== Automatic node attributes
  
  The `discovery-ec2` plugin can automatically set the `aws_availability_zone`
  node attribute to the availability zone of each node. This node attribute
  allows you to ensure that each shard has copies allocated redundantly across
  multiple availability zones by using the
  {ref}/allocation-awareness.html[Allocation Awareness] feature.
  
  In order to enable the automatic definition of the `aws_availability_zone`
  attribute, set `cloud.node.auto_attributes` to `true`. For example:
  
  [source,yaml]
  ----
  cloud.node.auto_attributes: true
  cluster.routing.allocation.awareness.attributes: aws_availability_zone
  ----
  
  The `aws_availability_zone` attribute can be automatically set like this when
  using any discovery type. It is not necessary to set `discovery.seed_providers:
  ec2`. However this feature does require that the `discovery-ec2` plugin is
  installed.
  
  [[discovery-ec2-network-host]]
  ===== Binding to the correct address
  
  It is important to define `network.host` correctly when deploying a cluster on
  EC2. By default each {es} node only binds to `localhost`, which will prevent it
  from being discovered by nodes running on any other instances.
  
  You can use the {ref}/modules-network.html[core network host settings] to bind
  each node to the desired address, or you can set `network.host` to one of the
  following EC2-specific settings provided by the `discovery-ec2` plugin:
  
  [cols="<,<",options="header",]
  |==================================================================
  |EC2 Host Value      |Description
  |`_ec2:privateIpv4_` |The private IP address (ipv4) of the machine.
  |`_ec2:privateDns_`  |The private host of the machine.
  |`_ec2:publicIpv4_`  |The public IP address (ipv4) of the machine.
  |`_ec2:publicDns_`   |The public host of the machine.
  |`_ec2:privateIp_`   |Equivalent to `_ec2:privateIpv4_`.
  |`_ec2:publicIp_`    |Equivalent to `_ec2:publicIpv4_`.
  |`_ec2_`             |Equivalent to `_ec2:privateIpv4_`.
  |==================================================================
  
  These values are acceptable when using any discovery type. They do not require
  you to set `discovery.seed_providers: ec2`. However they do require that the
  `discovery-ec2` plugin is installed.
  
  [[cloud-aws-best-practices]]
  ==== Best Practices in AWS
  
  This section contains some other information about designing and managing an
  {es} cluster on your own AWS infrastructure. If you would prefer to avoid these
  operational details then you may be interested in a hosted {es} installation
  available on AWS-based infrastructure from http://www.elastic.co/cloud.
  
  ===== Storage
  
  EC2 instances offer a number of different kinds of storage. Please be aware of
  the folowing when selecting the storage for your cluster:
  
  * http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html[Instance
  Store] is recommended for {es} clusters as it offers excellent performance and
  is cheaper than EBS-based storage. {es} is designed to work well with this kind
  of ephemeral storage because it replicates each shard across multiple nodes. If
  a node fails and its Instance Store is lost then {es} will rebuild any lost
  shards from other copies.
  
  * https://aws.amazon.com/ebs/[EBS-based storage] may be acceptable
  for smaller clusters (1-2 nodes). Be sure to use provisioned IOPS to ensure
  your cluster has satisfactory performance.
  
  * https://aws.amazon.com/efs/[EFS-based storage] is not
  recommended or supported as it does not offer satisfactory performance.
  Historically, shared network filesystems such as EFS have not always offered
  precisely the behaviour that {es} requires of its filesystem, and this has been
  known to lead to index corruption. Although EFS offers durability, shared
  storage, and the ability to grow and shrink filesystems dynamically, you can
  achieve the same benefits using {es} directly.
  
  ===== Choice of AMI
  
  Prefer the https://aws.amazon.com/amazon-linux-ami/[Amazon Linux AMIs] as these
  allow you to benefit from the lightweight nature, support, and EC2-specific
  performance enhancements that these images offer.
  
  ===== Networking
  
  * Smaller instance types have limited network performance, in terms of both
  https://lab.getbase.com/how-we-discovered-limitations-on-the-aws-tcp-stack/[bandwidth
  and number of connections]. If networking is a bottleneck, avoid
  https://aws.amazon.com/ec2/instance-types/[instance types] with networking
  labelled as `Moderate` or `Low`.
  
  * It is a good idea to distribute your nodes across multiple
  http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html[availability
  zones] and use {ref}/allocation-awareness.html[shard allocation awareness] to
  ensure that each shard has copies in more than one availability zone.
  
  * Do not span a cluster across regions. {es} expects that node-to-node
  connections within a cluster are reasonably reliable and offer high bandwidth
  and low latency, and these properties do not hold for connections between
  regions. Although an {es} cluster will behave correctly when node-to-node
  connections are unreliable or slow, it is not optimised for this case and its
  performance may suffer. If you wish to geographically distribute your data, you
  should provision multiple clusters and use features such as
  {ref}/modules-cross-cluster-search.html[cross-cluster search] and
  {stack-ov}/xpack-ccr.html[cross-cluster replication].
  
  ===== Other recommendations
  
  * If you have split your nodes into roles, consider
  https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html[tagging the
  EC2 instances] by role to make it easier to filter and view your EC2 instances
  in the AWS console.
  
  * Consider
  https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/terminating-instances.html#Using_ChangingDisableAPITermination[enabling
  termination protection] for all of your data and master-eligible nodes. This
  will help to prevent accidental termination of these nodes which could
  temporarily reduce the resilience of the cluster and which could cause a
  potentially disruptive reallocation of shards.
  
  * If running your cluster using one or more
  https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html[auto-scaling
  groups], consider protecting your data and master-eligible nodes
  https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-instance-termination.html#instance-protection-instance[against
  termination during scale-in]. This will help to prevent automatic termination
  of these nodes which could temporarily reduce the resilience of the cluster and
  which could cause a potentially disruptive reallocation of shards. If these
  instances are protected against termination during scale-in then you can use
  {ref}/shard-allocation-filtering.html[shard allocation filtering] to gracefully
  migrate any data off these nodes before terminating them manually.
  