anchor.diff:  [[discovery-ec2]]
   === EC2 Discovery Plugin
   
  -The EC2 discovery plugin uses the https://github.com/aws/aws-sdk-java[AWS API]
  -to identify the addresses of seed hosts.
  -
  -*If you are looking for a hosted solution of Elasticsearch on AWS, please visit http://www.elastic.co/cloud.*
  +The EC2 discovery plugin provides a list of seed addresses to the
  +{ref}/modules-discovery-hosts-providers.html[discovery process] by querying the
  +https://github.com/aws/aws-sdk-java[AWS API] for a list of EC2 instances
  +matching certain criteria determined by the <<discovery-ec2-usage,plugin
  +settings>>.
  +
  +*If you are looking for a hosted solution of {es} on AWS, please visit
  +http://www.elastic.co/cloud.*
   
   :plugin_name: discovery-ec2
   include::install_remove.asciidoc[]
   
   [[discovery-ec2-usage]]
  -==== Getting started with AWS
  -
  -The plugin adds a seed hosts provider named `ec2`. This seed hosts provider
  -finds other Elasticsearch instances in EC2 by querying the AWS metadata
  -service.  Authentication is done using
  -http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html[IAM
  -Role] credentials by default. To enable the plugin, configure {es} to use the
  -`ec2` seed hosts provider:
  +==== Using the EC2 discovery plugin
  +
  +The `discovery-ec2` plugin allows {es} to find the master-eligible nodes in a
  +cluster running on AWS EC2 by querying the
  +https://github.com/aws/aws-sdk-java[AWS API] for the addresses of the EC2
  +instances running these nodes.
  +
  +It is normally a good idea to restrict the discovery process just to the
  +master-eligible nodes in the cluster. This plugin allows you to identify these
  +nodes by certain criteria including their tags, their membership of security
  +groups, and their placement within availability zones. The discovery process
  +will work correctly even if it finds master-ineligible nodes, but master
  +elections will be more efficient if this can be avoided.
  +
  +The interaction with the AWS API can be authenticated using the
  +http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html[instance
  +role], or else custom credentials can be supplied.
  +
  +===== Enabling EC2 discovery
  +
  +To enable EC2 discovery, configure {es} to use the `ec2` seed hosts provider:
   
   [source,yaml]
   ----
   discovery.seed_providers: ec2
   ----
   
  -==== Settings
  -
  -EC2 discovery supports a number of settings.  Some settings are sensitive and
  -must be stored in the {ref}/secure-settings.html[elasticsearch keystore].  For
  -example, to use explicit AWS access keys:
  +===== Configuring EC2 discovery
  +
  +EC2 discovery supports a number of settings. Some settings are sensitive and
  +must be stored in the {ref}/secure-settings.html[{es} keystore]. For example,
  +to authenticate using a particular access key and secret key, add these keys to
  +the keystore by running the following commands:
   
   [source,sh]
   ----
  
positive.code: ----
  // NOTCONSOLE
  
  [[discovery-ec2-filtering]]
  ===== Filtering by Tags
  
  The ec2 discovery plugin can also filter machines to include in the cluster
  based on tags (and not just groups). The settings to use include the
  `discovery.ec2.tag.` prefix. For example, if you defined a tag `stage` in EC2
  and set it to `dev`, setting `discovery.ec2.tag.stage` to `dev` will only
  filter instances with a tag key set to `stage`, and a value of `dev`. Adding
  multiple `discovery.ec2.tag` settings will require all of those tags to be set
  for the instance to be included.
  
  One practical use for tag filtering is when an ec2 cluster contains many nodes
  that are not master-eligible {es} nodes. In this case, tagging the ec2
  instances that _are_ running the master-eligible {es} nodes, and then filtering
  by that tag, will help discovery to run more efficiently.
  
  [[discovery-ec2-attributes]]
  ===== Automatic Node Attributes
  
  Though not dependent on actually using `ec2` as discovery (but still requires the `discovery-ec2` plugin installed), the
  plugin can automatically add node attributes relating to ec2. In the future this may support other attributes, but this will
  currently only add an `aws_availability_zone` node attribute, which is the availability zone of the current node. Attributes
  can be used to isolate primary and replica shards across availability zones by using the
  {ref}/allocation-awareness.html[Allocation Awareness] feature.
  
  In order to enable it, set `cloud.node.auto_attributes` to `true` in the settings. For example:
  
  [source,yaml]
  ----
  cloud.node.auto_attributes: true
  
  cluster.routing.allocation.awareness.attributes: aws_availability_zone
  ----
  
  [[cloud-aws-best-practices]]
  ==== Best Practices in AWS
  
  Collection of best practices and other information around running Elasticsearch on AWS.
  
  ===== Instance/Disk
  When selecting disk please be aware of the following order of preference:
  
  * https://aws.amazon.com/efs/[EFS] - Avoid as the sacrifices made to offer durability, shared storage, and grow/shrink come at performance cost, such file systems have been known to cause corruption of indices, and due to Elasticsearch being distributed and having built-in replication, the benefits that EFS offers are not needed.
  * https://aws.amazon.com/ebs/[EBS] - Works well if running a small cluster (1-2 nodes) and cannot tolerate the loss all storage backing a node easily or if running indices with no replicas. If EBS is used, then leverage provisioned IOPS to ensure performance.
  * http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html[Instance Store] - When running clusters of larger size and with replicas the ephemeral nature of Instance Store is ideal since Elasticsearch can tolerate the loss of shards. With Instance Store one gets the performance benefit of having disk physically attached to the host running the instance and also the cost benefit of avoiding paying extra for EBS.
  
  
  Prefer https://aws.amazon.com/amazon-linux-ami/[Amazon Linux AMIs] as since Elasticsearch runs on the JVM, OS dependencies are very minimal and one can benefit from the lightweight nature, support, and performance tweaks specific to EC2 that the Amazon Linux AMIs offer.
  
  ===== Networking
  * Networking throttling takes place on smaller instance types in both the form of https://lab.getbase.com/how-we-discovered-limitations-on-the-aws-tcp-stack/[bandwidth and number of connections]. Therefore if large number of connections are needed and networking is becoming a bottleneck, avoid https://aws.amazon.com/ec2/instance-types/[instance types] with networking labeled as `Moderate` or `Low`.
  * When running in multiple http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html[availability zones] be sure to leverage {ref}/allocation-awareness.html[shard allocation awareness] so that not all copies of shard data reside in the same availability zone.
  * Do not span a cluster across regions.  If necessary, use a cross cluster search.
  
  ===== Misc
  * If you have split your nodes into roles, consider https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html[tagging the EC2 instances] by role to make it easier to filter and view your EC2 instances in the AWS console.
  * Consider https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/terminating-instances.html#Using_ChangingDisableAPITermination[enabling termination protection] for all of your instances to avoid accidentally terminating a node in the cluster and causing a potentially disruptive reallocation.
  
negative.code: bin/elasticsearch-keystore add discovery.ec2.secret_key
  ----
  
  The available settings for the EC2 discovery plugin are as follows.
  
  `discovery.ec2.access_key`::
  
      An EC2 access key. If set, you must also set `discovery.ec2.secret_key`.
      If unset, `discovery-ec2` will instead use the instance role. This setting
      is sensitive and must be stored in the {ref}/secure-settings.html[{es}
      keystore].
  
  `discovery.ec2.secret_key`::
  
      An EC2 secret key. If set, you must also set `discovery.ec2.access_key`.
      This setting is sensitive and must be stored in the
      {ref}/secure-settings.html[{es} keystore].
  
  `discovery.ec2.session_token`::
  
      An EC2 session token. If set, you must also set `discovery.ec2.access_key`
      and `discovery.ec2.secret_key`. This setting is sensitive and must be
      stored in the {ref}/secure-settings.html[{es} keystore].
  
  `discovery.ec2.endpoint`::
  
      The EC2 service endpoint to which to connect. See
      http://docs.aws.amazon.com/general/latest/gr/rande.html#ec2_region to find
      the appropriate endpoint for the region. This setting defaults to
      `ec2.us-east-1.amazonaws.com` which is appropriate for clusters running in
      the `us-east-1` region.
  
  `discovery.ec2.protocol`::
  
      The protocol to use to connect to the EC2 service endpoint, which may be
      either `http` or `https`. Defaults to `https`.
  
  `discovery.ec2.proxy.host`::
  
      The address or host name of an HTTP proxy through which to connect to EC2.
      If not set, no proxy is used.
  
  `discovery.ec2.proxy.port`::
  
      When the address of an HTTP proxy is given in `discovery.ec2.proxy.host`,
      this setting determines the port to use to connect to the proxy. Defaults to
      `80`.
  
  `discovery.ec2.proxy.username`::
  
      When the address of an HTTP proxy is given in `discovery.ec2.proxy.host`,
      this setting determines the username to use to connect to the proxy. When
      not set, no username is used. This setting is sensitive and must be stored
      in the {ref}/secure-settings.html[{es} keystore].
  
  `discovery.ec2.proxy.password`::
  
      When the address of an HTTP proxy is given in `discovery.ec2.proxy.host`,
      this setting determines the password to use to connect to the proxy. When
      not set, no password is used. This setting is sensitive and must be stored
      in the {ref}/secure-settings.html[{es} keystore].
  
  `discovery.ec2.read_timeout`::
  
      The socket timeout for connections to EC2,
      {ref}/common-options.html#time-units[including the units]. For example, a
      value of `60s` specifies a 60-second timeout. Defaults to 50 seconds.
  
  `discovery.ec2.groups`::
  
      A list of the names or IDs of the security groups to use for discovery. The
      `discovery.ec2.any_group` setting determines the behaviour of this setting.
      Defaults to an empty list, meaning that security group membership is
      ignored by EC2 discovery.
  
  `discovery.ec2.any_group`::
  
      Defaults to `true`, meaning that instances belonging to _any_ of the
      security groups specified in `discovery.ec2.groups` will be used for
      discovery. If set to `false`, only instances that belong to _all_ of the
      security groups specified in `discovery.ec2.groups` will be used for
      discovery.
  
  `discovery.ec2.host_type`::
  
  +
  --
  
  Each EC2 instance has a number of different addresses that might be suitable
  for discovery. This setting allows you to select which of these addresses is
  used by the discovery process. It can be set to one of `private_ip`,
  `public_ip`, `private_dns`, `public_dns` or `tag:TAGNAME` where `TAGNAME`
  refers to a name of a tag. This setting defaults to `private_ip`.
  
  If you set `discovery.ec2.host_type` to a value of the form `tag:TAGNAME` then
  the value of the tag `TAGNAME` attached to each instance will be used as that
  instance's address for discovery. Instances which do not have this tag set will
  be ignored by the discovery process.
  
  For example if you tag some EC2 instances with a tag named
  `elasticsearch-host-name` and set `host_type: tag:elasticsearch-host-name` then
  the `discovery-ec2` plugin will read each instance's host name from the value
  of the `elasticsearch-host-name` tag.
  http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html[Read more
  about EC2 Tags].
  
  --
  
  `discovery.ec2.availability_zones`::
  
      A list of the names of the availability zones to use for discovery. The
      name of an availability zone is the
      https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html[region
      code followed by a letter], such as `us-east-1a`. Only instances placed in
      one of the given availability zones will be used for discovery.
  
  [[discovery-ec2-filtering]]
  `discovery.ec2.tag.TAGNAME`::
  
  +
  --
  
  A list of the values of a tag called `TAGNAME` to use for discovery. If set,
  only instances that are tagged with one of the given values will be used for
  discovery. For instance, the following settings will only use nodes with a
  `role` tag set to `master` and an `environment` tag set to either `dev` or
  `staging`.
  
  [source,yaml]
  ----
  discovery.ec2.tags.role: master
  discovery.ec2.tags.environment: dev,staging
  ----
  
  NOTE: The names of tags used for discovery may only contain ASCII letters,
  numbers, hyphens and underscores. In particular you cannot use tags whose name
  includes a colon.
  
  --
  
  `discovery.ec2.node_cache_time`::
  
      Sets the length of time for which the collection of discovered instances is
      cached. {es} waits at least this long between requests for discovery
      information from the EC2 API. AWS may reject discovery requests if they are
      made too often, and this would cause discovery to fail. Defaults to `10s`.
  
  All **secure** settings of this plugin are
  {ref}/secure-settings.html#reloadable-secure-settings[reloadable], allowing you
  to update the secure settings for this plugin without needing to restart each
  node.
  
  
  [[discovery-ec2-permissions]]
  ===== Recommended EC2 permissions
  
  The `discovery-ec2` plugin works by making a `DescribeInstances` call to the AWS
  EC2 API. You must configure your AWS account to allow this, which is normally
  done using an IAM policy. You can create a custom policy via the IAM Management
  Console. It should look similar to this.
  
  [source,js]
  ----
  