Chapter 31. Elasticsearch
Chapter 31. Elasticsearch Since Camel 3.18.3 Only producer is supported The ElasticSearch component allows you to interface with an ElasticSearch 8.x API using the Java API Client library. 31.1. Dependencies When using elasticsearch with Red Hat build of Camel Spring Boot make sure to use the following Maven dependency to have support for auto configuration: <dependency> <groupId>org.apache.camel.springboot</groupId> <artifactId>camel-elasticsearch-starter</artifactId> </dependency> 31.2. URI format 31.3. Configuring Options Camel components are configured on two levels: Component level Endpoint level 31.3.1. Component Level Options The component level is the highest level. The configurations you define at this level are inherited by all the endpoints. For example, a component can have security settings, credentials for authentication, urls for network connection, and so on. Since components typically have pre-configured defaults for the most common cases, you may need to only configure a few component options, or maybe none at all. You can configure components with Component DSL in a configuration file (application.properties|yaml), or directly with Java code. 31.3.2. Endpoint Level Options At the Endpoint level you have many options, which you can use to configure what you want the endpoint to do. The options are categorized according to whether the endpoint is used as a consumer (from) or as a producer (to) or used for both. You can configure endpoints directly in the endpoint URI as path and query parameters. You can also use Endpoint DSL and DataFormat DSL as type safe ways of configuring endpoints and data formats in Java. When configuring options, use Property Placeholders for urls, port numbers, sensitive information, and other settings. Placeholders allows you to externalize the configuration from your code, giving you more flexible and reusable code. 31.4. Component Options The Elasticsearch component supports 14 options, which are listed below. Name Description Default Type connectionTimeout (producer) The time in ms to wait before connection will timeout. 30000 int hostAddresses (producer) Comma separated list with ip:port formatted remote transport addresses to use. The ip and port options must be left blank for hostAddresses to be considered instead. String lazyStartProducer (producer) Whether the producer should be started lazy (on the first message). By starting lazy you can use this to allow CamelContext and routes to startup in situations where a producer may otherwise fail during starting and cause the route to fail being started. By deferring this startup to be lazy then the startup failure can be handled during routing messages via Camel's routing error handlers. Beware that when the first message is processed then creating and starting the producer may take a little time and prolong the total processing time of the processing. false boolean maxRetryTimeout (producer) The time in ms before retry. 30000 int socketTimeout (producer) The timeout in ms to wait before the socket will timeout. 30000 int autowiredEnabled (advanced) Whether autowiring is enabled. This is used for automatic autowiring options (the option must be marked as autowired) by looking up in the registry to find if there is a single instance of matching type, which then gets configured on the component. This can be used for automatic configuring JDBC data sources, JMS connection factories, AWS Clients, etc. true boolean client (advanced) Autowired To use an existing configured Elasticsearch client, instead of creating a client per endpoint. This allow to customize the client with specific settings. RestClient enableSniffer (advanced) Enable automatically discover nodes from a running Elasticsearch cluster. If this option is used in conjunction with Spring Boot then it's managed by the Spring Boot configuration (see: Disable Sniffer in Spring Boot). false boolean sniffAfterFailureDelay (advanced) The delay of a sniff execution scheduled after a failure (in milliseconds). 60000 int snifferInterval (advanced) The interval between consecutive ordinary sniff executions in milliseconds. Will be honoured when sniffOnFailure is disabled or when there are no failures between consecutive sniff executions. 300000 int certificatePath (security) The path of the self-signed certificate to use to access to Elasticsearch. String enableSSL (security) Enable SSL. false boolean password (security) Password for authenticate. String user (security) Basic authenticate user. String 31.5. Endpoint Options The Elasticsearch endpoint is configured using URI syntax: with the following path and query parameters: 31.5.1. Path Parameters (1 parameters) Name Description Default Type clusterName (producer) Required Name of the cluster. String 31.5.2. Query Parameters (19 parameters) Name Description Default Type connectionTimeout (producer) The time in ms to wait before connection will timeout. 30000 int disconnect (producer) Disconnect after it finish calling the producer. false boolean from (producer) Starting index of the response. Integer hostAddresses (producer) Comma separated list with ip:port formatted remote transport addresses to use. String indexName (producer) The name of the index to act against. String maxRetryTimeout (producer) The time in ms before retry. 30000 int operation (producer) What operation to perform. Enum values: Index Update Bulk GetById MultiGet MultiSearch Delete DeleteIndex Search Exists Ping ElasticsearchOperation scrollKeepAliveMs (producer) Time in ms during which elasticsearch will keep search context alive. 60000 int size (producer) Size of the response. Integer socketTimeout (producer) The timeout in ms to wait before the socket will timeout. 30000 int useScroll (producer) Enable scroll usage. false boolean waitForActiveShards (producer) Index creation waits for the write consistency number of shards to be available. 1 int lazyStartProducer (producer (advanced)) Whether the producer should be started lazy (on the first message). By starting lazy you can use this to allow CamelContext and routes to startup in situations where a producer may otherwise fail during starting and cause the route to fail being started. By deferring this startup to be lazy then the startup failure can be handled during routing messages via Camel's routing error handlers. Beware that when the first message is processed then creating and starting the producer may take a little time and prolong the total processing time of the processing. false boolean documentClass (advanced) The class to use when deserializing the documents. ObjectNode Class enableSniffer (advanced) Enable automatically discover nodes from a running Elasticsearch cluster. If this option is used in conjunction with Spring Boot then it's managed by the Spring Boot configuration (see: Disable Sniffer in Spring Boot). false boolean sniffAfterFailureDelay (advanced) The delay of a sniff execution scheduled after a failure (in milliseconds). 60000 int snifferInterval (advanced) The interval between consecutive ordinary sniff executions in milliseconds. Will be honoured when sniffOnFailure is disabled or when there are no failures between consecutive sniff executions. 300000 int certificatePath (security) The path of the self-signed certificate to use to access to Elasticsearch. String enableSSL (security) Enable SSL. false boolean 31.6. Message Headers The Elasticsearch component supports 9 message header(s), which is/are listed below: Name Description Default Type operation (producer) Constant: PARAM_OPERATION The operation to perform. Enum values: Index Update Bulk GetById MultiGet MultiSearch Delete DeleteIndex Search Exists Ping ElasticsearchOperation indexId (producer) Constant: PARAM_INDEX_ID The id of the indexed document. String indexName (producer) Constant: PARAM_INDEX_NAME The name of the index to act against. String documentClass (producer) Constant: PARAM_DOCUMENT_CLASS The full qualified name of the class of the document to unmarshall. ObjectNode Class waitForActiveShards (producer) Constant: PARAM_WAIT_FOR_ACTIVE_SHARDS The index creation waits for the write consistency number of shards to be available. Integer scrollKeepAliveMs (producer) Constant: PARAM_SCROLL_KEEP_ALIVE_MS The starting index of the response. Integer useScroll (producer) Constant: PARAM_SCROLL Set to true to enable scroll usage. Boolean size (producer) Constant: PARAM_SIZE The size of the response. Integer from (producer) Constant: PARAM_FROM The starting index of the response. Integer 31.7. Message Operations The following ElasticSearch operations are currently supported. Simply set an endpoint URI option or exchange header with a key of "operation" and a value set to one of the following. Some operations also require other parameters or the message body to be set. operation message body description Index Map , String , byte[] , Reader , InputStream or IndexRequest.Builder content to index Adds content to an index and returns the content's indexId in the body. You can set the name of the target index by setting the message header with the key "indexName". You can set the indexId by setting the message header with the key "indexId". GetById String or GetRequest.Builder index id of content to retrieve Retrieves the document corresponding to the given index id and returns a GetResponse object in the body. You can set the name of the target index by setting the message header with the key "indexName". You can set the type of document by setting the message header with the key "documentClass". Delete String or DeleteRequest.Builder index id of content to delete Deletes the specified indexName and returns a Result object in the body. You can set the name of the target index by setting the message header with the key "indexName". DeleteIndex String or DeleteIndexRequest.Builder index name of the index to delete Deletes the specified indexName and returns a status code in the body. You can set the name of the target index by setting the message header with the key "indexName". Bulk Iterable or BulkRequest.Builder of any type that is already accepted (DeleteOperation.Builder for delete operation, UpdateOperation.Builder for update operation, CreateOperation.Builder for create operation, byte[], InputStream, String, Reader, Map or any document type for index operation) Adds/Updates/Deletes content from/to an index and returns a List<BulkResponseItem> object in the body You can set the name of the target index by setting the message header with the key "indexName". Search Map , String or SearchRequest.Builder Search the content with the map of query string. You can set the name of the target index by setting the message header with the key "indexName". You can set the number of hits to return by setting the message header with the key "size". You can set the starting document offset by setting the message header with the key "from". MultiSearch MsearchRequest.Builder Multiple search in one MultiGet Iterable<String> or MgetRequest.Builder the id of the document to retrieve Multiple get in one You can set the name of the target index by setting the message header with the key "indexName". Exists None Checks whether the index exists or not and returns a Boolean flag in the body. You must set the name of the target index by setting the message header with the key "indexName". Update byte[] , InputStream , String , Reader , Map or any document type content to update Updates content to an index and returns the content's indexId in the body. You can set the name of the target index by setting the message header with the key "indexName". You can set the indexId by setting the message header with the key "indexId". Ping None Pings the Elasticsearch cluster and returns true if the ping succeeded, false otherwise 31.8. Configure the component and enable basic authentication To use the Elasticsearch component it has to be configured with a minimum configuration. ElasticsearchComponent elasticsearchComponent = new ElasticsearchComponent(); elasticsearchComponent.setHostAddresses("myelkhost:9200"); camelContext.addComponent("elasticsearch", elasticsearchComponent); For basic authentication with elasticsearch or using reverse http proxy in front of the elasticsearch cluster, simply setup basic authentication and SSL on the component like the example below ElasticsearchComponent elasticsearchComponent = new ElasticsearchComponent(); elasticsearchComponent.setHostAddresses("myelkhost:9200"); elasticsearchComponent.setUser("elkuser"); elasticsearchComponent.setPassword("secure!!"); elasticsearchComponent.setEnableSSL(true); elasticsearchComponent.setCertificatePath(certPath); camelContext.addComponent("elasticsearch", elasticsearchComponent); 31.9. Index Example Below is a simple INDEX example from("direct:index") .to("elasticsearch://elasticsearch?operation=Index&indexName=twitter"); <route> <from uri="direct:index"/> <to uri="elasticsearch://elasticsearch?operation=Index&amp;indexName=twitter"/> </route> For this operation you need to specify a indexId header. A client would simply need to pass a body message containing a Map to the route. The result body contains the indexId created. Map<String, String> map = new HashMap<String, String>(); map.put("content", "test"); String indexId = template.requestBody("direct:index", map, String.class); 31.10. Search Example Searching on specific field(s) and value use the Operation \u0301Search \u0301. Pass in the query JSON String or the Map from("direct:search") .to("elasticsearch://elasticsearch?operation=Search&indexName=twitter"); <route> <from uri="direct:search"/> <to uri="elasticsearch://elasticsearch?operation=Search&amp;indexName=twitter"/> </route> String query = "{\"query\":{\"match\":{\"doc.content\":\"new release of ApacheCamel\"}}}"; HitsMetadata<?> response = template.requestBody("direct:search", query, HitsMetadata.class); Search on specific field(s) using Map. Map<String, Object> actualQuery = new HashMap<>(); actualQuery.put("doc.content", "new release of ApacheCamel"); Map<String, Object> match = new HashMap<>(); match.put("match", actualQuery); Map<String, Object> query = new HashMap<>(); query.put("query", match); HitsMetadata<?> response = template.requestBody("direct:search", query, HitsMetadata.class); Search using Elasticsearch scroll api in order to fetch all results. from("direct:search") .to("elasticsearch://elasticsearch?operation=Search&indexName=twitter&useScroll=true&scrollKeepAliveMs=30000"); <route> <from uri="direct:search"/> <to uri="elasticsearch://elasticsearch?operation=Search&amp;indexName=twitter&amp;useScroll=true&amp;scrollKeepAliveMs=30000"/> </route> String query = "{\"query\":{\"match\":{\"doc.content\":\"new release of ApacheCamel\"}}}"; try (ElasticsearchScrollRequestIterator response = template.requestBody("direct:search", query, ElasticsearchScrollRequestIterator.class)) { // do something smart with results } can also be used. from("direct:search") .to("elasticsearch://elasticsearch?operation=Search&indexName=twitter&useScroll=true&scrollKeepAliveMs=30000") .split() .body() .streaming() .to("mock:output") .end(); 31.11. MultiSearch Example MultiSearching on specific field(s) and value use the Operation \u0301MultiSearch \u0301. Pass in the MultiSearchRequest instance from("direct:multiSearch") .to("elasticsearch://elasticsearch?operation=MultiSearch"); <route> <from uri="direct:multiSearch"/> <to uri="elasticsearch://elasticsearch?operation=MultiSearch"/> </route> MultiSearch on specific field(s) MsearchRequest.Builder builder = new MsearchRequest.Builder().index("twitter").searches( new RequestItem.Builder().header(new MultisearchHeader.Builder().build()) .body(new MultisearchBody.Builder().query(b -> b.matchAll(x -> x)).build()).build(), new RequestItem.Builder().header(new MultisearchHeader.Builder().build()) .body(new MultisearchBody.Builder().query(b -> b.matchAll(x -> x)).build()).build()); List<MultiSearchResponseItem<?>> response = template.requestBody("direct:multiSearch", builder, List.class); 31.12. Document type For all the search operations, it is possible to indicate the type of document to retrieve in order to get the result already unmarshalled with the expected type. The document type can be set using the header "documentClass" or via the uri parameter of the same name. 31.13. Using Camel Elasticsearch with Spring Boot When you use camel-elasticsearch-starter with Spring Boot v2, then you must declare the following dependency in your own pom.xml . <dependency> <groupId>jakarta.json</groupId> <artifactId>jakarta.json-api</artifactId> <version>2.0.2</version> </dependency> This is needed because Spring Boot v2 provides jakarta.json-api:1.1.6, and Elasticsearch requires to use json-api v2. 31.13.1. Use RestClient provided by Spring Boot By default Spring Boot will auto configure an Elasticsearch RestClient that will be used by camel, it is possible to customize the client with the following basic properties: spring.elasticsearch.uris=myelkhost:9200 spring.elasticsearch.username=elkuser spring.elasticsearch.password=secure!! More information can be found in application-properties.data.spring.elasticsearch.connection-timeout . 31.13.2. Disable Sniffer when using Spring Boot When Spring Boot is on the classpath the Sniffer client for Elasticsearch is enabled by default. This option can be disabled in the Spring Boot Configuration: spring: autoconfigure: exclude: org.springframework.boot.autoconfigure.elasticsearch.ElasticsearchRestClientAutoConfiguration 31.14. Spring Boot Auto-Configuration The component supports 15 options, which are listed below. Name Description Default Type camel.component.elasticsearch.autowired-enabled Whether autowiring is enabled. This is used for automatic autowiring options (the option must be marked as autowired) by looking up in the registry to find if there is a single instance of matching type, which then gets configured on the component. This can be used for automatic configuring JDBC data sources, JMS connection factories, AWS Clients, etc. true Boolean camel.component.elasticsearch.certificate-path The path of the self-signed certificate to use to access to Elasticsearch. String camel.component.elasticsearch.client To use an existing configured Elasticsearch client, instead of creating a client per endpoint. This allow to customize the client with specific settings. The option is a org.elasticsearch.client.RestClient type. RestClient camel.component.elasticsearch.connection-timeout The time in ms to wait before connection will timeout. 30000 Integer camel.component.elasticsearch.enable-s-s-l Enable SSL. false Boolean camel.component.elasticsearch.enable-sniffer Enable automatically discover nodes from a running Elasticsearch cluster. If this option is used in conjunction with Spring Boot then it's managed by the Spring Boot configuration (see: Disable Sniffer in Spring Boot). false Boolean camel.component.elasticsearch.enabled Whether to enable auto configuration of the elasticsearch component. This is enabled by default. Boolean camel.component.elasticsearch.host-addresses Comma separated list with ip:port formatted remote transport addresses to use. The ip and port options must be left blank for hostAddresses to be considered instead. String camel.component.elasticsearch.lazy-start-producer Whether the producer should be started lazy (on the first message). By starting lazy you can use this to allow CamelContext and routes to startup in situations where a producer may otherwise fail during starting and cause the route to fail being started. By deferring this startup to be lazy then the startup failure can be handled during routing messages via Camel's routing error handlers. Beware that when the first message is processed then creating and starting the producer may take a little time and prolong the total processing time of the processing. false Boolean camel.component.elasticsearch.max-retry-timeout The time in ms before retry. 30000 Integer camel.component.elasticsearch.password Password for authenticate. String camel.component.elasticsearch.sniff-after-failure-delay The delay of a sniff execution scheduled after a failure (in milliseconds). 60000 Integer camel.component.elasticsearch.sniffer-interval The interval between consecutive ordinary sniff executions in milliseconds. Will be honoured when sniffOnFailure is disabled or when there are no failures between consecutive sniff executions. 300000 Integer camel.component.elasticsearch.socket-timeout The timeout in ms to wait before the socket will timeout. 30000 Integer camel.component.elasticsearch.user Basic authenticate user. String
<dependency> <groupId>org.apache.camel.springboot</groupId> <artifactId>camel-elasticsearch-starter</artifactId> </dependency> elasticsearch://clusterName[?options] elasticsearch:clusterName ElasticsearchComponent elasticsearchComponent = new ElasticsearchComponent(); elasticsearchComponent.setHostAddresses("myelkhost:9200"); camelContext.addComponent("elasticsearch", elasticsearchComponent); ElasticsearchComponent elasticsearchComponent = new ElasticsearchComponent(); elasticsearchComponent.setHostAddresses("myelkhost:9200"); elasticsearchComponent.setUser("elkuser"); elasticsearchComponent.setPassword("secure!!"); elasticsearchComponent.setEnableSSL(true); elasticsearchComponent.setCertificatePath(certPath); camelContext.addComponent("elasticsearch", elasticsearchComponent); from("direct:index") .to("elasticsearch://elasticsearch?operation=Index&indexName=twitter"); <route> <from uri="direct:index"/> <to uri="elasticsearch://elasticsearch?operation=Index&amp;indexName=twitter"/> </route> Map<String, String> map = new HashMap<String, String>(); map.put("content", "test"); String indexId = template.requestBody("direct:index", map, String.class); from("direct:search") .to("elasticsearch://elasticsearch?operation=Search&indexName=twitter"); <route> <from uri="direct:search"/> <to uri="elasticsearch://elasticsearch?operation=Search&amp;indexName=twitter"/> </route> String query = "{\"query\":{\"match\":{\"doc.content\":\"new release of ApacheCamel\"}}}"; HitsMetadata<?> response = template.requestBody("direct:search", query, HitsMetadata.class); Map<String, Object> actualQuery = new HashMap<>(); actualQuery.put("doc.content", "new release of ApacheCamel"); Map<String, Object> match = new HashMap<>(); match.put("match", actualQuery); Map<String, Object> query = new HashMap<>(); query.put("query", match); HitsMetadata<?> response = template.requestBody("direct:search", query, HitsMetadata.class); from("direct:search") .to("elasticsearch://elasticsearch?operation=Search&indexName=twitter&useScroll=true&scrollKeepAliveMs=30000"); <route> <from uri="direct:search"/> <to uri="elasticsearch://elasticsearch?operation=Search&amp;indexName=twitter&amp;useScroll=true&amp;scrollKeepAliveMs=30000"/> </route> String query = "{\"query\":{\"match\":{\"doc.content\":\"new release of ApacheCamel\"}}}"; try (ElasticsearchScrollRequestIterator response = template.requestBody("direct:search", query, ElasticsearchScrollRequestIterator.class)) { // do something smart with results } from("direct:search") .to("elasticsearch://elasticsearch?operation=Search&indexName=twitter&useScroll=true&scrollKeepAliveMs=30000") .split() .body() .streaming() .to("mock:output") .end(); from("direct:multiSearch") .to("elasticsearch://elasticsearch?operation=MultiSearch"); <route> <from uri="direct:multiSearch"/> <to uri="elasticsearch://elasticsearch?operation=MultiSearch"/> </route> MsearchRequest.Builder builder = new MsearchRequest.Builder().index("twitter").searches( new RequestItem.Builder().header(new MultisearchHeader.Builder().build()) .body(new MultisearchBody.Builder().query(b -> b.matchAll(x -> x)).build()).build(), new RequestItem.Builder().header(new MultisearchHeader.Builder().build()) .body(new MultisearchBody.Builder().query(b -> b.matchAll(x -> x)).build()).build()); List<MultiSearchResponseItem<?>> response = template.requestBody("direct:multiSearch", builder, List.class); <dependency> <groupId>jakarta.json</groupId> <artifactId>jakarta.json-api</artifactId> <version>2.0.2</version> </dependency> spring.elasticsearch.uris=myelkhost:9200 spring.elasticsearch.username=elkuser spring.elasticsearch.password=secure!! spring: autoconfigure: exclude: org.springframework.boot.autoconfigure.elasticsearch.ElasticsearchRestClientAutoConfiguration