Spring factories for Elasticsearch

Welcome to the Spring factories for Elasticsearch project.

Actually, since version 1.4.1, this project has been split in two parts:

• Elasticsearch Beyonder which find resources in project classpath to automatically create indices, types and templates.
• This project which is building Client beans using Spring framework.

From 5.0, this project provides 2 implementations of an elasticsearch Client:

• The REST client
• The Transport client (deprecated)

From 6.0, the REST client implementation has been replaced by a High Level REST client. It now also supports X-Pack for official security.

Starting from 7.0, only _doc as a document type is supported if you are not providing the mapping within index settings.

Starting from 7.0, TransportClient has been removed.

Documentation

• For 7.x elasticsearch versions, you are reading the latest documentation.
• For 6.x elasticsearch versions, look at es-6.x branch.
• For 5.x elasticsearch versions, look at es-5.x branch.
• For 2.x elasticsearch versions, look at es-2.x branch.
• For 1.x elasticsearch versions, look at es-1.4 branch.
• For 0.x elasticsearch versions, look at 0.x branch.

Build Status

Thanks to Travis for the build status:

Getting Started

Maven dependency

Import spring-elasticsearch in you project pom.xml file:

<dependency>
  <groupId>fr.pilato.spring</groupId>
  <artifactId>spring-elasticsearch</artifactId>
  <version>7.0</version>
</dependency>

If you want to set a specific version of the High Level Rest client, add it to your pom.xml file:

<dependency>
    <groupId>org.elasticsearch.client</groupId>
    <artifactId>elasticsearch-rest-high-level-client</artifactId>
    <version>7.8.0</version>
</dependency>

If you want to try out the most recent SNAPSHOT version deployed on Sonatype:

<dependency>
  <groupId>fr.pilato.spring</groupId>
  <artifactId>spring-elasticsearch</artifactId>
  <version>7.0-SNAPSHOT</version>
</dependency>

Don't forget to add if needed the following repository in your pom.xml:

<repositories>
    <repository>
        <id>oss-snapshots</id>
        <name>Sonatype OSS Snapshots</name>
        <url>https://oss.sonatype.org/content/repositories/snapshots/</url>
        <releases><enabled>false</enabled></releases>
        <snapshots><enabled>true</enabled></snapshots>
    </repository>
</repositories>

If you depend on an elasticsearch SNAPSHOT version, you need to add the following repository to your pom.xml:

<repositories>
    <repository>
        <id>elastic-snapshots</id>
        <name>Elastic Snapshots</name>
        <url>http://snapshots.elastic.co/maven/</url>
        <releases><enabled>false</enabled></releases>
        <snapshots><enabled>true</enabled></snapshots>
    </repository>
</repositories>

Logger

We are using slf4j for logging but you have to provide the logging implementation you want to use and bind it.

For example for this project we are using for tests log4j2. If you want to do so, add to your pom.xml:

<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-1.2-api</artifactId>
    <version>2.13.3</version>
</dependency>
<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-slf4j-impl</artifactId>
    <version>2.13.3</version>
</dependency>
<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-core</artifactId>
    <version>2.13.3</version>
</dependency>

Using Java Annotations

Let's say you want to use Spring Java Annotations, here is a typical application you can build.

pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>fr.pilato.tests</groupId>
    <artifactId>spring-elasticsearch-test</artifactId>
    <version>1.0-SNAPSHOT</version>

    <dependencies>
        <dependency>
            <groupId>fr.pilato.spring</groupId>
            <artifactId>spring-elasticsearch</artifactId>
            <version>7.0</version>
        </dependency>
    </dependencies>
</project>

App.java:

package fr.pilato.tests;

import fr.pilato.spring.elasticsearch.ElasticsearchRestClientFactoryBean;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestHighLevelClient;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.AnnotationConfigApplicationContext;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.stereotype.Component;

import java.io.IOException;

@Component
public class RestApp {

    @Configuration
    public class AppConfig {
        @Bean
        public RestHighLevelClient esClient() throws Exception {
            ElasticsearchRestClientFactoryBean factory = new ElasticsearchRestClientFactoryBean();
            factory.setEsNodes(new String[]{"http://127.0.0.1:9200"});

            // Begin: If you are running with x-pack
            Properties props = new Properties();
            props.setProperty("xpack.security.user", "elastic:changeme");
		    factory.setProperties(props);
            // End: If you are running with x-pack

            factory.afterPropertiesSet();
            return factory.getObject();
        }
    }

    @Autowired
    private RestHighLevelClient client;

    public static void main(String[] args) throws IOException {
        AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext();
        context.scan("fr.pilato.tests");
        context.refresh();

        RestApp p = context.getBean(RestApp.class);
        p.run();

        context.close();
    }

    private void run() throws IOException {
        // Run a High Level request
        client.info(RequestOptions.DEFAULT);
        // You still have access to the Low Level client
        client.getLowLevel().performRequest(new Request("GET", "/"));
    }
}

Using XML (deprecated)

Using elasticsearch spring namespace for XML files

In your spring context file, just add namespaces like this:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:elasticsearch="http://www.pilato.fr/schema/elasticsearch"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
		http://www.pilato.fr/schema/elasticsearch http://www.pilato.fr/schema/elasticsearch/elasticsearch-7.0.xsd">
</beans>

Getting a rest client bean

You can get a REST High Level Client implementation.

Define a rest client bean

In your spring context file, just define a client like this:

<elasticsearch:rest-client id="esClient" />

By default, you will get an Elasticsearch High Level Rest Client connected to an Elasticsearch node already running at http://localhost:9200.

You can set the nodes you want to connect to:

<elasticsearch:rest-client id="esClient" esNodes="http://localhost:9200,http://localhost:9201" />

Injecting the rest client in your java project

You can use the rest client in your java classes.

import org.elasticsearch.client.RestHighLevelClient;

RestHighLevelClient client = ctx.getBean("esClient", RestHighLevelClient.class);

Better, you should use @Autowired annotation.

// Inject your client...
@Autowired RestHighLevelClient client;

Connecting to a secured X-Pack cluster

You need to define the xpack.security.user property as follows:

<util:properties id="esProperties">
    <prop key="xpack.security.user">elastic:changeme</prop>
</util:properties>

<elasticsearch:rest-client id="esClient" properties="esProperties" />

Asynchronous initialization

Client bean initialization is by default synchronously. It can be initialized asynchronously with the attributes async and taskExecutor.

<task:executor pool-size="4" id="taskExecutor"/>
<elasticsearch:rest-client id="esClient" async="true" taskExecutor="taskExecutor"/>

Asynchronous initialization does not block Spring startup but it continues on background on another thread. Any methods call to these beans before elasticsearch is initialized will be blocked. taskExecutor references a standard Spring's task executor.

Automatically create indices

Managing indexes and types

If you want to manage indices at startup (creating missing indices and applying optional mapping):

<elasticsearch:rest-client id="esClient"
    mappings="twitter" />

This will create an Elasticsearch High Level Rest Client and will create an index twitter.

If you need to manage more than one index, just use a comma separated list:

<elasticsearch:rest-client id="esClient"
    mappings="twitter,facebook" />

If you add in your classpath a file named es/twitter/_settings.json, it will be automatically applied to define settings for your twitter index.

For example, create the following file src/main/resources/es/twitter/_settings.json in your project:

{
  "settings" : {
    "number_of_shards" : 3,
    "number_of_replicas" : 2
  }
}

Also, if you define a file named es/twitter/_doc.json, it will be automatically applied as the mapping for the _doc type in the twitter index.

For example, create the following file src/main/resources/es/twitter/_doc.json in your project:

{
    "properties" : {
      "message" : {"type" : "text", "store" : "yes"}
    }
}

But in general, it's better to use one single _settings.json file which combines all that, like:

{
  "settings" : {
    "number_of_shards" : 3,
    "number_of_replicas" : 2
  },
  "mappings" : {
    "properties" : {
      "message" : {"type" : "text", "store" : "yes"}
    }
  }
}

Using convention over configuration

By default, the factory will find every mapping file located under es directory. So, if you have a mapping file named es/twitter/_doc.json in your classpath, it will be automatically used by the factory without defining anything:

<elasticsearch:rest-client id="esClient" />

You can disable this automatic lookup by setting the autoscan property to false:

<elasticsearch:rest-client id="esClient" autoscan="false" mappings="twitter" />

Creating aliases to indexes

When creating an index, it could be useful to add an alias on it. For example, if you planned to have indexes per year for twitter feeds (twitter2012, twitter2013, twitter2014) and you want to define an alias named twitter, you can use the aliases property:

<elasticsearch:rest-client id="esClient"
    aliases="twitter:twitter2012,twitter:twitter2013,twitter:twitter2014" />

Creating templates

Sometimes it's useful to define a template mapping that will automatically be applied to new indices created.

For example, if you planned to have indexes per year for twitter feeds (twitter2012, twitter2013, twitter2014) and you want to define a template named twitter_template, you can use the templates property:

<!--
    We add also a facebook_template template just for showing how to
    define more than one template...
-->
<elasticsearch:rest-client id="esClient"
    templates="twitter_template,facebook_template" />

To configure your template you have to define a file named es/_template/twitter_template.json in your project:

{
    "index_patterns" : "twitter*",
    "settings" : {
        "number_of_shards" : 1
    },
    "mappings" : {
        "properties" : {
            "message" : {
                "type" : "text",
                "store" : "yes"
            }
        }
    }
}

Changing classpath search path for mapping and settings files

By default, the factory look in es classpath folder to find if there is index settings (_settings.json) or mapping definition (_doc.json). If you need to change it, you can use the classpathRoot property:

<elasticsearch:rest-client id="esClient" classpathRoot="myownfolder" />

So, if a myownfolder/twitter/_settings.json file exists in your classpath, it will be used by the factory.

Merge mappings

If you need to merge mapping for an existing type, set mergeMapping property to true.

<elasticsearch:rest-client id="esClient" mergeMapping="true" />

If merging fails, the factory will not start (BeanCreationException will be raised).

Merge settings

If you need to merge settings for an existing index, add a file named es/twitter/_update_settings.json in your classpath. The factory will detect it and will try to merge settings unless you explicitly set mergeSettings to false.

<elasticsearch:rest-client id="esClient" mergeSettings="false" />

If merging fails, the factory will not start.

Force rebuild indices (use with caution)

For test purpose or for continuous integration, you could force the factory to clean the previous indices when starting the client. It will remove all your datas for every index which has been defined. Just set forceMapping property to true.

<elasticsearch:rest-client id="esClient" forceMapping="true" />

Force rebuild templates (use with caution)

For test purpose or for continuous integration, you could force the factory to clean the previous template when starting the client. Just set forceTemplate property to true.

<elasticsearch:rest-client id="esClient" forceTemplate="true" />

Old fashion bean definition

Note that you can use the old fashion method to define your beans instead of using <elasticsearch:...> namespace:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:util="http://www.springframework.org/schema/util"
	xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
		http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-3.0.xsd">

    <util:map id="esproperties">
        <entry key="cluster.name" value="newclustername"/>
    </util:map>

    <!-- The Rest Client -->
    <bean id="esRestClient" class="fr.pilato.spring.elasticsearch.ElasticsearchRestClientFactoryBean" >
        <property name="esNodes">
            <list>
                <value>http://localhost:9200</value>
                <value>http://localhost:9201</value>
            </list>
        </property>

        <property name="autoscan" value="false" />
        <property name="mappings">
            <list>
                <value>twitter</value>
            </list>
        </property>
        <property name="classpathRoot" value="myownfolder" />
        <property name="forceMapping" value="true" />
        <property name="mergeSettings" value="true" />
        <property name="templates">
            <list>
                <value>twitter_template</value>
            </list>
        </property>
        <property name="forceTemplate" value="true" />
        <property name="aliases">
            <list>
                <value>twitter:twitter2012</value>
                <value>twitter:twitter2013</value>
                <value>twitter:twitter2014</value>
            </list>
        </property>
    </bean>

</beans>

Thanks

Special thanks to

• Nicolas Huray for his contribution about templates
• Nicolas Labrot for his contribution about async

Running tests

If you want to run tests (integration tests) from your IDE, you need to start first an elasticsearch instance.

If you are not using x-pack, then just run the tests from your IDE. Tests are expecting a node running at localhost:9200.

If you are using x-pack, tests are expecting a user named elastic with changeme as the password. You can set this user by running bin/x-pack/setup-passwords interactive.

To run the tests using Maven (on the CLI), just run:

mvn clean install

Note that when the tests are launched with maven, they are not running with x-pack yet. To run tests against x-pack, you need to start elasticsearch with x-pack manually and run the tests with:

mvn clean install -Px-pack

Release guide

To release the project you need to run the release plugin with the release profile as you need to sign the artifacts:

mvn release:prepare
git push --tags
git push
mvn release:perform -Prelease

If you need to skip the tests, run:

mvn release:perform -Prelease -Darguments="-DskipTests"

To announce the release, run:

cd target/checkout
# Run the following command if you want to check the announcement email
mvn changes:announcement-generate
cat target/announcement/announcement.vm

# Announce the release (change your smtp username and password)
mvn changes:announcement-mail -Dchanges.username='YourSmtpUserName' -Dchanges.password='YourSmtpUserPassword'

License

This software is licensed under the Apache 2 license, quoted below.

Copyright 2011-2020 David Pilato

Licensed under the Apache License, Version 2.0 (the "License"); you may not
use this file except in compliance with the License. You may obtain a copy of
the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations under
the License.