Interact with ElasticSearch using Go

by Kevin Pelletier · Published 20 January 2021 · Updated 25 January 2021

ElasticSearch Introduction

Elasticsearch is a distributed analytics and search engine. Elasticsearch is an open source NoSQL database with a focus on search functionalities. It allows you to store, search and analyze huge volumes of data quickly.

Before reading how to interact with Elasticsearch using Go we recommend reading our article explaining what is Elasticsearch and the different components : https://codethekey.com/databases/elasticsearch/

Article: Elasticsearch: The search engine database

ElasticSearch REST API

Elasticsearch is providing multiple REST APIs to configure and interact with the different components. The REST API are exposed using JSON over HTTP. In this page from Elasticsearch you can find the different APIs available to use it: https://www.elastic.co/guide/en/elasticsearch/reference/current/rest-apis.html.

You are able to access multiple components and functionalities. The primary ones are:

Once you go to an API check the different categories associated listed on the right of the webpage.

Example of the Cluster APIs listed on the right of the Cluster API webpage

By exploring the Cluster API you will be for example able to get the REST command to retrieve the status, the health of a cluster, or some informations of a specified node.

If we proceed with the Cluster health API category we can see the cURL command to get the health of an Elasticsearch cluster:

curl -X GET "localhost:9200/_cluster/health?pretty"

To use it you will create command with the following syntax:

curl -X <ACTION> "<HOSTNAME>:<PORT>/<API_CALL>"

The command to get the health of a running Elasticsearch cluster will return a JSON response like this:

{ 
 "cluster_name" : "testcluster", 
 "status" : "yellow",
 "timed_out" : false,
 "number_of_nodes" : 1,
 "number_of_data_nodes" : 1,
 "active_primary_shards" : 1,
 "active_shards" : 1,
 "relocating_shards" : 0,
 "initializing_shards" : 0,
 "unassigned_shards" : 1,
 "delayed_unassigned_shards": 0,
 "number_of_pending_tasks" : 0,
 "number_of_in_flight_fetch": 0,
 "task_max_waiting_in_queue_millis": 0,
 "active_shards_percent_as_number": 50.0 
}

There are also other APIs that can be interesting to mention:

Using standard Go modules

In the previous part, we described the way to interact with Elasticsearch over HTTP requests on the REST APIs. If you want to develop an interface to configure Elasticsearch from scratch, you will have to implement HTTP requests. There is a way for all of the programming languages to create HTTP requests, but we will present how to do it using Go.

The purpose here is to use the standard modules in Go. The main module focusing on HTTP is net/http: https://golang.org/pkg/net/http/.

import ( "net/http" )

Let’s say we want to create a method to retrieve the health of a cluster given the hostname and the port. First, we can implement a method to do HTTP GET requests.

func HttpGetWithJson(url string) (*http.Response, error) {
   client := &http.Client{}
   req, err := http.NewRequest("GET", url, nil)
   req.Header.Add("Accept", "application/json")
   response, err := client.Do(req)
   if err != nil {
      return nil, err
   } else {
      return response, nil
   }
}

Using this method we will be able to send GET requests and receive a JSON response. In the previous part, we’ve seen what the health request is returning. The response has multiple fields like cluster_name, status, timed_out, number_of_nodes etc… In order to retrieve and map the JSON response properly, we can create a struct containing the different fields and annotate it using the encoding/json module.

type ClusterInfo struct {
   Epoch string `json:"epoch"`
   Timestamp string `json:"timestamp"`
   Name string `json:"cluster"`
   Status string `json:"status"`
   Node_total string `json:"node.total"`
   Node_data string `json:"node.data"`
   Shards string `json:"shards"`
   Pri string `json:"pri"`
   Relo string `json:"relo"`
   Init string `json:"init"`
   Unassign string `json:"unassign"`
   Pending_tasks string `json:"pending_tasks"`
   Max_task_wait_time string `json:"max_task_wait_time"`
   Active_shards_percent string `json:"active_shards_percent"`
}

Note that this struct example have different fields from the health response we discussed earlier. This one is based on the _cat health request, which is not recommended as it is better to work directly with JSON responses. It uses this _cat/health API instead of the regular Cluster health API.

With the struct defined, we are then able to implement a method to retrieve and return a struct containing all the status and health informations of a specified cluster.

type ClusterInfo struct {func GetClusterHealth(hostname string, port string) (ClusterInfo, error) {

   var clusterInfo ClusterInfo
   // Url to get the health of the Elasticsearch cluster
   healthURL := "http://" + hostname + ":" + port + "/_cat/health?v&pretty"
   responseHealth, err := HttpGetWithJson(healthURL)
   if err != nil {
      return clusterInfo, err
   } else {
      log.Printf("Successfuly retrieved the health status of the cluster")
   }

   defer responseHealth.Body.Close()

   body, err := ioutil.ReadAll(responseHealth.Body)
   if err != nil {
      return clusterInfo, err
   }

   var clusterInfoToBeFilled []ClusterInfo
   err = json.Unmarshal(body, &clusterInfoToBeFilled)
   if err != nil {
      return clusterInfo, err
   }

   // The result is encapsulated in an array. The cluster is always at index 0.
   clusterInfo = clusterInfoToBeFilled[0]

   return clusterInfo, nil
}

Calling this method will make the HTTP GET request to the Elasticsearch, wrap the JSON response to the struct you created and return it. You can then implement similar methods for most of the APIs available with Elasticsearch. Here is the code to retrieve the health of a cluster, and of a specified index: Github health.go

You may also want to create HTTP PUT requests to create an object in the cluster. For example we may want to create an index in our Elasticsearch cluster. By reading the Create index API documentation we know that the cURL command is simply:

curl -X PUT "localhost:9200/my-index-000001?pretty"

Where my-index-000001 is the name you want to give to this index. In order to do it programmatically in Go, we’ll create a method implementing a PUT request.

func CreateIndexInCluster(hostname string, port string, indicename string) (string, error) {

   // Url to get all the indices of the Elasticsearch cluster
   indexURL := "http://" + hostname + ":" + port + "/" + indicename
   client := &http.Client{}

   req, err := http.NewRequest("PUT", indexURL, nil)
   response, err := client.Do(req)
   if err != nil {
      return "", err
   }

   body, err := ioutil.ReadAll(response.Body)
   if err != nil {
      return "", err
   }

   log.Printf("Delete index request response : %s", string(body))

   return string(body), err
}

Here is the code to create an index, delete an index and list all the indices in Go: Github indices.go

The 2 Github Go files are part of a small Go project interacting with Elasticsearch components and providing gRPC endpoints. It is also wrapped in a Dockerfile and a Helm chart to create a microservice providing a cluster user informations about an Elasticsearch cluster. Here is the Github repository: https://github.com/pelletierkevin/go_microservice_elasticsearch

Using elasticsearch Go module

If you don’t want to implement from scratch the different HTTP requests and interfaces to interact with Elasticsearch there is another solution. You can use the official go client module for Elasticsearch from this Github repository: https://github.com/elastic/go-elasticsearch.

It is provided by the Elastic group and allows any user to interact with their Elasticsearch using Go. You can import it easily using:

import ( "github.com/elastic/go-elasticsearch/v8" )

Note that there are different versions and you can specify it at the end using v8, v7, v6 etc… The elasticsearch is essentially composed of 2 packages:

esapi: (ElasticSearch API) Contains all the definitions to make requests to the APIs
estransport: (ElasticSearch Transport) Contains all the network transports definitions, basically implementing the HTTP requests.

You can then read and get help from the esapi package documentation: https://godoc.org/github.com/elastic/go-elasticsearch/esapi

It then is easy to get started an instanciate an Elasticsearch client using this module by specifying your Elasticsearch URL. You can then retrieve the info of the configured Elasticsearch cluster using this instance:

package main

import (
   "encoding/json" 
   "log" 
   "strings" 
   "github.com/elastic/go-elasticsearch/v8" 
   "github.com/elastic/go-elasticsearch/v8/esapi"
)

func main() {
   cfg := elasticsearch.Config{   
      Addresses: []string{ "http://localhost:9200" }
   }
   es, err := elasticsearch.NewClient(cfg)

   res, err := es.Info()   
   if err != nil {     
      log.Fatalf("Error getting response: %s", err)   
   }   
   defer res.Body.Close()   
   // Check response status   
   if res.IsError() {     
      log.Fatalf("Error: %s", res.String())   
   }   
   // Deserialize the response into a map.   
   if err := json.NewDecoder(res.Body).Decode(&r); err != nil {   
      log.Fatalf("Error parsing the response body: %s", err)    
   }   
   // Print Elasticsearch server version numbers.   
   log.Printf("Server: %s", r["version"].(map[string]interface{})["number"])
}

You may also like...

Leave a Reply Cancel reply

Categories