Simple and Powerful ReverseProxy in Go

In this article we will learn about reverse proxy, where to use it and how to implement it in Golang.

A reverse proxy is a server that sits in front of web servers and forwards client (e.g. web browser) requests to web servers. They give you control over the request from clients and responses from the servers and then we can use that to leverage benefits like caching, increased security and many more.

Before we learn more about reverse proxy, lets quickly understand the difference between a normal proxy (aka forward proxy) and reverse proxy.

In Forward Proxy, proxy retrieves data from another website on the behalf of original client. It sits in front of a client (your browser) and ensures that no backend server ever communicates directly with the client. All the client requests go through the forward proxy and hence the server only communicates with that proxy (assuming proxy is its client). In this case, the proxy masks the client.

Forward Proxy Flow

On the other hand, a Reverse Proxy sits in front of backend servers and ensures that no client ever communicates directly with the servers. All the client requests go to server via reverse proxy and hence client is always communicating to reverse proxy and never with the actual server. In this case, the proxy masks the backend servers. Few examples of reverse proxy are Nginx Reverse proxy, HAProxy.

Reverse Proxy Flow

Reverse Proxy Use cases

Load balancing: a reverse proxy can provide a load balancing solution which will distribute the incoming traffic evenly among the different servers to prevent any single server from becoming overloaded

Preventing security attacks: since the actual web-servers never need to expose their public IP address, attacks such as DDoS can only target the reverse proxy which can be secured with more resources to fend off the cyber attack. Your actual servers are always safe.

Caching: Let’s say your actual servers are in a region far from your users, you can deploy regional reverse proxies which can cache content and serve to local users.

SSL encryption: As SSL communication with each client is computationally expensive, using a reverse proxy it can handle all your SSL related stuff and then freeing up valuable resources on your actual servers.

Golang Implementation

package main

import (
	"log"
	"net/http"
	"net/http/httputil"
	"net/url"
)

// NewProxy takes target host and creates a reverse proxy
func NewProxy(targetHost string) (*httputil.ReverseProxy, error) {
	url, err := url.Parse(targetHost)
	if err != nil {
		return nil, err
	}

	return httputil.NewSingleHostReverseProxy(url), nil
}

// ProxyRequestHandler handles the http request using proxy
func ProxyRequestHandler(proxy *httputil.ReverseProxy) func(http.ResponseWriter, *http.Request) {
	return func(w http.ResponseWriter, r *http.Request) {
		proxy.ServeHTTP(w, r)
	}
}

func main() {
	// initialize a reverse proxy and pass the actual backend server url here
	proxy, err := NewProxy("http://my-api-server.com")
	if err != nil {
		panic(err)
	}

	// handle all requests to your server using the proxy
	http.HandleFunc("/", ProxyRequestHandler(proxy))
	log.Fatal(http.ListenAndServe(":8080", nil))
}

And yes! Thats all it takes to create a simple reverse proxy in Go. We used standard library “net/http/httputil” and created a single host reverse proxy. Any request to our proxy server is proxied to the backend server located at http://my-api-server.com. The code is pretty much self-explanatory if you are from Go background.

Modifying the response

HttpUtil reverse proxy provides us a very simple mechanism to modify the response we got from the servers. This response can be cached or changed based on your use cases. Let’s see how we can make this change.

// NewProxy takes target host and creates a reverse proxy
func NewProxy(targetHost string) (*httputil.ReverseProxy, error) {
	url, err := url.Parse(targetHost)
	if err != nil {
		return nil, err
	}

	proxy := httputil.NewSingleHostReverseProxy(url)
	proxy.ModifyResponse = modifyResponse()
	return proxy, nil
}

func modifyResponse() func(*http.Response) error {
	return func(resp *http.Response) error {
		resp.Header.Set("X-Proxy", "Magical")
		return nil
	}
}

You can see in modifyResponse method, we are setting a custom header. Similarly you can read the response body, make changes to it, cache it and then set it back for the client.

In ModifyResponse you can also return an error (if you encounter it while processing response) which then will be handled by proxy.ErrorHandler. ErrorHandler is automatically called if you set error inside modifyResponse.

// NewProxy takes target host and creates a reverse proxy
func NewProxy(targetHost string) (*httputil.ReverseProxy, error) {
	url, err := url.Parse(targetHost)
	if err != nil {
		return nil, err
	}

	proxy := httputil.NewSingleHostReverseProxy(url)
	proxy.ModifyResponse = modifyResponse()
	proxy.ErrorHandler = errorHandler()
	return proxy, nil
}

func errorHandler() func(http.ResponseWriter, *http.Request, error) {
	return func(w http.ResponseWriter, req *http.Request, err error) {
		fmt.Printf("Got error while modifying response: %v \n", err)
		return
	}
}

func modifyResponse() func(*http.Response) error {
	return func(resp *http.Response) error {
		return errors.New("response body is invalid")
	}
}

Modifying the request

You can also modify the request before sending it to the server. In below example we are adding a header before sending it to server. Similarly, you can make any changes to the request before sending.

// NewProxy takes target host and creates a reverse proxy
func NewProxy(targetHost string) (*httputil.ReverseProxy, error) {
	url, err := url.Parse(targetHost)
	if err != nil {
		return nil, err
	}

	proxy := httputil.NewSingleHostReverseProxy(url)

	originalDirector := proxy.Director
	proxy.Director = func(req *http.Request) {
		originalDirector(req)
		modifyRequest(req)
	}

	proxy.ModifyResponse = modifyResponse()
	proxy.ErrorHandler = errorHandler()
	return proxy, nil
}

func modifyRequest(req *http.Request) {
	req.Header.Set("X-Proxy", "Simple-Reverse-Proxy")
}

Complete code

package main

import (
	"errors"
	"fmt"
	"log"
	"net/http"
	"net/http/httputil"
	"net/url"
)

// NewProxy takes target host and creates a reverse proxy
func NewProxy(targetHost string) (*httputil.ReverseProxy, error) {
	url, err := url.Parse(targetHost)
	if err != nil {
		return nil, err
	}

	proxy := httputil.NewSingleHostReverseProxy(url)

	originalDirector := proxy.Director
	proxy.Director = func(req *http.Request) {
		originalDirector(req)
		modifyRequest(req)
	}

	proxy.ModifyResponse = modifyResponse()
	proxy.ErrorHandler = errorHandler()
	return proxy, nil
}

func modifyRequest(req *http.Request) {
	req.Header.Set("X-Proxy", "Simple-Reverse-Proxy")
}

func errorHandler() func(http.ResponseWriter, *http.Request, error) {
	return func(w http.ResponseWriter, req *http.Request, err error) {
		fmt.Printf("Got error while modifying response: %v \n", err)
		return
	}
}

func modifyResponse() func(*http.Response) error {
	return func(resp *http.Response) error {
		return errors.New("response body is invalid")
	}
}

// ProxyRequestHandler handles the http request using proxy
func ProxyRequestHandler(proxy *httputil.ReverseProxy) func(http.ResponseWriter, *http.Request) {
	return func(w http.ResponseWriter, r *http.Request) {
		proxy.ServeHTTP(w, r)
	}
}

func main() {
	// initialize a reverse proxy and pass the actual backend server url here
	proxy, err := NewProxy("http://my-api-server.com")
	if err != nil {
		panic(err)
	}

	// handle all requests to your server using the proxy
	http.HandleFunc("/", ProxyRequestHandler(proxy))
	log.Fatal(http.ListenAndServe(":8080", nil))
}

Reverse proxy is very powerful and can be used for multiple use cases as explained above. You can try customising it as per your case and if you face any issues, I will be very happy to help you with that. If you found the article interesting, please share it so that it can reach to other gophers! Thanks a lot for reading.

Tagged : / /

Golang gRPC communication made easy!

In this post we are going to build a simple gRPC client and server application. I was recently trying to set this up in one of my projects and was struggling to do it. Hence I decided to write a post which focuses only on the exact steps to setup gRPC communication between server and client written in Golang.

I will not cover the basics of gRPC and why we should use it. If you are new to it you can read about it here. Before we start here are the pre-requisites:

Create project directory and initialise Go modules

Don’t worry if you are unaware with some of the files in below tree structure, for now just create empty files / directories using the given commands.

$ mkdir exchange-app
$ cd exchange-app
$ go mod init bank/exchange-app
$ mkdir proto
$ touch server.go client.go proto/currency_conversion.proto

$ tree .
.
├── client.go
├── go.mod
├── proto
│   └── currency_conversion.proto
└── server.go

Install gRPC and protoc-gen-go

$ go get -u google.golang.org/grpc
$ go get -u github.com/golang/protobuf/protoc-gen-go

Install protobuf

# Linux 
$ apt install -y protobuf-compiler
$ protoc --version 

# Mac OS
$ brew install protobuf
$ protoc --version 

For other operating systems, please refer this link.

Let’s setup the client and server communication

Define a service in a .proto file

The first step is to complete the currency_conversion.proto file which defines the gRPC service and the request and response messages using protocol buffers. Now open the currency_conversion.proto file in a text editor and paste the below content in it.

// currency_conversion.proto
// this file will be used to generate the grpc client and server interfaces in Go

syntax = "proto3";

package proto;

message CurrencyConversionRequest {
    string from = 1;
    string to = 2;
    float value = 3;
}

message CurrencyConversionResponse {
    float converted_value = 1;
}

service CurrencyConversionService {
    rpc Convert(CurrencyConversionRequest) returns (CurrencyConversionResponse) {}
}

Generate server and client code using the protocol buffer compiler

From the project root directory, run the below command to generate the gRPC client and server interfaces in Go which can be used as a Go package in the client and server code.

$ protoc -I proto/ proto/currency_conversion.proto --go_out=plugins=grpc:proto/

After executing the above command, if we again look at the tree structure of the current directory, we can see that a currency_conversion.pb.go file is generated.

$ tree .
.
├── client.go
├── go.mod
├── proto
│   ├── currency_conversion.pb.go // generated file
│   └── currency_conversion.proto
└── server.go

The generated file contains the following contents:

  • Golang struct type for each message defined in currency_conversion.proto file. It also has helper methods to get value of each field defined in the struct. As an example, look at the generated code for proto message CurrencyConversionRequest
type CurrencyConversionRequest struct {
    state         protoimpl.MessageState
    sizeCache     protoimpl.SizeCache
    unknownFields protoimpl.UnknownFields

    From  string  `protobuf:"bytes,1,opt,name=from,proto3" json:"from,omitempty"`
    To    string  `protobuf:"bytes,2,opt,name=to,proto3" json:"to,omitempty"`
    Value float32 `protobuf:"fixed32,3,opt,name=value,proto3" json:"value,omitempty"`
}

func (x *CurrencyConversionRequest) GetFrom() string {
    if x != nil {
        return x.From
    }
    return ""
}

func (x *CurrencyConversionRequest) GetTo() string {
    if x != nil {
        return x.To
    }
    return ""
}

func (x *CurrencyConversionRequest) GetValue() float32 {
    if x != nil {
        return x.Value
    }
    return 0
}
  • An interface type which clients will use to call the methods implemented by the grpc server
type CurrencyConversionServiceClient interface {
    Convert(ctx context.Context, in *CurrencyConversionRequest, opts ...grpc.CallOption) (*CurrencyConversionResponse, error)
}

type currencyConversionServiceClient struct {
    cc grpc.ClientConnInterface
}

func NewCurrencyConversionServiceClient(cc grpc.ClientConnInterface) CurrencyConversionServiceClient {
    return &currencyConversionServiceClient{cc}
}

func (c *currencyConversionServiceClient) Convert(ctx context.Context, in *CurrencyConversionRequest, opts ...grpc.CallOption) (*CurrencyConversionResponse, error) {
    out := new(CurrencyConversionResponse)
    err := c.cc.Invoke(ctx, "/currency_conversion.CurrencyConversionService/Convert", in, out, opts...)
    if err != nil {
        return nil, err
    }
    return out, nil
}
  • An interface type which the servers will implement to serve the client requests
// CurrencyConversionServiceServer is the server API for CurrencyConversionService service.
type CurrencyConversionServiceServer interface {
    Convert(context.Context, *CurrencyConversionRequest) (*CurrencyConversionResponse, error)
}

// this is the function servers will use to register their struct object which implements the grpc service methods
func RegisterCurrencyConversionServiceServer(s *grpc.Server, srv CurrencyConversionServiceServer) {
    s.RegisterService(&_CurrencyConversionService_serviceDesc, srv)
}

Finish the server.go file

Now let’s implement the server.go

package main

import (
	"bank/exchange-app/proto"
	"context"
	"fmt"
	"net"

	"google.golang.org/grpc"
)

// myGRPCServer implements CurrencyConversionServiceServer interface
type myGRPCServer struct {
}

// actual implementation of the Convert method
func (s *myGRPCServer) Convert(ctx context.Context, request *proto.CurrencyConversionRequest) (*proto.CurrencyConversionResponse, error) {
	fmt.Printf("request received to convert currency %+v \n", request)

	// NOTE: here you can write the logic to convert the currency

	return &proto.CurrencyConversionResponse{
		ConvertedValue: 74.02,
	}, nil
}

func main() {
	fmt.Println("starting gRPC server application")

	// start a listener on the port you want to start the grpc server
	lis, err := net.Listen("tcp", ":9000")
	if err != nil {
		fmt.Println(err)
		return
	}

	// create a grpc server object
	srv := grpc.NewServer()

	// pass the address of the struct which implements the CurrencyConversionServiceServer interface
	proto.RegisterCurrencyConversionServiceServer(srv, &myGRPCServer{})

	// start the grpc server
	err = srv.Serve(lis)
	if err != nil {
		return
	}

	return
}

But what have we done here ? Let’s see what steps can be followed to write the server.go code

  1. Create a struct (give it any name you want). It should implement the CurrencyConversionServiceServer interface
// myGRPCServer implements CurrencyConversionServiceServer interface
type myGRPCServer struct {
}

// actual implementation of the Convert method
func (s * myGRPCServer) Convert(ctx context.Context, request *proto.CurrencyConversionRequest) (*proto.CurrencyConversionResponse, error) {
    fmt.Printf("request received to convert currency %+v \n", request)

    // NOTE: here you can write the logic to convert the currency

    return &proto.CurrencyConversionResponse{
        ConvertedValue: 74.02,
    }, nil
}

2. Start a listener on 9000 port (you can use port of your choice)

lis, err := net.Listen("tcp", ":9000")
if err != nil {
    fmt.Println(err)
    return
}

3. Using google.golang.org/grpc package’s NewServer method, create a new grpc server object. It creates a gRPC server object which has no service registered and has not started to accept requests yet

 srv := grpc.NewServer()

4. Register the defined struct to inform grpc about the object which implements the grpc server

proto.RegisterCurrencyConversionServiceServer(srv, & myGRPCServer{})

5. Start the gRPC server now which will listen for incoming requests at port 9000

// start the grpc server
err = srv.Serve(lis)
if err != nil {
    return
}

Finish the client.go file

Here is the implemented grpc client.go file and you can find the breakdown of each step after it.

package main

import (
	"context"
	"fmt"

	"bank/exchange-app/proto"

	"google.golang.org/grpc"
)

func main() {
	fmt.Println("starting grpc client application")

	// use grpc.Dial to create a grpc connection to running grpc server
	conn, err := grpc.Dial(":9000", grpc.WithInsecure(), grpc.WithBlock())
	if err != nil {
		fmt.Println(err)
		return
	}

	defer conn.Close()

	// use the NewCurrencyConversionServiceClient function defined in generated .pb.go file
	// to generate a grpc client object
	// this object can be used to call methods implemented in grpc server
	client := proto.NewCurrencyConversionServiceClient(conn)

	// create the grpc request
	request := &proto.CurrencyConversionRequest{
		From:  "INR",
		To:    "USD",
		Value: 1,
	}

	// call the grpc server method Convert using the client
	response, err := client.Convert(context.Background(), request)
	if err != nil {
		fmt.Println(err)
		return
	}

	// print the response
	fmt.Println("converted currency value is: ", response.GetConvertedValue())
}
  1. Dial a connection to the server
// use grpc.Dial to create a grpc connection to running grpc server
conn, err := grpc.Dial(":9000", grpc.WithInsecure(), grpc.WithBlock())
if err != nil {
    fmt.Println(err)
    return
}

2. Create a grpc client object using the connection object. This object will be used to call server methods

client := proto.NewCurrencyConversionServiceClient(conn)

3. Create the grpc request and call the server methods

// create the grpc request
request := &proto.CurrencyConversionRequest{
    From:  "INR",
    To:    "USD",
    Value: 1,
}

// call the grpc server method Convert using the client
response, err := client.Convert(context.Background(), request)
if err != nil {
    fmt.Println(err)
    return
}

And that’s it.

Time to test

  1. Run the GRPC server
$ go run server.go
starting gRPC server application

2. Run the client

$ go run client.go
starting grpc client application
converted currency value is:  74.02

When you run the client code, server will also show the log message

request received to convert currency from:"INR" to:"USD" value:1

That was it! I hope you enjoyed the post and learned how we can establish gRPC communication between client and server written in Golang. If you enjoyed the post, please like and share. Any feedbacks or suggestions will encourage me to write more. Thanks for reading!

Tagged : / / / / /

5 simple examples to understand Elasticsearch aggregation

Elasticsearch aggregation give us the ability to ask questions to our data. The ability to group and find out statistics (such as sum, average, min, max) on our data by using a simple search query. In this post, we will see some very simple examples to understand how powerful and easy it is to use Elasticsearch aggregation. I will also share a postman collection link at the bottom of this post in case you want to try out these queries on your own.

Let’s say we have a car store and to formulate some reports we are interested in the following:

  • What is the average price of sold cars having manufacturer Audi ?
  • Find all cars made by Ford and average price of ford cars sold in Jul 2020
  • What is the total price of all cars sold in Jul 2020 ?
  • Which are the most popular car manufacturers?
  • How much sales were made each month ?

So let’s get started right away and look at our sample data on which we will be performing aggregation.

{
  "_index": "cars",
  "_type": "_doc",
  "_id": "1",
  "_score": 1.0,
  "_source": {
    "manufacturer": "Audi",
    "model": "A6",
    "price": 3900000,
    "sold_date": "2020-03-10"
  }
},
{
  "_index": "cars",
  "_type": "_doc",
  "_id": "2",
  "_score": 1.0,
  "_source": {
    "manufacturer": "Ford",
    "model": "Fiesta",
    "price": 580000,
    "sold_date": "2020-07-18"
  }
},
{
  "_index": "cars",
  "_type": "_doc",
  "_id": "3",
  "_score": 1.0,
  "_source": {
    "manufacturer": "Audi",
    "model": "A7",
    "price": 6500000,
    "sold_date": "2020-05-28"
  }
},
{
  "_index": "cars",
  "_type": "_doc",
  "_id": "4",
  "_score": 1.0,
  "_source": {
    "manufacturer": "Audi",
    "model": "A8",
    "price": 14900000,
    "sold_date": "2020-06-10"
  }
},
{
  "_index": "cars",
  "_type": "_doc",
  "_id": "5",
  "_score": 1.0,
  "_source": {
    "manufacturer": "Ford",
    "model": "Linea",
    "price": 420000,
    "sold_date": "2020-05-26"
  }
},
{
  "_index": "cars",
  "_type": "_doc",
  "_id": "6",
  "_score": 1.0,
  "_source": {
    "manufacturer": "Ford",
    "model": "Figo",
    "price": 480000,
    "sold_date": "2020-07-13"
  }
},
{
  "_index": "cars",
  "_type": "_doc",
  "_id": "7",
  "_score": 1.0,
  "_source": {
    "manufacturer": "Maruti",
    "model": "Swift",
    "price": 680000,
    "sold_date": "2020-05-25"
  }
},
{
  "_index": "cars",
  "_type": "_doc",
  "_id": "8",
  "_score": 1.0,
  "_source": {
    "manufacturer": "Tata",
    "model": "Altroz",
    "price": 680000,
    "sold_date": "2020-03-25"
  }
},
{
  "_index": "cars",
  "_type": "_doc",
  "_id": "9",
  "_score": 1.0,
  "_source": {
    "manufacturer": "Tata",
    "model": "Tigor",
    "price": 520000,
    "sold_date": "2020-07-25"
  }
}

What is the average price of sold cars having manufacturer Audi ?

From our sample data let’s find this manually

SnoCarPrice
1.Audi A63900000
2.Audi A76500000
3. Audi A814900000
Average 8433333.33

Now in order to find it using aggregation, we have to use the following query:

{
    "size": 0,
    "query" : {
      "match": {
        "manufacturer": "audi"
      }
    }, 
    "aggs": {
      "average_price": {
        "avg": {
          "field": "price"
        }
      }
    }
}

The query is very simple, we are just asking elasticsearch to first filter the records which have manufacturer Audi and then on all the records in the result do an average aggregation on the field price. Now lets see the response from elasticsearch:

{
  "took": 29,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "average_price": {
      "value": 8433333.333333334
    }
  }
}

Wow! here we go, with such a simple query we are able to find the correct result. Let’s spice up things now and move to the next one.

Find all cars made by Ford and average price of ford cars sold in Jul 2020

Now this is an interesting one, we want to see all the cars which have manufacturer Ford, but we need average price of only those which are sold in month of Jul 2020.

All cars manufactured by Ford

S.NoCarPriceSold On
1Ford Fiesta58000018 Jul 2020
2Ford Linea42000026 May 2020
3Ford Figo48000013 Jul 2020

So there are total 3 cars made by Ford and the average price of cars sold in Jul 2020 is (580000 + 480000) / 2 = 530000

Now lets see what elasticsearch query we can use to get this result:

{
  "query": {
    "match": {
      "manufacturer": "ford"
    }
  },
  "aggs": {
    "recent_sales": {
      "filter": {
        "range": {
          "sold_date": {
            "gte": "2020-07-01", 
            "lte": "2020-07-31"
          }
        }
      },
      "aggs": {
        "average_price": {
          "avg": {
            "field": "price"
          }
        }
      }
    }
  }
}

If you compare this query with the first one, the only difference is that we have added one extra date filter inside the aggs block. This is how we can filter results before performing aggregation on them.

Also if you look carefully, there is one more difference i.e in the first query we have used a

"size": 0

parameter.

There are many occasions when aggregations are required but search hits are not. For these cases the hits can be ignored by setting size=0

You can verify this by looking at the response of the first query, in that only the aggregation result was returned and we do not see the actual documents which are used to evaluate that result. Now let’s see the result of our second query:

{
  "took": 61,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": 1.1451323,
    "hits": [
      {
        "_index": "cars",
        "_type": "_doc",
        "_id": "2",
        "_score": 1.1451323,
        "_source": {
          "manufacturer": "Ford",
          "model": "Fiesta",
          "price": 580000,
          "sold_date": "2020-07-18"
        }
      },
      {
        "_index": "cars",
        "_type": "_doc",
        "_id": "5",
        "_score": 1.1451323,
        "_source": {
          "manufacturer": "Ford",
          "model": "Linea",
          "price": 420000,
          "sold_date": "2020-05-26"
        }
      },
      {
        "_index": "cars",
        "_type": "_doc",
        "_id": "6",
        "_score": 1.1451323,
        "_source": {
          "manufacturer": "Ford",
          "model": "Figo",
          "price": 480000,
          "sold_date": "2020-07-13"
        }
      }
    ]
  },
  "aggregations": {
    "recent_sales": {
      "doc_count": 2,
      "average_price": {
        "value": 530000.0
      }
    }
  }
}

Wohoo! The results are accurate again. Also all the cars made by Ford are returned in the response. Let’s move to the next one now.

What is the total price of all cars sold in Jul 2020 ?

Again, let’s solve this manually first:

S.NoCarSold DatePrice
1Ford Fiesta18 Jul 2020580000
2Ford Figo13 Jul 2020480000
3Tata Tigor25 Jul 2020520000
Total1580000

Let’s see what query we can use to solve this:

{
    "size": 0,
    "query" : {
      "range": {
        "sold_date": {
          "gte": "2020-07-01",
          "lte": "2020-07-31"
        }
      }
    },
    "aggs": {
      "total_price": {
        "sum": {
          "field": "price"
        }
      }
    }
}

Cool, a simple one. We simply apply a query range to filter out all cars sold in month of Jul 2020 and then we do a sum aggregation on them to find out the result. Let’s check the response:

{
  "took": 3,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "total_price": {
      "value": 1580000.0
    }
  }
}

Awesome, its correct.

Which are the most popular car manufacturers?

For the sake of this article, let’s say the manufacturers which have sold most cars in last 3 months are considered to be popular.

S.NoCarTotal sold in last 3 months
1Ford3
2Audi2
3Maruti1
4Tata1

Now let’s see what query we can use to find out this result:

{
  "size": 0,
  "query": {
    "range": {
      "sold_date": {
        "from": "now-3M"
      }
    }
  },
  "aggs": {
    "group_by_make": {
      "terms": {
        "field": "manufacturer.keyword"
      }
    }
  }
}

Again the query is very simple, we first filter out all the cars which were sold in last 3 months and then we simply group them by their manufacturer. Cool, lets see the response now:

{
  "took": 86,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 7,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "group_by_make": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "Ford",
          "doc_count": 3
        },
        {
          "key": "Audi",
          "doc_count": 2
        },
        {
          "key": "Maruti",
          "doc_count": 1
        },
        {
          "key": "Tata",
          "doc_count": 1
        }
      ]
    }
  }
}

Great, our query worked fine!

How much sales were made each month ?

This is a tricky one!

S.NoMonthTotal Sales
1Mar 20204580000
2Apr 20200
3May 20207600000
4Jun 202014900000
5Jul 20201580000

Using the below query:

{
  "size": 0,
  "aggs": {
    "sales_over_time": {
      "date_histogram": {
        "field": "sold_date",
        "interval": "month",
        "format": "MM-yyyy"
      },
      "aggs": {
        "monthly_sales": {
          "sum": {
            "field": "price"
          }
        }
      }
    }
  }
}

Don’t worry if the query looks complex. We are just doing nested aggregation. First we group all the data in monthly buckets and then on each bucket we perform a metric aggregation to sum the price. Think of this as a pipeline of aggregations.

{
  "took": 29,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 9,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "sales_over_time": {
      "buckets": [
        {
          "key_as_string": "03-2020",
          "key": 1583020800000,
          "doc_count": 2,
          "monthly_sales": {
            "value": 4580000.0
          }
        },
        {
          "key_as_string": "04-2020",
          "key": 1585699200000,
          "doc_count": 0,
          "monthly_sales": {
            "value": 0.0
          }
        },
        {
          "key_as_string": "05-2020",
          "key": 1588291200000,
          "doc_count": 3,
          "monthly_sales": {
            "value": 7600000.0
          }
        },
        {
          "key_as_string": "06-2020",
          "key": 1590969600000,
          "doc_count": 1,
          "monthly_sales": {
            "value": 1.49E7
          }
        },
        {
          "key_as_string": "07-2020",
          "key": 1593561600000,
          "doc_count": 3,
          "monthly_sales": {
            "value": 1580000.0
          }
        }
      ]
    }
  }
}

Wow! that was accurate and definitely not very difficult to achieve.

All right that was it for this post. Hope you had fun and experienced power of Elasticsearch aggregations. If you enjoyed the post, please like and share it so that it also reaches other valuable readers. If you have any doubts or feedbacks, please scroll to the bottom and leave a comment. Again, thanks for reading 🙂

Bonus:

Postman collection link: https://www.postman.com/collections/201d2f5fea372d02fc55

Tagged : / / / /

Simple and elegant Vim IDE setup for Go

I just love working in Vim and cannot even think of switching to a new editor. If you are also a Vim lover and you use Golang, you must read this post to make sure you have the right setup.

Here is how my vim looks like and yes the look is very simple though powerful.

Following are the most important features which I prefer to have:

  1. Run gofmt and goimports on save
  2. Peek at documentation
  3. Autocomplete
  4. Jump to symbols or functions definitions
  5. Automatic variable type and function signature info
  6. Pretty and lightweight status bar at bottom
  7. Version control information

In order to have all these features integrated in Vim, I used the following plugins:

vim-go development plugin

Installation
git clone https://github.com/fatih/vim-go.git ~/.vim/pack/plugins/start/vim-go
Uninstall
rm -r ~/.vim/pack/plugins/start/vim-go 

Lean and Mean status bar

Install

git clone https://github.com/vim-airline/vim-airline ~/.vim/pack/dist/start/vim-airline

Uninstall

rm -r ~/.vim/pack/dist/start/vim-airline

An awesome Git wrapper

Install

git clone https://github.com/tpope/vim-fugitive.git ~/.vim/pack/dist/start/vim-fugitive 

Uninstall

rm -r ~/.vim/pack/dist/start/vim-fugitive

After installing all the plugins, here is how you can configure ~/.vimrc file:

syntax on

colo pablo

set encoding=utf-8

set fileencoding=utf-8

set number

set shiftwidth=2

" settings specific to Go file types
autocmd Filetype go setlocal tabstop=4 shiftwidth=4 softtabstop=4

filetype plugin indent on

set backspace=indent,eol,start

" vim-go commands
let g:go_fmt_command = "goimports"
let g:go_auto_type_info = 1
let g:go_highlight_functions = 1
let g:go_highlight_methods = 1
let g:go_highlight_fields = 1
let g:go_highlight_types = 1
let g:go_highlight_operators = 1
let g:go_highlight_build_constraints = 1

I hope you have enjoyed this post and it will be helpful for you to setup your Vim. Please like and share and feel free to comment if you have any suggestions or feedback.

Tagged : / / / /

Most common openssl commands

This post is a collection of most common and helpful openssl commands which we need to use to deal with tasks like generating keys, CSRs, certificates.

# Verify or check content

Verify a certificate file

openssl x509 -in thecodersstop.pem -text -noout

Verify a CSR request

openssl req -text -noout -verify -in thecodersstop.csr

Verify a private key file

openssl rsa -in thecodersstop-key.key -check

Check contents of PKCS12 format cert (.pfx or p12)

openssl pkcs12 -info -nodes -in thecodersstop.pfx

Verify a certificate against a CA certificate

openssl verify -CAfile ca.pem thecodersstop.pem

# Create using openssl

Create a Self-Signed Certificate

openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout thecodersstop-key.key -out thecodersstop.pem

Create private key and CSR (certificate signing request)

openssl req -out thecodersstop.csr -newkey rsa:2048 -nodes -keyout thecodersstop-key.key

Create RSA private key

openssl genrsa -out thecodersstop-key.key 2048

Generate a CSR using an existing private key

openssl req -out thecodersstop.csr -key thecodersstop-key.key -new

Create unecrypted key from a password protected private key

openssl rsa -in thecodersstop-key.key -out thecodersstop-key-unencrypted.pem -passin pass:1234

Remove passphrase from a private key

openssl rsa -in thecodersstop-key.key -out thecodersstop-key-new.key

# Convert using openssl

Convert pem certificate and private Key to PKCS#12 format (.pfx or p12)

openssl pkcs12 -export -out thecodersstop.pfx -inkey thecodersstop-key.key -in thecodersstop.pem

Convert PEM certificate to DER format

openssl x509 -outform der -in thecodersstop.pem -out thecodersstop.der

Please like and share if you found this post useful. If you need help with a command which is not there in the list, please let us know, we will help you and add it here also.

Any suggestions and feedback are welcome, please comment so that we can consider your valuable feedback. Thanks for reading!

Tagged : / / / /
%d bloggers like this: