Deep Dive to Distributed Service Registry

In my previous article, I discussed how to maintain Resiliency in Microservice/Distributed Architecture. In this tutorial, I discussed Distributed Service Registry.

What is a Distributed Service Registry ?

In a service registry pattern, all the services are registered in a Registry(A Map Data Structure). If any service needs an instance of an another service it contacts Registry and gets the service instance. Very simple isn’t it But when we deal with a distributed environment it's getting highly complex.

Let’s discuss why it is getting so complex in a distributed environment. When we are talking about the distributed environment we assume that each service is deployed in a separate web server/application server to be very simple each service runs in a separate JVM. per service per JVM. As Service registry holds the service name and its instances in a key/Value pair so we need a centralized system which only maintains this service registry and performs CRUD operations on it and each service contact this registry for getting required services, so far so good. But the problem is when we make service registry centralized i.e a separate service or JVM we bring a Single point of failure in our architecture.  Think about the scenario if the Service registry is unavailable then our application falls down as no service can contact service registry so it not gets the desired service and fails. So makes service registry centralize is a bad idea.

See the following picture and think how can we improve the architecture.

 

One thing we can do to improve the architecture if we maintain multiple instances of  Service registry so if one is unavailable other can serve the purpose then there will be no SPF(Single point of failures).

Distributed Service Registry Algorithm

The idea is like there will be one Leader Node(Service registry) all the CRUD operation applied on this node, and it is leader node duty to distribute the updated state of the registry to another Service registry(peer nodes). Although by reading it seems very easy and robust solution but it is very complex to implement and a new set of challenges comes with this architecture.

I am trying to talk about the challenges one by one

Leader Election: As the Service registries are connected to each other How we can select one node as a Leader also if that node fails basis on which we will elect another leader who will take over the responsibility?

State Synchronization: As there are multiple service registry nodes now the question is if Leader registry got updated by adding a new instance or deleting an instance then how leader node distribute that state to others and how it becomes to know that all nodes are updated with the latest state?

To solve these problems Distributed registry take helps from Algorithm.

Bully Algorithm for Leader Election :  In Bully algorithm each node/Service registry has a process Id, greater process Id will select as the Leader node. Suppose assume Leader node is not available then How Bully Algorithm selects the next leader.

Step 1: When a node discovers that Leader node is unavailable it starts the election by sending a claim message to all other nodes who have greater id than this node.

Step 2:  When greater nodes are received this claim message, they either show interest or not provide the response for the message, If they are interested they send previous node a response to stop as they will take it from here and send claim message to all other nodes which have greater Id than this.

Step 3: If it does not receive any response from the higher node, then it selects itself as Leader node and sends message to all other nodes by saying it is the Leader.

State Synchronization by Consensus: If any instances are added or deleted from the leader node it has to be reflected on other nodes so every service registry got the same value. To decide the correct state of the registry majority of the node has to agree on a certain value/latest value.  And faulty nodes needs to update its state to be sync with other nodes and system will be eventually consistent.

Consensus works on two major criteria

Value proposition : Each node/Service registry propose a value..

Agreement: Majority of the node should agree on the proposed value.

A consensus algorithm works in following way

Termination

    Every node decides some value.

Validity

    If all registry proposes the same state say  X, then all correct nodes decide X as an updated state.

Integrity

    Every correct node decides at most one value, and if it decides some value X then X  must have been proposed by Leader Node.

Agreement

    Every correct process must agree on the same value X.

Conclusion:  To maintains a Highly available and fault tolerant service registry in a distributed system Consensus and Leader election is the main techniques to be followed.

Read more

Deep Dive to Hystrix Resiliency Maintenance

In my previous Microservice Tutorial , I have shown How We can use Hystrix as a circuit Breaker and gives our service breathing room to recover itself.

In this tutorial, we will deep dive into Hystrix architecture and will eventually get to know How it manages resiliency in a complex system.

Let assume there are more than 100 Microservices In a complex system and to perform any business functionality more or less it depends on 5-10 underlying Microservices(approximately).

Generally, There are three types of services are found in aforesaid Microservice based architecture

1. Core Services
2. Aggregator service.
3. Edge Service.

Where the Edge service takes the request from UI/client then it forwards to an Aggregator service, Then Aggregator service fans out the request and sends the calls to core-services and Eventually collects responses from Core services and sends back to the Edge Service, Which actually sends the response to UI/Client.

Imagine we have a complex microservice based system where a single incoming request distributed over 10 dependent services, so one request then split to 10 internal requests (Fans out) and goes to individual core service, Now if there are 100 concurrent requests per seconds in a peak time.

So there are 100*10 = 10000 internal requests will be created per second.  Think about the load of the system, Even a minute of delay response from one of the core services can create a bottleneck as in web server(Tomcat, Jetty) there are fixed numbers of Threads in the Thread pool.

and we have 60*100*1=6000 requests are waiting for an individual service for a minute, So We can assume failure is inevitable even if you have 99.99% of uptime for a service-- because when we do a service call we have to depend on two external entities Network and Socket and the sad part is --The control of Network not in developers hand. To make the system resilience we have to deal with this phenomena and have to plan according to it.

Now let see the options we have to deal the above scenario.

Non Blocking request: If Requests are not waiting for the service irrespective of the service is available or not.Only then all request will be either served or rejected. If we follow this then there will be no requests in the waiting queue as if the service is not available or response is getting delayed immediately that request will be rejected and a fallback/default response will be provided to the caller service.  

So request has been short-circuited and the fallback path invoked -- dependent service gets a chance to recover itself. We know Hystrix do this stuff for us. But it is not a very easy task to implement. Internally a complex flow has been maintained by Hystrix to offer Resilient Microservice Architecture.

We will discuss that workflow now.

Deep Dive to Hystrix Workflow :

ThreadPool : Hystrix maintains a Thread pool so that when a service calls to another service --The call will be assigned to one of the thread from that Thread pool, Basically to handle the concurrency It offers Worker Threads. Each Thread has a timeout limit, If the response is not returned within the Thread time out mentioned then the request will be treated as a failure and fallback path will be invoked.

Network and Socket Timeout :  When a Service call to another service, Two things are happening, the data travels through Network and service talks each other via socket. When data travels through Network it has to cross multiple hops so if a hop is slow or down then we can feel network slowness so we have to calculate an average time for a request/response travel time as well as Socket read/write time to calculate thread timeouts in an optimum way.

Hystrix Workflow :  

1. when a service calls a dependent service Hystrix stepped in and checks Hystrix Circuit is open or not if it is open then it returns to the fallback path.
2. If Circuit is not open then  Hystrix checks are all the worker Threads in the pools are in use if so it returns immediately and the fallback path is invoked.
3. If threads are available in pools then it assigns one free thread and waits for the response from dependent service If response time is greater than thread timeout then it again invokes fallback path.
4. If all is well then the actual response is back to the caller service.
5. If for a certain amount of time (default 10 sec) if 50% of the request is failed then Hystrix opens the circuit.
   
   
   
   
   

It is very important to choose Thread pool and Thread time out wisely unless necessary implications are coming into the picture, if Thread pool count is large say it is greater the database connection thread pool  then in spite of the hystrix all the Connection are consumed by the service again if the Thread pools count is less then we can’t serve many request concurrently may be that will cause a performance hit. So choose the Thread pool count based on your system configuration.

Same for Timeouts if the timeout is big then there will more incoming requests in the queue but if Timeout is short then all calls are timeouts even if your service is healthy because the average response time of your service is greater than the Timeout.

Conclusion :  

Nowadays whatever architecture (Microservices/Monolith) you opt for your project, Resiliency is the foremost criteria, it is not an add on feature good to have it is the first class citizen now.

So Resiliency is mandatory -- when you are planning and developing your project also think about Resilience spend time on it, think how you can offer a Resilient Architecture. There are many strategies to achieve resilience like  Request Timeout, Maximum Retries, failing rate etc. So while developing your project Identify the resource intensive areas try to build it in that fashion, so that there is no resource hogging happens. Also, provide a Fallback mechanism so you can avoid cascade failures and eventually achieve a robust system.

Read more

Microservices Communication: Zuul API Gateway

In the previous Microservice tutorial, we have learned How Microservices handling Client side Load balancing.

Here, we will discuss Zuul proxy.

What is a Zuul Proxy ?

The Crux of Microservices pattern is to create an Independent service which can be scaled, deployed independently. So in a complex Business Domain-- more than  50-100 Microservices is very common. Let's Imagine a System where we have fifty Microservices now we have to implement a UI which is kind of a dashboard -- So it calls Multiple services to fetch the important information and show that in UI.

From UI developer perspective --  to collect information from fifty underlying Microservices it has to call fifty Rest API as each Microservice exposes a Rest API for Communication. So the client has to know the details of all Rest API and URL pattern /port to call them. Certainly, it is not sound like a good design. It is kind of a breaching of  Encapsulation, UI has to know all Microservices server/port details to query the services.

Moreover think about the Common aspects of a Web programming like CORS, Authentication, Security, Monitoring in terms of this design -- Each Microservice team has to develop all these aspects into its own service so same code has been replicated over fifty Microservices, Change in the Authentication requirements or CORS policy rippled overall services. It is against of DRY principle. So This type of design is very error prone and rigid. To make it robust It has to be changed in such way so that we have only one entry point where all common aspects code are written and client communicates with that common service. Here the Zuul(The Gatekeeper/demigod ) Concept pops up.

Edge Service: Zuul acts as an API gateway or Edge service. It receives all the request comes from UI and then delegates the request to internal Microservices. So we have to create a brand new Microservice which is Zuul enabled and this service sits on top of all other Microservices. It acts as an Edge service or Client facing service. Its Service API should be exposed to client/UI, Client calls this service as a proxy for Internal Microservice then this service delegates the request to appropriate service.

The advantage of this type of design is common aspects like CORS, Authentication, Security can be put into a centralized service, so all common aspects will be applied on each request, and if any changes occur in the future we just have to update the business logic of this Edge Service.

Also, we can implement any Routing rules or any Filter implementation says we want to append a special tag into the request header before it reaches to internal Microservices we can do it in the Edge service.

As Edge service itself is a Microservice so it can be independently scalable, deployable. So we can perform some load testing also.

Coding Time:

Create a project using http://start.spring.io/ named it as EmployeeZuluService select Zuul and Eureka Discovery module as Edge service itself a Eureka client.

The pom.xml  will look like following

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

   xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

   <modelVersion>4.0.0</modelVersion>

   <groupId>com.example</groupId>

   <artifactId>EmployeeZuulService</artifactId>

   <version>0.0.1-SNAPSHOT</version>

   <packaging>jar</packaging>

   <name>EmployeeZuulService</name>

   <description>Demo project for Spring Boot</description>

   <parent>

      <groupId>org.springframework.boot</groupId>

      <artifactId>spring-boot-starter-parent</artifactId>

      <version>1.5.6.RELEASE</version>

      <relativePath/> <!-- lookup parent from repository -->

   </parent>

   <properties>

      <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

      <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>

      <java.version>1.8</java.version>

      <spring-cloud.version>Dalston.SR2</spring-cloud.version>

   </properties>

   <dependencies>

      <dependency>

          <groupId>org.springframework.cloud</groupId>

          <artifactId>spring-cloud-starter-eureka</artifactId>

      </dependency>

      <dependency>

          <groupId>org.springframework.cloud</groupId>

          <artifactId>spring-cloud-starter-zuul</artifactId>

      </dependency>

      <dependency>

          <groupId>org.springframework.boot</groupId>

          <artifactId>spring-boot-starter-test</artifactId>

          <scope>test</scope>

      </dependency>

   </dependencies>

   <dependencyManagement>

      <dependencies>

          <dependency>

              <groupId>org.springframework.cloud</groupId>

              <artifactId>spring-cloud-dependencies</artifactId>

              <version>${spring-cloud.version}</version>

              <type>pom</type>

              <scope>import</scope>

          </dependency>

      </dependencies>

   </dependencyManagement>

   <build>

      <plugins>

          <plugin>

              <groupId>org.springframework.boot</groupId>

              <artifactId>spring-boot-maven-plugin</artifactId>

          </plugin>

      </plugins>

   </build>

</project>

Next, rename the application.properties to bootstrap.properties and write down the following configuration there.

spring.application.name=EmployeeAPIGateway

eureka.client.serviceUrl.defaultZone:http://localhost:9091/eureka

server.port=8084

security.basic.enable: false   

management.security.enabled: false

zuul.routes.employeeUI.serviceId=EmployeeDashBoard

zuul.host.socket-timeout-millis=30000

As this is a Microservice so its need to be registered in Eureka server so it can be aware of other services.

Also, this service runs on port 8084.

Pay attention to lats two properties

zuul.routes.employeeUI.serviceId=EmployeeDashBoard

By this, we say if any request comes to API gateway in form of

/employeeUI it will redirect to EmployeeDashBoard Microservice

So if you hit the following URL

http://localhost:8084/employeeUI/dashboard/1

It will redirect to

http://localhost:8081/dashboard/1

Note that UI developer only aware of the Gateway service port, it is Gateway service responsibility to route the service to appropriate Microservice.

NB: Zuul can be implemented without Eureka server, In that case, you have to provide the exact URL of the service where it will be redirected.

zuul.host.socket-timeout-millis=30000 -- by this we instruct Spring boot to wait response for 30000 ms unless Zuuls internal Hystrix timeout will kickoff and showing you the error.

Now add @EnableZuulproxy and @EnableDiscoveryClient on top of  EmployeeZuulServiceApplication class

package com.example.EmployeeZuulService;

import org.springframework.boot.SpringApplication;

import org.springframework.boot.autoconfigure.SpringBootApplication;

import org.springframework.cloud.client.discovery.EnableDiscoveryClient;

import org.springframework.cloud.netflix.zuul.EnableZuulProxy;

@EnableZuulProxy

@EnableDiscoveryClient

@SpringBootApplication

public class EmployeeZuulServiceApplication {

   public static void main(String[] args) {

      SpringApplication.run(EmployeeZuulServiceApplication.class, args);

   }

}

Our API gateway service is ready.

Noe starts the config server, Eureka server, EmployeeSearchService, EmployeeDasboradService and, EmployeeZuulService respectively.

If you hit the following URL

http://localhost:8084/employeeUI/dashboard/1

You will see the following output

{

 "employeeId": 1,

 "name": "Shamik  Mitra",

 "practiceArea": "Java",

 "designation": "Architect",

 "companyInfo": "Cognizant"

}

Read more