Java:: What is invokeDynamic

In my previous article, I discussed --How JVM determines which methods to call at runtime?Also, we learned that In bytecode level, java compiler uses 4 special opcodes for method invoking
invokestatic,invokeinterface,invokevirtual,invokespecial.

Now, the question arises if all types of methods invoking (interface, static method, instance method, constructor etc.) are covered by 4 opcodes, why there is another opcode invokeDynamic has been introduced?
Why invokeDynamic?
To understand why invokedymic is required let dig down to a real problem,  for a moment think you are building a framework where based on the command passed from UI you load a class runtime and invoke a particular method of that class.
Say you are building a Web framework like Struts -- Now based on the URL you have to map a java class and invoke a specific method so you will create an XML(like struts-config.xml) where developer explicitly put the strategy -- which URL map to which class and which method to be called.
Let's see the prototype of the XML.

<controller url="/employee" class="com.java.example.EmployeeController">
<param key="add" method="executeAdd">
<param key="update" method="executeUpdate">
<param key="delete" method="executeDelete">
<goto key="success"  path="/success.jsp">
<goto key="faliure"  path="/error.jsp">
</controller>
Seeing, this configuration XML one thing is clear, based on the action taken by the user in UI, different method will be called for EmployeController class, If user performs an add then internally an 'add' parameter will be passed and framework checks that which method has to call for 'add' parameter and found it is executeAdd. Until  User does not take an action no one can tell which method will be called, so all the decision has to be taken at runtime based on the parameters passed(add, edit, delete).
Think how you can design the same in java, what is the weapon available in java to implement the same, Of course, by the Reflection mechanism-- By the reflection you are able to load the Employee controller class and based on the parameter you can invoke the appropriate method.

Pseudocode will be like that.

Class controller = Class.forname("com.example.EmployeeController");
Controller empController = controller.newInstance();
Class inputParams[] = {javax.servlet.http.HttpRequest,javax.servlet.http.HttpResponse};
String methodSuffix = makeFirstLetterCapital(empController.getParam());//add to Add
Method tobeInvoked= Class.declaredMethod("execute"+methodSuffix,inputParams);
tobeInvoked.invoke(empController,request,response);
This runs very well, It will take the decision on runtime except for one problem. Reflection has huge performance penalties because it is always checking security constraints like what is the method access specifiers, is the caller has permission to call the method etc, so for this reflection is bit slow.
This is the one reason to introduce invokeDynamic in the place of reflection, it facilitates dynamic programming -- where type checking and method resolution done at runtime. So by that you can add remove methods programmatically -- You can do bytecode engineering runtime I mean, you can insert a new method which is not in the class definition !!! or override a method runtime!!! invokeDynamic provides this type of flexibility, unlike reflection.
invokeDynamic is as fast as other opcodes, but to introducing invoke dynamic in Java is not an easy task.
Java is a statically type so type checking is done at compile time also compiler checks a method is available in the class or not if the method is not available it throws compiler error so the questions is
How can we trick compiler in such a way where we can introduce a new method at runtime to be specific type checking and method resolution done at run-time?

How invokeDyamic works internally?
To work the invokeDynamic opcode correctly two important components are MethodHandle and CallSite
MethodHandle: Method handles wraps the metadata about a method-- It holds the method signature so by invoking it you can invoke a method on an Object at runtime.
CallSite : Callsite can hold Method handle and if the Call site is mutable we can change the Method handle time to time, so based on the parameter we can change the MethodHandle and as Method handle holds the method so runtime we can change the method call without altering the bytecode instruction that is invoked dynamic, so same method invocation can execute different methods based on the parameters.
Actually when the invokeDynamic instruction is read by the interpreter following procedure happens underhood--  invokedynamic instruction is associated with a special method called the bootstrap method (BSM). When the invokedynamic instruction is read by the interpreter, the BSM is invoked. It returns an object called Callsite (hold a method handle) that indicates which method actually execute.

So, Calliste works like the subline in the Train track-- Trani always runs on a straight track but when it needs to change it's direction Line Engineers are pull the liver of subline so it joins with another line according to the needs and Train just pass on that line and change the track, Calliste mimic the same methodology when based on parameter it changes the underlying method handle and same method invocation call diffrent method at runtime.
Let see the diagram,



Conclusion: invokeDynamic is a valuable inclusion in terms of framework level developer, By invokedynamic we can get the real essence of Dynamic programming in JVM, Many languages which run on JVM uses the invokedynamic to achieve dynamic type checking also Lamda uses invokedynamic, In my next tutorial I will show you how to write Callsite and MethodHandle and How you can changeMethodHandle under Callsite dynamically.




How does method dispatch happen in Java?

 Have you ever wondered when you call a method like a list.add("Shamik"), How the actual method invoke in runtime?
If you want to discover the How part then you are in the right place else you can easily skip the article as it is not related to coding perspective. we know in Java we maintain two steps process
Compiler compiles and make the bytecodes
The interpreter takes bytecode and changes the instruction to machine code.
But think, When your code compiles to bytecode how the method call looks like and in the runtime how the dynamic linking happens, In a simple word how JVM find the actual method and call that method.
In this tutorial, we will discuss the same.
In java(till Java7) we have four types of method

1. Static methods.
2. private, package private or public methods
3. Interface methods declaration.
4. Some special methods like the constructor, super etc.

Now, as the actual method call happens at runtime, somehow at compile time(bytecode) we have to instruct JVM where to find the method or location of the method. But for some cases it is not possible to tell earlier(compile time) which method will be invoked( like in case of overriding, Polymorphism) so compiler has to defer the lookup of the method until runtime, so there are different types of opcodes are used by compiler to tell JVM what to do in runtime.
At runtime, JVM maintains a runtime table called vtable where each entry represents the precise location of the method. The help of this vtable, JVM actually dispatches the call to an actual method.
opcodes
In bytecode, java uses 4 opcodes still java6 but in java7 there is a new opcode introduced called invokedynamic, I will write a separate article on invokedynamic opcode but in this article, we will discuss the other opcodes.
invokestatic: invokestatic opcode is used at compile time to dispatch static methods.
invokevirtual: invokevirtual used to dispatch instance methods.
invokespecial: invokespecial is used to dispatch special methods like constructor or super or for the private method.
invokeinterface: invokeinterface is used to dispatch a method call via an interface.
Now, we will write a java example and try to see the bytecode representation of that example.

package com.example.methodcall;

import java.util.ArrayList;
import java.util.List;

public class MethodCall {

public void addCity() {
List<String> city = new ArrayList<String>();
city.add("Kolkata");

}


public void addState() {
ArrayList<String> state = new ArrayList<String>();
state.add("WestBengal");

}

public static void main(String[] args) {
MethodCall target = new MethodCall();
target.addCity();
target.addState();
}

}




Now I want to see the bytecode representation of the above java program so I will run the following command 

javap -c MethodCall.class 


Bytecode will look like following

 public class com.example.methodcall.MethodCall {
  public com.example.methodcall.MethodCall();
    Code:
       0: aload_0
       1: invokespecial #8                  // Method java/lang/Object."<init>":()V
       4: return

  public void addCity();
    Code:
       0: new           #15                 // class java/util/ArrayList
       3: dup
       4: invokespecial #17                 // Method java/util/ArrayList."<init>":()V
       7: astore_1
       8: aload_1
       9: ldc           #18                 // String Kolkata
      11: invokeinterface #20,  2           // InterfaceMethod java/util/List.add:(Ljava/lang/Object;)Z
      16: pop
      17: return

  public void addState();
    Code:
       0: new           #15                 // class java/util/ArrayList
       3: dup
       4: invokespecial #17                 // Method java/util/ArrayList."<init>":()V
       7: astore_1
       8: aload_1
       9: ldc           #31                 // String WestBengal
      11: invokevirtual #33                 // Method java/util/ArrayList.add:(Ljava/lang/Object;)Z
      14: pop
      15: return

  public static void main(java.lang.String[]);
    Code:
       0: new           #1                  // class com/example/methodcall/MethodCall
       3: dup
       4: invokespecial #39                 // Method "<init>":()V
       7: astore_1
       8: aload_1
       9: invokevirtual #40                 // Method addCity:()V
      12: aload_1
      13: invokevirtual #42                 // Method addState:()V
      16: return
}


Deep dive into the bytecode Representation :
In the above bytecode representation, except invokestatic all opcodes has been used.
If you noticed the bytecode minutely you can explore that for each method a section is entitled and each java line converted to a command. Let go through each method section

com.example.methodcall.MethodCall(): This is the constructor of MethodCall class, here you can find an invokespecial call because this opcode is used for calling a special method like constructor or super etc. if you pay attention to the commented line beside the invokespecial call you will find the method details
// Method java/lang/Object."<init>":() V: This says constructor can be found in java.lang.object which is detonated by a special symbol <init> and it takes nothing as an argument

 public void addCity() In this section bytecode use invokeinterface opcode for  the line
List<String> city = new ArrayList<String>();
city.add("Kolkata");
and it is commented
as

// InterfaceMethod java/util/List.add:(Ljava/lang/Object;)Z which means add is an interface method  which is in java.util.List and it takes Object as an input argument.

Here invokeInterface opcode is used because, as we did the polymorphic assignment so at the compile time there is no way to know where is the actual add method implementation, so compiler has to put such opcode which will instruct JVM to dispatch the call to exact method from Vtable at runtime, so method resolution happens at runtime.

 public void addState():
In this section bytecode use invokevirtual opcode for  the line
ArrayList<String> state = new ArrayList<String>();
state.add("WestBengal");
and it is commented as

// Method java/util/ArrayList.add:(Ljava/lang/Object;)Z which means add can be found in java.util.ArrayList and it takes Object as an input argument.

There, is very subtle difference in coding -- we use ArrayList instead of List so it is not a polymorphic assignment so it creates a huge difference in bytecode now bytecode knows the exact class where to find the add method at compile-time but still call will be dispatched in runtime as if some other class can extend ArrayList. But it uses Invokevirtual opcode which is used for calling an instance method.

public static void main(java.lang.String[]): The last section is entitled to the main method where we call two instance methods addCity and addState so it uses invokevirtual opcode.

Conclusion : In this article we have a fair bit of an idea how method call is happened using different opcodes, But in Java7 an important opcode has been added that is invokeDynamic, which opens the door to allow dynamic type language in JVM, so other languages which run on top of JVM uses this invokeDynamic opcode to make them dynamic language certain extent also Lambda Expression in Java8 uses the invokedynamic opcode, In my next tutorial I will give a detailed overview on -- invokeDynamic opcode.


Java8: Oogways final advise on Optional

Master Oogways enlighten us about Optional in his previous two sessions, and Now he set the stage for the final advise on Optional and believe me that is the most searchable questions on the internet about Optional, also many peoples mailed in OpenJDK forum regarding "Why Optional is not Serializable?" , Oogways promises to PO that he will try to give the best possible answers and we all keen to know the answer from Oogways as he is the wisest master in the world.
Let see the conversation happened beet ween PO an Oogways.

PO: Master I try to figure out why Optional is not serializable also searched on the internet but not found any compelling answers can you please enlighten me?
Master gives a calm look to Po and start his speech
Oogways: Before starting why Optional is not serializable I want to talk about an upcoming project in jdk10 which is Project Valhalla.
Project Valhalla: The slogan of project Valhalla i“Codes like a class, works like an int!” -- so, project Valhalla try to create a new user-defined datatype which can be treated as class i.e you can have data and behaviours tied into this but when that datatype runs on JVM it will act as primitives , i.e this datatype not stored in heap memory unlike Object and it does not have reference it will act as primitives like int, float -- it will stores in the stack memory. So obvious question comes
why so?
what are the benefits of that new datatype?
To answer this question we have to know a concept of ValueObject-- In Java sometimes we design some Objects which just carry value do nothing else-- act on the value, or preserve the state, this Object does not have any identity think about Integer what it does it wraps a primitive int value and why it is created?-- Just to support
java collection framework, It just a Wrapper Object, think about the ValueObject pattern in JEE or a Money Object(Value and currency), these are  used to pass values so conceptually these objects are not a proper object by definition, those objects are immutable in nature , and two objects are equals if their values are equal no identity check and obviously does not preserve a state which is the crux of an Object it acts as primitives more than Object so we called the Value Object, Now we all know there is cost associate to create Object, Even to make an Integer from int , it has to create a header(8-16 byte) and reference(4-8 bytes) in case of a reachable object, so instead of 4 bytes it takes 4+8+4=16 bytes minimum. also in case of the array of Integers, it has to traverse through pointers to fetch value as Integer array stores the references, not the value but in case of primitives it will store value locally in the stack so traversing is very fast, So the concept Project Valhalla it treats value type as primitives it gives us to create user-defined primitives.
Now I will tell you something about serialization.
Serialization: Serialization is a technique by which one can store the Object state in form of the stream so in future, we can reconstruct the state of the Object in same or another JVM from the stream. To achieve serialization is not a problem but Serialization has a huge maintenance cost, Because when we make an Object serializable we have to take care of the following
Back and forward JVM support: The Key feature of a serialization is it would be supported across any versions of Java including current and future release, so if an Object serializes in Java8 should be deserialized in java1.4 and vice versa so it supports backward and forward compatibility.So it has huge maintenance cost as various scenarios can occur where it may be broken, think about a situation if you want to add a new HAS-A relationship in your serializable Object and that class has been introduced in upper version of JDK how it would be deserialized in lower JVM as lower JVM does not know about that class.  Also in case deserializing it needs back-references to reconstruct the graph and need the Object reference. So Serialization hindering further modification of the Object, so maintaining Serialization is not a cup of tea.

Now, I come to the point why the Optional is not serializable, PO, If you look about the Optional it just wraps a value so it is a boxed class, If two Optional value is same we can say those are equals, so it treats like a Value type even in JavaDoc of Optional says
"This is a value-based class; use of identity-sensitive operations (including reference equality (==), identity hash code, or synchronization) on instances of Optional may have unpredictable results and should be avoided."
so it is clear Optional is a Value type and they know the Project Valhalla is in roadmap so obviously, they want to change Optional to a Value type (Identityless) when Project Valhalla onboard. And if they make them Serializable, now it needs to reside in heap memory and to deserialize the same need a reference but Project  Valhalla remove them from heap and stores in the stack so the question is how you deal with the same. there are lots of If and buts so they don't confident enough to make it serializable.
Another thing is the motto of Serialization is to reconstruct the state but Optional is a Valuetype and stateless so why they make it serializable it does not meet the criteria of serialization. Those are the reasons to not makeOptional serializable.
PO : I understood the Objective of not making Optional serializable but if they make it value type and make it serializable both by following technique, Suppose they make Optional as value type so it does not have any reference, but in the case of de-serialization  we can box them in an Object and make it present on heap memory, so it will be dubious nature, by default it is a value type but in case of deserialization it will act as an Object, by this we can solve the problem.
Oogways: PO, I like your thought, it proves that why I choose you as a Dragon Warrior. Really it is a compelling idea but think Optional is a Value type and in case of deserialization you want to make footprint in heap at that point it breaks the purpose of value type, To support serialization we unnecessary breaks the principle of project Valhalla, and bring back the problem which project Valhalla tries to solve, Optional is the category of Value Object to make it dubious nature to just support serialization is just a lame idea to me, To carrying the cost of serialization should have a compelling use cases, I don't see anything comelling in case of Optional.
PO: I understood master, Thanks for giving me a new direction I will learn more about project Valhalla and value type.
Oogway: Well, I am planning to give a session on Project Valhalla, you can attend that session.
PO: yes Master, I will and I bring my friend Shamik with me.
Oogways nodded his head.

Java8: Oogways more advice on Optional.

Oogway's previous talk clears the confusions about Why Optional is added on java8? But PO is a Dragon warrior he is the finest Java warrior so he wants more, He wants to know when is the right time to use Optional. What are the best practices so he again went to Oogways and I am very lucky PO called me to take the note of the talk.
Here is the Conversation.

PO: Master Oogway, I completely understood why we use Optional, the crux of the Optional is -- it gives the caller a hint that output may not be available so, design your code accordingly. So it is a Conceptual improvement which force caller to tackle maybe scenario and the outcome -- less null pointer exception. But How to use it efficiently.

Oogways: PO, Listen carefully Optional is created to check a return value is present or not, So it is the one and the only purpose you should use Optional nothing else. Optional act as a container it wraps the return value and then applies different functions on it to determine value is present or not in an advance if the value is present it can take necessary functions on it in a functional way. but whatever the case Optional must be used with method return type. Use Optional<T> as a Composition or pass it is an input is a very lame idea and should be avoided.

PO: What is the problem to pass Optional as an input suppose I have a program to search a name so if I pass the name as Optional<String> developer don't have to do the null check.
see the below program.

package com.example.optional;

import java.util.ArrayList;
import java.util.List;
import java.util.Optional;

public class OptionalTest {

private static List<String>nameList = new ArrayList<String>();

static {
nameList.add("shamik");
nameList.add("samir");
nameList.add("swastika");
}

Optional<String> findName(Optional<String> name){
return name.isPresent()?Optional.of(nameList.get(nameList.indexOf(name.get().toLowerCase()))):Optional.empty();
}

public static void main(String[] args) {
OptionalTest optionalTest = new OptionalTest();
Optional<String> searchedNameOptional = optionalTest.findName(Optional.of("Shamik"));
Optional<String> searchedNameOptionalSecond  = optionalTest.findName(Optional.ofNullable(null));
searchedNameOptional.ifPresent(System.out::println);
searchedNameOptionalSecond.ifPresent(System.out::println);
}

}

Here, I create a method called findName which takes an Optional<String >, So that developer can check if the value is present or not if present same returns an Otional<String> else returns an Empty optional, So no null check involved and passing Optional caller signal to the developer that passing parameter may be present or absent so deal accordingly. Nice way to tackle input parameters. Master then why you are telling passing an Optional in the input is a bad idea?

Oogways: PO, There is a subtle conceptual error in your thinking,
You are right Optional is used for signaling value can be present or absent, But think about who signaling to whom, here caller signaling to the author of the code, The author is the creator of the code, author is very sure about nature of method input and returns value. As he wrote the method. So here signalling is meaningless, if caller only pass the name , the author knows value may be null as default value of String is null, so he can take care of it , So here use of Optional is redundant-- here Optional means Developer of the code reminds himself passing parameter may be present or absent-- just nonsense. Optional works fine for the client of the method as the client does not know about what the method is doing inside he only knows by call findName I can get an Optional<String >, So this method may give a blank result so I need to tackle it. But the reverse perspective is just absurd. There is no need to signal developer as he controls the implementation he knows what to do with inputs, so in this case null check is better than Optional, Another thing is -- by passing an Optional you create a wrapper on a value so it takes more memory space an unnecessary complex the code violation of KISS(keep it simple stupid), Also caller has to create Optional container which is break of encapsulation. So the best way to represent your code is like that

package com.example.optional;

import java.util.ArrayList;
import java.util.List;
import java.util.Optional;

public class OptionalTest {

private static List<String>nameList = new ArrayList<String>();

static {
nameList.add("shamik");
nameList.add("samir");
nameList.add("swastika");
}

Optional<String> findName(String name){
return name !=null?Optional.of(nameList.get(nameList.indexOf(name.toLowerCase()))):Optional.empty();
}




public static void main(String[] args) {
OptionalTest optionalTest = new OptionalTest();
Optional<String> searchedNameOptional = optionalTest.findName("Shamik");
Optional<String> searchedNameOptionalSecond  = optionalTest.findName(null);
searchedNameOptional.ifPresent(System.out::println);
searchedNameOptionalSecond.ifPresent(System.out::println);
}

}

By this, Developer put the null check, Always remember Optional is not an option for replace null or apply some functional fancy methods apply on value like (filter/map) etc. The Optional is for signaling a value is present or not. So don't use it in composition or input variables for just sake for using Optional.
PO: Now, I understood the same master, Now please tell some of the operation we can do using Optional?

Oogways: Yes PO Now we are in a position where we can use some of the Optional 's utility methods.

orElse and orElseGet: Sometimes, you are sure about what would be the default value if a value is not present in that case you can use orElse on the Optional. Suppose if the name is not found we show "NA" as default value, in that case, we can change the findName method as following

package com.example.optional;

import java.util.ArrayList;
import java.util.List;
import java.util.Optional;

public class OptionalTest {

private static List<String>nameList = new ArrayList<String>();

static {
nameList.add("shamik");
nameList.add("samir");
nameList.add("swastika");
}



public String findName(String name){
return Optional.ofNullable(name).map(val->nameList.get(nameList.indexOf(val.toLowerCase()))).orElse("NA");

}





public static void main(String[] args) {
OptionalTest optionalTest = new OptionalTest();
String blankName = optionalTest.findName(null);
String name = optionalTest.findName("Shamik");
System.out.println("Name is :: " + blankName);
System.out.println("Name is :: " + name);

}

}

Here I use a map function which will check the name is in name List or not if it is not found it will return "NA" as in orElse I provide the default value as "NA".
Now, If the default value fetched from Database or from a locale based properties file then we need to write a separate method which returns the default value, in that case, we can use that separate method as a reference and pass a supplier interface in the orElseGet method. See the following example.

isPresent and ifPresent : Optional has two utility functions called isPresent and ifPresent, former returns true if a value present later takes a callback function which will apply on the value if the value is present. So when you see a code block like following

if(optional.isPresent()){
 doSomething();
}

replace the same with ifPresent

optional.ifPresent(val->doSomething())

see the below Example,

package com.example.optional;

import java.util.ArrayList;
import java.util.List;
import java.util.Optional;

public class OptionalTest {

private static List<String>nameList = new ArrayList<String>();

static {
nameList.add("shamik");
nameList.add("samir");
nameList.add("swastika");
}

Optional<String> findName(Optional<String> name){
return name.isPresent()?Optional.of(nameList.get(nameList.indexOf(name.get().toLowerCase()))):Optional.empty();
}

public static void main(String[] args) {
OptionalTest optionalTest = new OptionalTest();
Optional<String> searchedNameOptional = optionalTest.findName(Optional.of("Shamik"));
if(searchedNameOptional.isPresent()) {
System.out.println(searchedNameOptional.get());
}
searchedNameOptional.ifPresent(System.out::println);

}

}
Here, I replace isPresent with ifPresent.

flatMap: The flatmap function works same as map function, i.e change one data structure to another data structure but if the return  data structure holds an Optional it does not create a nested Optional Structure Optional<Optional<T>> it  just returns only Optiona<T>
see the example

package com.example.optional;

import java.util.ArrayList;
import java.util.List;
import java.util.Optional;

public class OptionalTest {

public void flatMapTest(String name) {
Optional<String> retName = Optional.of(name).flatMap(val->Optional.of(val));
System.out.println(retName);
}


public static void main(String[] args) {
OptionalTest optionalTest = new OptionalTest();
optionalTest.flatMapTest("Shamik");
}

}

in FlatMap fuction I delibaretly return Optional.of(val) and as flatMap returns Otional<T> it should be then Optional<Optional<String>> but it returns only
Optional<String> if we use map function we got Optional<Optional<String>>

filter: we can use filter function on Optional so we can filter the value based on some criteria

package com.example.optional;

import java.util.ArrayList;
import java.util.List;
import java.util.Optional;

public class OptionalTest {

private static List<String>nameList = new ArrayList<String>();

static {
nameList.add("shamik");
nameList.add("samir");
nameList.add("swastika");
}

Optional<String> findName(Optional<String> name){
return name.isPresent()?Optional.of(nameList.get(nameList.indexOf(name.get().toLowerCase()))):Optional.empty();
}


public static void main(String[] args) {
OptionalTest optionalTest = new OptionalTest();
Optional<String> searchedNameOptional = optionalTest.findName(Optional.of("Shamik"));
searchedNameOptional.filter(val->val.contains("am")).ifPresent(System.out::println);

}

}



PO: Master I learned all the important functions is anything still left to know about Optional?

OogWays: Yes, PO we have covered 90% but one more thing is still pending why Optional is not Serializable, I will give you the answer but I want you to think about it. Tomorrow I will give the answer if you are not able to answer it.

PO: Ok Master.

Java8: Oogways Advice on Optional

While PO reading Java 8 Optional he has few question in mind and can not understand why Optional added in Java, How it saves us from the almighty villain null pointer Exception. So he goes to master Shifu, But the Irony is Shifu does not sure about How it will save Java users from the null pointer, but it is oogways foretold so they go to Oogways for advice on Optional.
I was there when the conversation happened, I will try to depict the conversation.
PO : Master oogways, I am really confused why Optional added in Java, I can't see any benefits , and I am bit skeptical how it saves from Nullpointer exception as it is nothing but a Container/Wrapper Object which held a reference to another Object, So the reference can be null and itself Optional wrapper class can be null, so I am confused what is the use case.
Oogway: From the birth of java the greatest sin is, The invention of null, Java tries to represent something absent as a null, but absent does not mean null it means not present. Java is a strong type language but null is such a value which can be assigned to any type, so it acts as a loose type variable, act as default value of reference, It is not blessing but a curse,
Every time we don't know what would be the returned Object, developers returns a null, and caller, unfortunately, bears the load of the null check for return value unless null pointer exception blows up the flow.
Let see a small example,

package com.example.optional;

import java.util.ArrayList;
import java.util.List;

public class OptionalTest {
private static List<String> nameList = new ArrayList<String>();
static{
nameList.add("Shamik");
nameList.add("Samir");
nameList.add("Swastika");
}

public String findName(String nameToFind){
return nameList.contains(nameToFind)?nameList.get(nameList.indexOf(nameToFind)):null;
}

public static void main(String[] args) {
OptionalTest test = new OptionalTest();
String searchedName = test.findName("Mayukh");
System.out.println(" Hi , " + searchedName.toUpperCase());
}


}
Here, The API developer exposes a method called findName, and silently return a null if the name is not found.  The caller of the API trust the API developer and called the method with an arbitrary name and try to print it in Uppercase. But it blows the caller's code with null pointer exception.
Whose fault is it Callers or API developer?

API Developers Perspective: When caller gives a name and that name is not found in nameList what should I do? return a blank string or return a String "NA" but if the caller says that they don't want "NA" or blank String, or One caller OK with the blank or "NA" String  and Other wants to handle the scenario by itself, so  as a developer when I am not sure what to return I always returns null. Yes, I am a good developer I can handle all the boundary cases.
Callers Perspective: findName is an API method I only command it to give me a search result "Tell Don't Ask", It is API method responsibility to handle boundary cases or cases if the searchable name is not found, It should give me a dummy implementation if the searchable name not found, like "NA". So I do not bother to check the return value, It is not my responsibility, so he skips the null check and programs blown up.

Really who's fault it is API developer or Caller?
Caller and Developer are right in their perspective, the confusion grows because API developer can't signal the caller, The return value may be present or absent. Null means no reference !!! (But How caller will  Understand current retuned reference not point to any Object instance?), so always it is caller responsibility to do a null check even if the value is present. null is not so expressive, to indicate caller a value is present or absent as it can be easily assigned to return value silently. So Optional step in here it tries to say value may be present or may be absent so caller it is your duty to check value is present or absent whereas null returns silently without informing the nature of the value.
Optional is acting as a Container, It holds the reference and provides some method to test the reference inside is present or absent.
Let see How we can return Optional with above example.

package com.example.optional;

import java.util.ArrayList;
import java.util.List;
import java.util.Optional;

public class OptionalTest {
private static List<String> nameList = new ArrayList<String>();
static{
nameList.add("Shamik");
nameList.add("Samir");
nameList.add("Swastika");
}

public Optional<String> findName(String nameToFind){
return nameList.contains(nameToFind)?Optional.of(nameList.get(nameList.indexOf(nameToFind))):Optional.empty();
}

public static void main(String[] args) {
OptionalTest test = new OptionalTest();
Optional<String> searchedName = test.findName("Mayukh");
if(searchedName.isPresent()){
System.out.println("HI,"+ searchedName.get());
}else{
System.out.println("Name not found");
}
}

}




Here, I use Optional.of() method which creates a new Optional object with the value passed in the 'of' method as a parameter, and if the value is not found in nameList I return and Empty optional which does not contain anything, don't say it contains a null, Empty represents the Value is absent, In this way we say to the caller, my method may gives you a result which does not contain anything so prepare for it.
PO : Master Oogways I understood your point but still I have  confusions, here caller has to check if(Optional.ispresent()) to get a value which is present but if caller forget to do this check, It gives you a NoSuchElement Exception, So what is the value to use optional -- seems it changed the null pointer exception to NoSuchElement Exception, nothing else, I don't see a point here to use Optional, It can't shield us from Null Pointer Exception.
Oogway: Ha Ha, That's why I choose you as a Dragon Warrior, You are so intelligent and to the point, You learned new things very quickly. Let me share you some thoughts,
Null is Omnipresent no one can never ever remove or defeat it completely, What we can do is protect ourselves from null, educate our Java villagers from the rage of nulls.
Optional is like a warning bell in the city, rather than nulls come silently, By Optional API developers always warned  Java Villagers/ or callers, Null can be entering your village at any point of time.
So my point is, till now only efficient villagers or callers understand when to use, not null check to escape from Null pointers rage but still in a complex call or lots of chained calls they fall in the null pointers trap. Optional tries to fix that it gives remind about absent values. Absent values can come so  protect your call from that or rather always use Optional.ispresent() check with Optional.get.
Optional does not remove the null pointer exception it warns caller to handle the absent scenarios.
It is not a Technical improvement rather it is a Conceptual improvement added into Java8 which educate java programmers to handle a scenario where value can be present or absent.

PO: I realize this Master Now tell me the Best practices of Optional.
Oogway: I am too late for today, Please come Tomorrow I will discuss best practices.

Conclusion: Oogway taught us a valuable lesson Optional is not a replacement for the null check rather it tries to say caller about the nature of the returns value and implicitly reminds caller to handle the absent cases. If Caller forgets to do that it will end up with the NoSuchElementException, which is a clear evidence you are trying to do an operation where the element/value is not present. It is a big conceptual improvement, it encourages caller to handle absent cases, unlike null return value where the caller has a confusion API developer already tackle the null value or I have to do this?