DDD::Interchange Context and Microservices

In my previous two articles, I describe lots about Bounded context and context map, where we have learned, How to segregate the domain models using Bounded context?  Also learned, How to use context map judiciously to understand the relationship between two contexts, here relationship is a broader statement it does not only represent technical relationship it also defines the relationship between owners of the services, who is in commendable position, who act as downstream etc. Also, we learned different strategy based on the relationships like (Partnership, upstream/downstream,anti-corruption layer etc).

In this article, we will be dealing with a special case when something goes wrong while designing a Microservice. Then how that error bleeds and corrupted other services, Then we will learn how to solve that problem using Interchange context.

Take an Example, In case of our online Student registration module where Registration Module is in commendable position, as it has a partnership relationship with analytics module, Payment module has a downstream relationship with it, Batch scheduler has a downstream relationship with the same.   so it is clear Registration module is in commendable position, so the language it published every one has to follow that language, by Registration module API whatever the message it publishes batch, and payment module has to receive the same message , so I can say if the Registration module publish an R language every other module has to consume that R language, So  in this system R is the defacto language standard. You can think of this situation on the real-world basis as British people conquered the world in ancient time, so the language they speak i.e English is the defacto language standard for world, A British man and an Indian automatically communicate with English not the reverse even an Indian and a Chinese also talk to each other by English (where no British people are involved).

Now, The problem occurs If the Registration Module has some issues in Design, say Boundary is not properly defined, Message structure not properly designed or Edge cases or Exceptions are not handled properly in that case in spite of a Physical boundary(Bounded context) this problems are bleed into all dependent services, as whole context map is polluted due the error in Registration module. So Registration Module now acting as a Big ball of mud. (BBoM). As per Eric Evans, he called it a Small ball of Mud. So one thing is clear if upstream/commandable system is in danger then the whole Microservice architecture become a mess. As an Architect somehow we have to prevent the bleeding of errors from upstream Microservices/ System to downstream.

If we dig through the DDD patterns we can found ACL is doing the same (Anti-corruption layer) which sits in front of consumer Microservice/System and translate the producer /upstream message structure to Consumer Microservice/System message structure also it handles any kind of errors , So ACL protecting the consumer services from External systems.  But think about a system where two services are in a Partnership relationship

they produce and consumes each other messages both dependent on each other and other services dependent on them, in that case rather than putting ACL in front of both Partners we can use a new concept called Interchange Context.

Interchange Context, is such a technique where both partners agreeing upon the model and message structure, so there is one ubiquitous language for both partners, they publish one Message together and other systems consume that message. In this way, two partners can independently change there internal code, internal message structure but in Interchange context, they create a common terminology/Ubiquitous language which understood by this two partners. Every other service, who are dependent on these two partners can consume message from Interchange context.

Conclusion : Interchange context is more robust that Anti-corruption layer, it stops the error to be bled from upstream services , If upstream services has a direct consumers then , upstream service developer should be very careful about the API , any changes in API or errors break others , so Interchange context is kind of centralized facade where partners are created the ubiquitous language and other costumes message from interchange context, partners and dependents services all are free to modify there own data model, message structure but when they communicate to interchange context internal model should converted to Interchange context message structures.

Read more

DDD: Thinking in terms of Context Map

In my previous article, I did a detailed discussion about the Bounded Context and learn that how to tackle the complexity of a Domain, it is the best way to divide the domains into several subdomains and mapped them with different bounded contexts where each business entity/Value object has certain meaning on that context, so every stakeholder of the business like Product owner, Developers, Architect, sponsorers are understood the context and referring entity with the proper name, there should be no confusion of the naming when we discuss the terminology basis on a context. A ubiquitous language which creates a unified language among business stakeholder.

By Bounded context, we properly define a business model, create a different context based on the business domains, but always a functionality spans over multiple business entities , and those entities lie in the different Bounded contexts/domains, so it is utmost important to understand the relationship between the Bounded contexts, to architecting the business solution Context Map is a technique by which we can visualize the relationship between different contexts and Integration architect choose the best integration patterns to communicate to other contexts.

Why Context Map is so important while designing solution?

By UML diagram architect cam understand how different parts will communicate with other parts, It gives the architect a view on the communication between different contexts, that is fine but context map stepped in before UML diagram, it helps to visualize the nature of the relationship and based on the nature , architects can decide what kind of technical solution would be adopted.

The best part of Context Map visualization is, it talks about nature of the relationship, It not only tell is that relationship is upstream or downstream, Publisher or subscriber, it also tells how different team dependent on another team their motifs, their politics everything. so counting all of that now architect in a position to decide optimal solution to minimize the risk while integrating with another context.

In the era of Microservices, Context Map is the key player, because before design the holistic Microservice architecture where every team owns a Microservice, It is important to understand how one team is dependent on other, which teams are in commendable position, which team only seeks help then you can architect the solution best possible way .

Think of our Student online Tutor app, as a full grown app say it has more than fifty microservices are deployed in production , so here fifty teams are coming into play and for developing a functionality say Student registrations , multiple contexts will be affected, so we can say to implement that feature multiple teams will be involved so what would be their relationship, while designing this feature who is the pivot service whose data is most needed obviously that service is in commendable position as if that team is not ready other teams can't do anything, so all teams should align with that team , they have to sync there product backlog with that team, so here the internal politics comes to the picture , if the data of the service comes from an External team not with in the organization then the solution is way more complex as you can't force them so only way is to request them and wait for there changes, so based on these diffrent scenarios, politics context map has diffrent solutions. I will cover most important solutions here

 1. Shared Kernel : Shared kernel talks about a partnership relation where two or many teams shared common data model/ value object, it reduces the code duplication as different context use that common model, but that common model/value object is very sensitive, any changes major/minor should be agreed upon all the parties unless it would break other parties code, so more communication and synchronization needs among those teams say one team need a change in the common model but another team is not ready so the priro team has to wait for other oartner to be ready or otherhand othar partner has to change there code unnecessery although that is not heliping them, but to be in sync with other partners, diffrent shades of problems woven while maintaining the pertnership relation ship so choose that pattern  when your common model remains constant or changes once in an era.

In our example, say we developed an analytical module which analysis which courses are most chosen by the students, which students are chosen more than five courses etc, so that module works with Student model, course model so I can say our Analytics module can share Registration modules student model, and they also agreed upon any changes on student model.

Customer/ Supplier: Generally this is the common relationship between two contexts, where a context consumers or depend on data from another context, the context which produces data marked as upstream and the context which consumes data called downstream. When we visualize this relationship in terms of politics we can visualize the power distribution has many shades.

like following

Upstream as the leader: In this type of relationship, the upstream team is in a commendable position, that team does not care about the downstream team, as they producing the data and downstream team is on the mercy of that upstream team they are always changing their model based on the data structure produces by upstream.

In our Student Registration app relationship between Payment app and Notification, app is kind of upstream and downstream where Payment app decide what information in which structure they provide and Notification module consumes that data structure.

Downstream as the Leader: In some cases, the relationship is revert although upstream is produces data, it must have to follow the rule, the data structure for downstream, in this scenario downstream is in a commendable position.

say in our Studen registration system we need to submit Form 16 to government as a tax payee so our payment module has to submit form 16 data to Government exposed API but government API has certain rules and data structure for submitting form 16 data, so although government API is downstream it has total control, our Payment module should communicate with down steam in such a way so it can fulfil downstream rules.

The customer-supplier relation works best when both the parties upstream and downstream are aligned with the work both party agreed upon the interfaces and change in the structure, in case of any changes in the contract both parties will do a discussion synchronize their priority backlogs and agreed upon the changes, If one party does nor care upon another party then every time contract will be broken and it is tough to maintain a customer-supplier relationship.

Conformist: Sometimes, there is a relation between two parties in such a way that downstream team always dependent on an upstream team and they can't do a mutual agreement with upstream about requirements.  The upstream is not aligned with downstream and does not care they are free to change there published endpoint or contract any time and not taking any request from downstream. It is happens when Upstream team is an external system or under a different management hierarchy , and many downstream systems are registered with it so it can't give a priority to any downstream, rather then all downstream system must be aligned with upstream contact and data structures if upstream contracts or data structures change it ps downstream responsibility to changes accordingly.

Say, In our online Student registration app we have a free tutorial module where all students or other application can consume our free tutorials and embed them in their application, so here out free tutorial module acts as upstream and independent of any other third party app who consumes our free tutorials , we can;t give any priority to them and we don't have any contract with them if we change the contracts or data structures it is other third parties duty to change their application accordingly to consume our free tutorials. Other parties are acting as a conformist.

Anti Corruption Layer: When two system interacts if we consume the data directly from upstream we pollute our downstream system as upstream data structure leak through the downstream so if the upstream become polluted our downstream too as it imitates the upstream data while consuming. So it is a good idea while consume data from third party or from a legacy application always use a translation layer where the upstream data translate to downstream data structure before fed in to downstream in that way we can resist the data leakage from upstream, if upstream contract changes it does not pollute downstream internal system only Translation layer has to be changed in order to  adopt new data structure from upstream and convert it into downstream data structure, this technique is called Anti-corruption layer. Anti-corruption layers save the downstream system from upstream changes.

In our application Notification module can implement an ACL while consuming data from payment module so if payment module data structure changes only ACL layers affected.

Open Host : In some cases, your Domain API needs to be accessed by many other services like our Free Tutorial Publisher module, Many external or internal domains want to consume this service, so as Upstream it should be hosted as a service and maintains a protocol and service contract like REST and JSON structure so another system can consume the data.

Published Language: Often two or more system receive and send messages among themselves, in that case, a common language will be needed for the transformation of the data from one system to another like XML, JSON we call that structure as Published language.

The holistic  view of our Student online registration app in terms of context Map

 Conclusion: Context Map is a very important exercise to realize how one domain communicate with other, It gives a proper view of the organization structure, how different domains are distributed, how domain owners are dependent on each other? What is the relationship between team structure? Can they be aligned while developing a feature, based on all the parameters Integration architect can adopt a suitable integration pattern to integrate domains? Prior to designing the integration solution, always architect has to define context map to understand the relationship and structure of the teams based on that Architect can choose the best possible solution.

Read more

5 great points why you use event source solutions?

The Event Sourcing Pattern

Joe has a habit whenever he did some transaction by his Debit card he used to write them in his Personal Diary so he can track his transactions. Joe is not a technology savvy and not able to check account statements online.

At the month end, his bank SMS him his current balance but he immediately notices a discrepancy between current savings what bank shows and as per his calculation based on the Diary. He immediately calls Bank helpline and arguing about the discrepancy. Then the bank sends him an Account statement with all transactions recorded.

When he is trying to match transaction records with his diary he understood one transaction not in the Diary as that day he was so sick, he thought it to write it next day but somehow forgot.

But the question is what we can extract from this story?

If we look minutely we discover one fact that Bank always stores all the events/transactions happens on the Account. Like Account creation, credit, debit etc. and the current balance is nothing but the outcome of those transactions. So what I meant to say is that account balance is not a fixed column in a database rather than it is a derivative/outcome of the transactions/events what were applied on the Account.We called it Event Sourcing.

Now think what we generally do in software if we credit or debit any amount we just add or subtract that amount from current balance and update the account with new balance right.

So we have the current state but lost the information how that state is achieved some system uses Audit trail still it is not fully deterministic. So anytime anyone challenges the system this is not the desired system state we don’t have any solid proof other than pleaded to them that system can not be wrong. But if you maintain that history or cause of the state changed like Bank then you just give them the History and asking to check -- a solid way to store proofs.

This is a very common story may anyone of us gone through the same and then look the Account statement for doubt clearing.

Technically what is Even Sourcing?

Event Sourcing is a technique by that we store all the changes of application state as a sequence of events. we can rebuild the state anytime from those events, also can query on the events to construct the desired state.

So the two key benefits are

1.we store all the events in a sequence which enables huge opportunities.

2.The state is the derivative of events so we don’t have to maintain state in the database rather we can programmatically derive state based on the event.

Now this opens a new direction that we don’t have to persist state rather we can derive state and it bring many advantages I will talk about 5 such advantages.

1. State Rebuild :  As we stores every event applies on an application object, we can create a blank /initial application object and apply every event in the same sequence it applied will bring the same state, so anywhere any point of time we can rebuild a state from events. So systems must have a mechanism to apply event, Then you can rebuild a state if the state is blown up for some reason. One may argue if your application state derives from millions of events applied on it, so computing all events may take time and also storing all events need a big storage area. but the fact is nowadays memory are really cheap also we can have TB of in memory space so computation is also faster, alternatively, we can store snapshot i.e milestone of the state and apply event and rebuild state from latest snapshot.

2.  Temporal Query : Event sourcing is perfect for Auditors. Business analysis team always want to see the past state so they can compare the growth or loss or any valuable statistical data so they need the flexibility to query the system in all possible way to collect statistical data. So If system has a feature to build the past state by passing parameters then analyst team will be delighted and the System which maintains all the state they can easily rebuild /compute the state by the parameters provide by the analyst team say analyst want to see the Account details for 10th December 2016, by event sourcing we can fetch all events till 10 the December and apply them in sequence to build the state and return the result to analysts -- easy job isn’t it.

Add caption

3. Comparing State : Sometimes in a complex system, you need to know if events were applied in different ways what would be the outcome and how much deviation it cause from the current state say, A bank saving account interest rate is 8% previously it was 8.5. Now if the bank wants to know due to the decrease of the interest what is the actual amount bank benefits so they will replay events of 8.5 percents in all accounts and compare the state with current state to know the actual benefits although it is not very easy to implement but we can.

4. Debugging State : Suppose there is a bug in production system and we need to debug why the bug happens by event sourcing it is very easy like copy the Account in Dev environment then change the Log level to Debug and apply event one by one in the sequence and check the outcome is predicted or not ,if not then  found the Event and check how it applies to change the application state to found the defect.

5. Future Prediction :  In some Business domain it is important task to analysis what will be outcome if we take some business decision, if the outcome is successful they will take the decision,But in a naked eye it is impossible to predict the application state as different services are interlinked with each other and based on one event they can change, dependent services are subscribed to certain events when that event occurs they take action on basis of event value.  say A bank’s stock share worth is 8 INR but bank analysis team predict  within 1 month it will be increased to 12 INR and they have moreover 30K stocks are public so analysis team wants to know what will be the effects of the application state if stock worth is 12 INR so they will run some ad-hoc future events on top of current state  based on two criteria.

Taking per stock as 12 INR

Taking per stock as 8 INR

Then compare two application states to find out what are the effect of this stock value increase for each interlinked services.

Conclusion : Some systems are inherently Event sourced like Version control (GIT), Banking application, Order Tracking application etc. but we can implement the same in general system also.Using Event sourcing you can easily back and forth you application state by replaying events and state cloning into any environment is just a matter of time but the Irony is, This pattern not used broadly in industry.

Read more