Designing Highly Available Systems

Building Composite Architectures

Recently after Gartner in its recent report highlighted “Composite Architecture” or “Composable Architecture” as one of the five emerging trends in modern innovation and technology for the next 10 years. I started coming across this topic in various technical forums.

“Composability” as such is not a new topic, as we have used this frequently in object-oriented programming to achieve Polymorphism. In software architecture terms it is defined as the combination of software systems to produce a new system. In other words, it is directly connected to the goal of agility and reusability and the whole crux of it is to respond to the changing business spectrum.

Domain-Driven Design to build Composable Application

If we take a step back and go back to the way a simple application created using domain-driven design using an onion architecture. The orchestration layer plays a pivotal role in making an application composable by interacting directly with the repository or service layers.

The orchestration layer as such can either be a WebHooks API, a Data importer, API Controller, Messaging service, or a simple REST or SOAP request.

This kind of atomic structure if done properly can result in designing a system that is open to change its external integration seamlessly and also meet the changing business landscape.

Atomic Architecture

If we take the earlier example and apply it in a larger context, the below visualization depicts a circular relationship between different layers in a typical business domain.

Here the applications are interconnected in an atomic way making the organization landscape plug-in and plug-out systems in an easier way. With the advent of the native SaaS-based platforms, this type of “Composable architecture” is getting more and more noticeable.

Elements of Composable Architecture

The basic building blocks of a composable system is still around the foundation of Containerization, Microservices, Cloud, API’s, headless architecture, etc.

Conclusion

With a Composable mindset, organizations can uplift isolated business operating models and move towards a more practical loosely coupled technology landscape where systems can be plugged in and out flexibly.

This kind of model perfectly fits with organizations adopting agile ways of working or building modern omnichannel integrations with different types of native Cloud-based SaaS platforms.

This model can also be applied to bridge gaps across the entire ecosystem of legacy and modern applications including areas of a unified experience, operations, transformations, infrastructure, external and internal system integrations.

Using Level Of Connascence to Break the Monolith Application

10 Fundamental Principles one needs to ask before breaking the monolith platform

          Below are some of the key principles that need to be evaluated when one starts to break out services from a monolithic platform.

1.    Target Core Services or Fringe Services First?

          Target Functionality that doesn’t require changes to the end customer application and possibly also doesn’t need any core database migration or changes. It becomes easier for subsequent services by building CICD pipelines, required alert and monitoring systems, testing strategies, and version control.

2.     Split Schema or Code First?

          If the core services are clear then always first split out the schema and keep the services together before splitting the application code out into microservices. If the services are too coarse-grained they will be split into smaller services creating another data migration. Also, two services accessing the same database results in tight coupling between these services.

3.    Moving out Services Vertically or Horizontally?

        Moving out of Services can happen either vertically or horizontally. Try to move out a single core service at a time by first moving the database, functionality, and then the front end. This technique avoids costly and repeated data migrations and makes it easier to adjust the service granularity when needed.

4.     Building Micro or Macro or Mini services?

       

          When creating a service, first identify the core services and define clear bounded contexts. Until then, the first step is to create a macro service until the core services are clearly demarcated. Once the demarcations are clear it is easy to further split into microservices. 

5.     Outside in or Inside Out Creation of Services?

        

          The easiest way to create services is from outside-in, understanding how the various integrations need to talk to various applications. However, this leads to data inconsistencies and data integrity issues. Designing service inside-out is more time consuming but cleaner with clear defined boundaries for each service. If approached properly, this will reduce possible data integrity issues. 

6.     Where to build New functionalities?

       Target any new functionality getting created as new micro-services, target services that are business-centric. Do not add a dependency to the monolithic platform. Ensure that the new services do not call the monolithic application directly and always access it via anti-corruption layer. 

7.     Rewriting Code or Capability?

       

          When building new functionality try to rewrite capability and not the code. This may be time-consuming to build, but the monolithic platform already has a lot of redundant code. By rewriting capability it gives an opportunity to improve the granularity of the service, revisit business functionality, and maintain a clean codebase. 

8.    Incremental or Radical updates?

          Target to decouple modules or services that result in reducing traffic towards the monolithic application, this will improve the performance of the application as well as help in decommissioning of infrastructure and helping cost (licenses).

9.   Versioning Services Incrementally or Concurrently?

          Having multiple versions of the same code leads to issue concerning maintainability and cost, but until the microservices and surrounding integrations are matured, maintaining multiple versions of the service endpoint at any given time helps in reducing failure risks and less dependent on external systems.

10.   Where to build New functionalities?

          Target any new functionality getting created as new micro-services, target services that are business-centric. Do not add a dependency to the monolithic platform. Ensure that the new services do not call the monolithic application directly and always access it via anti-corruption layer. 

The myth of Sharing State when breaking large applications

One of the complex puzzles in a microservices journey is how and when to break the database. When thinking about breaking a legacy monolith application, the very first non-risky thought that comes to mind is to decompose the platform module by module as standalone microservices using multiple ORMs and hitting the same database. 

If it was an application with limited tables and modules,  would have been the simplest approach to move towards. If there is a firm partition between each microservices data with fewer dependencies, then it becomes fairly easy to adopt services and maintain one large database with several schemas.

However, legacy applications are seldom portable, and sharing data or state to all intents and purposes is convoluted. Below are some of the typical concerns that need to be evaluated building or maintaining applications with a single state.

Tight Coupling of Services

One of the key principles that Architects strive is to build a loosely coupled application that can be catered to future unknown requirements. In data terms what that essentially means is to build functionalities using new ways of persisting state without impacting the existing application or state. 

Most of the legacy applications are built and maintained for years and years and have a very tight coupling of out of box and custom modules and libraries. This results in huge state dependencies between modules. If any new requirements to either build a module to be event-driven design or build a new non-SQL database for solving certain quality attributes is no easy task and requires a complete revamp of several services. 

 Weak Cohesion

The basic principle of building microservices is the Separation of Concern, i.e. each service, or a group of services to have its own dedicated state.  

Large legacy applications generally have a large database with several schemas. Each database schema is accessed by several services, hence if any change to the logic that requires a DB change, it will impact all corresponding services. If a database table changes, all the related services will have to change and this creates huge dependencies between development teams with huge sunk cost fallacy. 

Dealing with Concurrency Issues in large applications

The last few days have been hectic dealing with concurrency issues with our monolith application during the peak traffic period.

Concurrency issues are not easy to resolve, especially when you have an application with thousands of files. The error was in the order pipeline during checkout when hundreds of custom pipelines execute in parallel. When the error occurred, all the previous transactions got revoked. 

Since the issue happened for the first time, to begin with, we just ignored the error, hoping it did not crop up again. As the traffic increased, the errors also increased simultaneously, and every error in the log pointed to a concurrent exception.

We did not have much logging, and that's when we started evaluating every table in the transaction and their relationships. We got the list of all the tables, and there were like close to 100 tables getting accessed. We decided to split the table in terms of read-only and write. Once we got the number of tables that were getting updated, we tried pinpointing the tables that had a foreign-key relationship. That further filtered the number of tables where the issue could potentially be present.

Lastly, on further analysis came across a table where locking was a possibility. Meanwhile, enabling logs gave details about concurrency errors on the same set of tables. The first thing noticed was there was no last-modified timestamp column on these tables. Then went back to the application code and added an explicit locking in the code and a check for validating the last modified timestamp.

All this took a week to resolve, and the issues made me realize how difficult it is to eradicate concurrency in systems. Years later, when I look back at this article, it will be a surprise not to have come across the same issues again.

Product Oriented Development Methodolgy

In several Organizations, the focus of project execution has recently changed from more product-specific deliveries replacing project-specific deliveries. This requires a complete organization-wide change to make every internal/external project run on AGILE ways of working.

What this essentially means is that every project needs to be viewed from an MVP approach. i.e. Most valuable product.  This requires

1. Changes in the organization and team structures

Teams are more defined based on specific goals. Teams are asked to think more in terms of product innovation and deliver specific features to evolve the product. There is no one-time delivery system and the business teams are aligned directly with these teams to incorporate faster feedbacks on the product roadmaps. Each team is attached to a dedicated product owner who helps create the specific goal required by the business teams. 

2. Changes to the ways of working.

Different stakeholders (Both IT and business) meet in every few weeks and define the product goals. Teams are more cross-functional so that they can be moved around once the specific need of the goal is achieved. From the business perspective, the money allocation to building these products is more incremental and on a quarter by quarter basis rather than estimating long term projects. If an idea is not flying it can be terminated at any point in time. Teams are more strengthened with Agile and DevOps culture. It’s a more flat structure where the individual teams mostly interact with an Agile Coach, PO, and a PMO in the hierarchy. 

3. Product Goal Definition

The product goal definition is based on the lifecycle of the idea from start to production and the idea is split into the specific weeks road map. The goals are created based on different objectives related to parameters that can be real-time customer behavior or any quantifiable metric. Each team is asked to present the product's progress in incremental demos every second week. Several of the goals are also based on the market competition and the main idea is faster velocity. 

Simple way to create automatic alerts for your application using Slack API

Simple Springboot application for Slack notification

Slack has become one of the most widely used channel for communication between teams in many organizations. It has an open source API that can be used to notify alerts and events to different channels and stakeholders.

Below is a simple spring boot application that uses JPA repository to fetch database contents and communicates to Slack via JSlack api



The spring boot application is present in the following path on github.

https://github.com/shailendrabhatt/spring-boot-slack-notification

Quick 5 Point Website Inspection for Peak Holiday Traffic

The holiday season is around the corner and with the code freezes, teams fear to risk any major changes to production to hamper the critical business time of the year.

Below are quick 5 pointer non-code checks that can help prepare and enhance the website holiday traffic and sales.

1) Examine the Infrastructure Sizing

If on-premise servers are running critical applications, recalculate the approximate capacity or how much traffic these applications can hold up.

This is based on different data points and includes the number of requests per second, average application server response times, instances and their details of CPU, cores, threads, sockets, etc. All the required information typically can be gathered from the analytics or monitoring tools.

Calculating the number of Cores?

For doing this all that is needed is to load the CPU information (lscpu) and view the information related to Threads per socket, Cores per socket and several Sockets.

Calculate the maximum load or throughput of the system?

The next step is to calculate the number of average requests to the application servers. This can again be calculated by viewing the data of the monitoring tool. The information needs to be fetched for the most peak traffic or expected peak traffic for an application or website.

Calculate the Average response times?

The next value that needs to be calculated is the average response times from the application server. This information also is available using any monitoring tool or can also be calculated by using the expected average value in seconds.

Now with the required information in place, the number of cores can be calculated using the formulae

Number of cores = Requests per second * average response time in seconds

2) Tweak the Enterprise Search Relevancy

Several Research articles show that Browse and Search pages are the most visited pages as well as the pages where most customers are lost. Several websites spend a lot of money and time on external Search engine optimization and one of the key features that are overlooked is the Relevancy Ranking of products on the website browse and search.

Below are some tips that can help boost the relevant products to the top and bury the least preferred products on the site.

Prioritization of Attributes

Attributes precedence is key for relevancy. There needs to be a clear definition of the data associated with these attributes. Special considerations have to be taken for data like Category Name, Product Name, Descriptions and other keywords, as these attributes form the basis of the Search Engine Index. Custom attributes can be created as required and added to the prioritization list.

Monitoring Search Results

Use analytics to monitor user behavior on the search and browse results. Information like most clicked products and categories, most popular searches for a product and category, error results, zero product search, the search result with auto spell correction, page behavior, etc indicates clearly how relevant results are shown to the user. These data form the basis of boosting the required results and ignoring irrelevant results.

Enterprise Search Engines

The setting of core search functionalities like Synonyms, Thesaurus, Stemming, Spell Corrections, Did you mean? play a vital role in displaying the relevant results. For sites having multiple locales update custom stemming files with the top key search words. Also, it’s very essential to keep building the dictionary as and when new data sets are added.

3) Keep a Close Vigil on the Browser Load times

Page load times matter a lot during the peak traffic time. While application health is one aspect of evaluation, the end customer load times also need to be closely monitored.

Below are a few of the areas that need to be evaluated so that the browser load times are not hampered.

Evaluate the Caching TTL

Retrieving static content over the public network is not cheap. Larger the file size larger is the bandwidth and higher cost and lower response times. Caching of static content in the browser plays a critical role in reducing those server calls resulting in faster performance and quick page loads.

Ensure HTTP cache is enabled which enables servers to direct the browser cache to serve the content for a longer duration.

If CDN is used, reevaluate the TTL for those static contents especially the images.

Avoid those Content Publishes

Every content push to the live site requires clearing of caches. Try avoiding the risks of these content pushes during the peak times and if required ensure that only the specific content cache is cleared.

Avoid those External Scripts Calls

All the third-party integrations that the website refers to especially from the browsers have to be looked at. During peak traffic hours every external script is extremely vulnerable and can bring down the browser load times and in some unfortunate cases, the website as well.

4) Tune Database Performance

Evaluate the Relational database performance to prevent those long-running queries. Below are a few of the basic tips that can help optimize the database in a quick time.

Evaluate the Database Statistics

One of the critical aspects of tuning the database is collecting and understanding the statistics. While the monitoring tools give certain information on the most time-consuming queries, the statistics collected from the database server can give detailed information about specific tables, the index distribution, details related to database sizing, etc.

Optimize Indexes

It is vital to have the right insight of indexes on tables. This is the most common issue found when dealing with RDBMS. It’s very essential to understand the fine gap between no indexes and too many indexes, Indexing strategies for inserting, reading and updating data.

Most of the modern databases give information regarding columns that are required to be indexed, their priority order, missing or incorrect indexes, etc and also provide required suggestions on improving them.

5) Network Optimization

Retrieving website over enterprise network is underestimated. Responses within the internal organization network require many round trips between the enterprise internet gateway, firewalls, load balancers, security gateways and those web and application servers placed in different data centers. Below are some suggestions that can help optimize the internal network.

Try to remove those additional firewall rules which can gain few milliseconds for every request.

Try to keep the internal network free from any excessive internal traffic.

Identify those infrastructures with a single point of failure and monitor them closely.

Look out for those Bots and Hostile IP addresses that are clogging the network bandwidth.

How to develop a quick load test for your micro service application

Ensuring the performance and scalability of the applications is crucial. Load testing is a key practice to identify bottlenecks and optimize your Spring Boot microservices. This article explains the process of developing a quick load test on Spring Boot microservice using Apache JMeter.

Load Test Setup:

Follow the below steps to set up a quick load test:

Step 1: Install JMeter Plugins (Optional)

If you haven't installed the JMeter Plugins Manager, do so by following the instructions on the official website: https://jmeter-plugins.org/wiki/PluginsManager/

Once installed, open JMeter, go to "Options," and select "Plugins Manager" to install the necessary plugins for your load test.

Step 2: Create a New JMeter Test Plan

Open JMeter and create a new Test Plan:

Right-click on the Test Plan node.

Choose "Add" > "Threads (Users)" > "Thread Group."

Set the number of threads (virtual users) and loop count according to your testing requirements.

Step 3: Add HTTP Request

Right-click on the Thread Group.

Choose "Add" > "Sampler" > "HTTP Request."

Configure the HTTP Request sampler with your microservice's endpoint details.

Step 4: Add Listeners for Monitoring

To analyze the test results, add listeners:

Right-click on the Thread Group.

Choose "Add" > "Listener."

Add listeners such as "View Results Tree," "Summary Report," and "Response Times Over Time."

Step 5: Configure Timers (Optional)

To simulate realistic user behavior, add timers:

Right-click on the Thread Group.

Choose "Add" > "Timer."

Configure timers like "Constant Timer" or "Gaussian Random Timer" to introduce delays between requests.

Step 6: Run the Load Test

Save your test plan and run the load test:

Click on the "Run" menu and choose "Start" or use the green "Play" button.

Monitor the results in real-time using the listeners you added. Pay attention to response times, throughput, and error rates.

Step 7: Analyze and Optimize

After the test completes, analyze the results to identify performance bottlenecks. Optimize your microservice accordingly and iterate the load testing process.

Conclusion:

Load testing with JMeter is a powerful way to ensure the performance and scalability of any Spring Boot microservices. By following these steps, you can quickly set up a load test, identify potential issues, and fine-tune your microservices for optimal performance in a production environment. Remember to regularly conduct load tests to catch performance issues early in the development lifecycle.

Changing Organization Perception in a Microsevices journey - Series Part 6

A very core principle of building microservices is enabling teams to deliver at a constant incremental pace and not in a big bang or spasmodic approach. Teams must be trained and prepared for cogent disruption in adopting agile methodologies processes.

Microservices requires teams to follow the core principle of agile in working at a constant pace, which in turn enables teams to deliver at a constant pace.

DevOps and DataOps culture needs to be embedded into teams. Teams must be enabled to have greater control over their environment and make multiple releases into production with a fail fast and fail forward approach.

Don’t get into a situation where certain teams are more agile than others. This can lead to a lot of slow down, especially with cross-team communication and integration.

Containers should be a standard part of any agile infrastructure, especially when working on legacy platforms that require specific infrastructure and length installation. Build service containerization to take advantage of the flexibility, speed of deployment, configurability, and automation.

Do spend resources and time on the monolith and its improvement. Seldom, teams just start concentrating on building new-age technology solutions and hardly spend time understanding the legacy system. Without understanding the monolith, it’s hysterical to even attempt breaking it.

Building Microservices by decreasing Entropy and increasing Negentropy - Series Part 5

Microservice’s journey is all about gradually overhaul, every time you make a change you need to keep the system in a better state or the state that it was before.

Moving out of Services can happen either Vertically or Horizontally. Clearly nail down those horizontal services that need to be common overall. Try to identify those vertical services and the best way typically are to migrate them by first moving the data, then the business logic and later the front end.

Always try to target the first step as is to create a macro service until the core services are demarcated. Once the demarcations are clear it is easy to further split into microservices.

In the beginning, the teams have less operational maturity, during this phase minimize the number of microservices and reduce the cognitive load.

The simplest of services, to begin with, in my experience are read-only applications. Especially, the ones that are business-centric and that change very often. This allowed us getting the business team’s confidence that the teams can move faster and deliver those features rapidly.

The team that I was part of initially were depending a lot on the monolith application. The services were deployed multiple times a day, whereas the monolith application was deployed once a week. The services should never wait for the monolith to be deployed.

Also, every time changes went into the monolith, it was always ensured that there were feature flags. This gave the developers the leeway to revert and test changes if in case hell broke loose.

Do not add a dependency to the monolithic platform. Ensure that the new services do not call the monolithic application directly and always access it via the anti-corruption layer.

Concluding Thoughts

A microservice journey is complex and seldom have organizations been successful. And if you have gone ahead and started that journey, do take intermittent checks on where you stand and correlate to where you started from.

• If after building services you are in a situation where all developers congregate to fix production issues
• if teams require several developers and take umpteen number of days to fix issues
• If your applications have several hours of downtime
• If your services cannot be tested as a single entity
• If the teams fear to make changes to the code when adding new features
• If you are reverting code and releases instead of failing fast or failing forward
• If you are building services that access the same database
• If the services and functionality is spread across multiple services and teams
• If your applications take multiple teams and several people to deploy changes to production
• If there are too many services depending on each other
• If your teams are still writing and performing manual tests
• If you are building services with hundred of Classes having hundreds of lines of code
• If you have several services that have not been modified for several months

Maybe, you have ended up building another monolith.

Dividing and Conquering Microservices using Concern Based Modularization - Series Part 4

Modularization is another indispensable prerequisite, to begin with when designing microservices. Without modules defined, I have seen teams building services in a wild goose chase fulfilling the anti-patterns of complex data interoperability across distributed systems, tightly coupling of services by maximizing dependencies, minimal composability and abstraction and the list goes on….

Uncle Bob once famously said

“Gather together the things that change for the same reasons. Separate those things that change for different reasons”.

It’s essentially an abstraction of concerns of business capabilities by understanding and defining the exact core services or modules of an application landscape. InIt’she same basic concept followed by designing any monolithic application.

Where teams fail to understand is that with an improper composition of core components or modules, all you are doing is shifting complexities to more gray areas. This is a whole reason why in the first place the monolithic application was designed to be a haphazardly structured application or otherwise famously defined as “Big Ball of Mud”.

“Deciphering the concern based modularization of an enterprise is an essential requisite to amalgamate the intricacies of the microservices system.”

In one of the microservices journeys, I was part of a team that wanted to jump-start creating services and in parallel define their core modules. In doing so they came across a situation where they had the same core source information in multiple systems and this resulted in a pretty complex data interoperability across multiple services. Fixing things at this point is tedious impacting several applications.

Also, another issue seen is the over creating of core services which is a mess. It will lead to several unnecessary layers. It just adds complications to the application landscape where every service is depending on the other without any realm of responsibility.

Get the core and sub-core concern components that are baselined at the earliest.

Domain-driven design (DDD) needs to be adopted to help choose domain boundaries or business context.

Applications need to be divided and conquered to identify the organic sized chunks of these components.

Core services by definition are services that mainly focus on persistence and are pure data-centric. Each isolated core services will be the future master of information and the discrete source of truth.

Once the core services are nailed down, detailing out the Peripheral and Composite services becomes much easier.