Building Software Architecture is Fun but not a Game.

Image Credit: Dasun Hegoda

I will talk about ‘architecting’ in specific to real-time data pipeline and I will not like to talk about any architecture styles like Lambda or Kappa or XYZ.

In my personal opinion and experience these seems to be a marketing terms and don’t offer a complete solution and can be best used to understand their cons and then every architect should find it’s own unique architecture let’s call it Z Architecture;

In my last 20+ years of experience building data systems in networking and web domain I have encountered several challenges and hardest were not necessarily technical.

Image Courtesy: vidlsolutions.mykajbi
Image Credit:

C. Collection System

A stable and ‘scale-able’ collection system; which is most important as that’s where data incarnates, One should try to solve as many problems as possible at this layer(quality, security , privacy, scale , reliability and more).

I. In-flight Data Processing

Image Source:

S. Storage


If we don’t architect well the system is destined to start bleeding from day one and destined for a slow and painful demise.

And one may have to start from beginning to build a new system all together which will be expensive and will add no real value or offer anything new to the application, team, organisation or end users building or using the application.

B. Batching System

Image Credit: knowyourphrase

This strategy helps to detect and heal the system; it’s like a tonic which keeps the data pipeline healthy and doesn’t let it get tired.

And if the data pipeline gets tired business decisions can not remain healthy. So it’s pertinent to rekindle this friendship and employ an ‘observability’ to real data pipeline and correct them in real-time as much as possible and for those difficult cases use a batching system which in my experience just works fine for non mission critical use cases.

T. Team

Image Credit:

Yes we can consider ourselves smart if we ever able to build with all this till then; Team will do the real hard work on the ground; So we need to build for Team too.

And for that we need to involve each and every person for whom we are building the architecture; Team will maintain and use this day to day;

I have seen and done all but never able to make it right at first instance but endeavour is still on and someday I will build a system which I will be proud of as an Architect.

Till then I will keep reading architectural blogs and avoid reading architectural books which I could never go beyond few pages. But some are really good and those we should and must read.


Architect with passion with earphones on.Will suggest this melodious song from John Denver; It really helps in building good Architectural solutions.

Image Credit: azquotes

Building Software Architecture is both Fun and Game as long as we enjoy the fun while going through this journey and play the game fairly.

Image Credit:
Image Credit: TPSG

It amuses me when;

People managers start managing technology;

Technology folks start managing people;

Both can’t go very far;

Ultimately Team and Architecture will suffer.

Which is Sad!!

About the Me:

If you enjoyed this blog post, please clap 👏(as many times as you like) and follow Here and LinkedIn if you can.

Photo by Carl Heyerdahl on Unsplash



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store


I love to write about Philosophy,Humanity,Music, Poems and Software Architecture.