Building Software Architecture is Fun but not a Game.

Image Credit: Dasun Hegoda

I will talk about ‘architecting’ in specific to real-time data pipeline and I will not like to talk about any architecture styles like Lambda or Kappa or XYZ.

In my last 20+ years of experience building data systems in networking and web domain I have encountered several challenges and hardest were not necessarily technical.

Image Courtesy: vidlsolutions.mykajbi
Image Credit:

C. Collection System

Any problem which is left dissonant at this layer will become increasingly onerous or sometimes implausible at later stages of data pipeline.

I. In-flight Data Processing

Image Source:

But it’s race against time , we can’t and shouldn’t over complicate it, more we spend time here more the latency to end user to use this data for making data driven decisions in real time.

S. Storage


There are several choices but none are valid options but we still need to choose an option and live with that and after making this choice.

If we don’t architect well the system is destined to start bleeding from day one and destined for a slow and painful demise.

B. Batching System

Image Credit: knowyourphrase

We must preserve original and clean data and replay them slowly every day

This strategy helps to detect and heal the system; it’s like a tonic which keeps the data pipeline healthy and doesn’t let it get tired.

T. Team

Image Credit:

We as an Architect can’t do all alone, We May think we are God which we are not;

Yes we can consider ourselves smart if we ever able to build with all this till then; Team will do the real hard work on the ground; So we need to build for Team too.

I have seen and done all but never able to make it right at first instance but endeavour is still on and someday I will build a system which I will be proud of as an Architect.

I am curious to know from fellow software Architectural community their views and any counter arguments about what I have written because I know it’s a very opinionated article which in my view is a necessary trait of a good architect to have opinion and ability to convince and take it to finish line.


Architect with passion with earphones on.Will suggest this melodious song from John Denver; It really helps in building good Architectural solutions.

Image Credit: azquotes

Building Software Architecture is both Fun and Game as long as we enjoy the fun while going through this journey and play the game fairly.

Image Credit:

Fair Game: Helps you to know what you might have to go through to play the game fairly; Not an easy Job at All!! No one should able to break you; You as an architect have to be strong to not have a breaking point.

And One More Thing;

Image Credit: TPSG

It amuses me when;

People managers start managing technology;

Technology folks start managing people;

Both can’t go very far;

Ultimately Team and Architecture will suffer.

Which is Sad!!

I’m a Computer Science Engineer; I love to and attempt to write about; Humans and Philosophy, Poets and Poems, Artists and Arts and Humans and Computers; all together, all intermingled, all united as an unison like a Beethoven Symphony.


This is a personal account. The opinions expressed here represent my own and not those of my employer — past, present or future.

Photo by Carl Heyerdahl on Unsplash



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store