Building Software Architecture is Fun but not a Game.
Today I want to write about a my views and opinions about building a Data Architecture and challenges an Architect faces , choices he or she has to make in ‘architecting’ them;
I will talk about ‘architecting’ in specific to real-time data pipeline and I will not like to talk about any architecture styles like Lambda or Kappa or XYZ.
In my personal opinion and experience these seems to be a marketing terms and don’t offer a complete solution and can be best used to understand their cons and then every architect should find it’s own unique architecture let’s call it Z Architecture;
“Z” signifying last and only option an Architect will have to invent considering ground realities in mind.
In my last 20+ years of experience building data systems in networking and web domain I have encountered several challenges and hardest were not necessarily technical.
About non-technical challenges one will have to follow and subscribe for email notification to my Medium blogs as they would take time and motivation to ponder and reproduce here without revealing or hurting the egos of affected parties.
So. here I enumerate a gist of challenges I face and how I try to overcome them to the best of my abilities.
C. Collection System
A stable and ‘scale-able’ collection system; which is most important as that’s where data incarnates, One should try to solve as many problems as possible at this layer(quality, security , privacy, scale , reliability and more).
Any problem which is left dissonant at this layer will become increasingly onerous or sometimes implausible at later stages of data pipeline.
Solving problems at this layer is relatively easier and more cost effective. Be it data quality, data loss, data security and even data governance.
I. In-flight Data Processing
This layer is most tricky to architect and probably second most important and the final chance where all the problems remain unsolved must be solved.
But it’s race against time , we can’t and shouldn’t over complicate it, more we spend time here more the latency to end user to use this data for making data driven decisions in real time.
Most of the architectural creativity, patents in my career has come from this layer. They may not be a path breaking patents which are capable of solving world’s hunger.
But I am yet to come across a technology, idea or patent or person who claims or capable of doing that.
S. Storage
If we got all whilom right this layer can still be our Achilles Heel as an Architect.
There are several choices but none are valid options but we still need to choose an option and live with that and after making this choice.
If we don’t architect well the system is destined to start bleeding from day one and destined for a slow and painful demise.
And one may have to start from beginning to build a new system all together which will be expensive and will add no real value or offer anything new to the application, team, organisation or end users building or using the application.
B. Batching System
While one is building real-time data pipeline never forget your old and trusted friend i.e; Batching system even if one tries hard will not able to perfect any of these layers.
We must preserve original and clean data and replay them slowly every day
This strategy helps to detect and heal the system; it’s like a tonic which keeps the data pipeline healthy and doesn’t let it get tired.
And if the data pipeline gets tired business decisions can not remain healthy. So it’s pertinent to rekindle this friendship and employ an ‘observability’ to real data pipeline and correct them in real-time as much as possible and for those difficult cases use a batching system which in my experience just works fine for non mission critical use cases.
T. Team
If one is able to solve all this and didn’t consider the team while ‘architecting’, there will be no one to understand, maintain or use this system; A team internally which will do this day to day;
We as an Architect can’t do all alone, We May think we are God which we are not;
Yes we can consider ourselves smart if we ever able to build with all this till then; Team will do the real hard work on the ground; So we need to build for Team too.
And for that we need to involve each and every person for whom we are building the architecture; Team will maintain and use this day to day;
It’s important to ask right questions at right time with everyone involved with the system without delay and try to make it part of software avoiding anything manual to make it a ‘scale-able’ Team Architecture.
I have seen and done all but never able to make it right at first instance but endeavour is still on and someday I will build a system which I will be proud of as an Architect.
Till then I will keep reading architectural blogs and avoid reading architectural books which I could never go beyond few pages. But some are really good and those we should and must read.
I am curious to know from fellow software Architectural community their views and any counter arguments about what I have written because I know it’s a very opinionated article which in my view is a necessary trait of a good architect to have opinion and ability to convince and take it to finish line.
In-Short;
Architect with passion with earphones on.Will suggest this melodious song from John Denver; It really helps in building good Architectural solutions.
After writing this and pondering later on I decided to write this epilogue and I have come to conclusion for now and i.e;
Building Software Architecture is both Fun and Game as long as we enjoy the fun while going through this journey and play the game fairly.
Fair Game: Helps you to know what you might have to go through to play the game fairly; Not an easy Job at All!! No one should able to break you; You as an architect have to be strong to not have a breaking point.
And One More Thing;
It amuses me when;
People managers start managing technology;
Technology folks start managing people;
Both can’t go very far;
Ultimately Team and Architecture will suffer.
Which is Sad!!
About the Me:
I’m a Computer Science Engineer; I love to and attempt to write about; Humans and Philosophy, Poets and Poems, Artists and Arts and Humans and Computers; all together, all intermingled, all united as an unison like a Beethoven Symphony.
If you enjoyed this blog post, please clap 👏(as many times as you like) and follow Here and LinkedIn if you can.
My More Works on:
LinkedIn: https://www.linkedin.com/in/santoshakhilesh
GitHub: https://github.com/akhileshsantosh
Disclaimer:
This is a personal account. The opinions expressed here represent my own and not those of my employer — past, present or future.