I am curious how you see this working out in a Fortune 500 or a top 20 Pharma company where the data sources are systems like SAP or Salesforce/Veeva held and managed centrally and often with lots of offshore ops? Or in instances where you purchase data (Experian, IQVIA, etc) which itself has defects in classification and quality? In both these scenarios, the “shift left” can only go so far as the Bronze layer of the Data Lake. Enterprise Teams and Commercial data vendors do not take responsibility for data quality nor do their own tools allow for this movement. So…Can we talk about a Federated Data Team (distinct from Enterprise IT) supporting a business function in which the analytics is suffering from data quality and losing trust? How do they shift left? Or are we just back to standard DQM and SDLC processes to stem the bleeding?
This is an excellent deep dive into the state of the industry and a well-defined solution!
One key point I'd add: shifting left helps solve problems at the source, but it doesn’t guarantee that downstream teams won’t break things. Data folks don’t always follow SWE best practices, and for shift left to work, both sides need to learn from each other. If engineers adopt a data mindset, shouldn't data folks also adopt an engineering mindset?
Great article, especially the part that defines the organizational context behind an architectural decision. One nudging thought: what if shift left does not “end” with the code? As you write in the article code quality (and consequently data quality) issues could arise from unclear expectations regarding what the code should do. As a data management professional working mainly between business and IT I see further space left from coding where business sets the direction / expectations. Either way, it is in line with your arguments, e.g.: a business glossary could be a starting point to help business to formulate what they want in a language which can be understood by developers.
This is one of the most compelling articulations of why data must shift left and not just how. At Saras, we’ve learned the hard way that downstream data firefighting is a signal of upstream ownership gaps, not engineering failure. What resonated most: “data quality problems are code quality problems.” Embedding contracts, lineage, and compliance into CI/CD isn’t just about better governance it’s about creating shared accountability, speaking the language engineers already know, and scaling trust across the entire data ecosystem. Looking forward to the next post, Chad
- We solved the security review by including defence-in-depth static & dynamic vulnerabilty & pattern scanners into the pipeline and shifting responsibility to the team
- We solved the handoff to Release Management by empowering teams to deploy to production and shifting responsibility to the team
- We solved User Acceptance Testing by adding the 'ambassador user' into the mix and shifting responsibility to the team
- We solved the handoff to Acceptance Testing by including automated test suites into the pipeline and shifting responsibility to the team
- We solved the handoff to Quality Assurance by including developer-in-test (dit) and and shifting responsibility to the team
-...
We can solve for data management by providing the tools for effective (day1 AND day2) governance into pipelines & platforms, and shifting responsibilty to the team
Does moving data engineering upstream mean exploding the engineering->data->business paradigm? You mention how DevOps resulted in more cross functional teams. My experience as a consultant rarely sees the data folks talking to engineering during development but maybe engineers are once again working alongside SWEs and analysts are…somewhere? Everywhere?
Thank you for sharing, really interesting!
I am curious how you see this working out in a Fortune 500 or a top 20 Pharma company where the data sources are systems like SAP or Salesforce/Veeva held and managed centrally and often with lots of offshore ops? Or in instances where you purchase data (Experian, IQVIA, etc) which itself has defects in classification and quality? In both these scenarios, the “shift left” can only go so far as the Bronze layer of the Data Lake. Enterprise Teams and Commercial data vendors do not take responsibility for data quality nor do their own tools allow for this movement. So…Can we talk about a Federated Data Team (distinct from Enterprise IT) supporting a business function in which the analytics is suffering from data quality and losing trust? How do they shift left? Or are we just back to standard DQM and SDLC processes to stem the bleeding?
Very interesting post, thanks.
I have a question for you: what is the role, if any, for Semantic technologies like (RDF) Knowledge Graphs in implementing Shift Left?
I'm asking because KGs are becoming important in Data Centric Architectures (https://datacentricmanifesto.org/,https://www.ekgf.org/), with companies like data.world (https://data.world/), Semantic Arts (https://www.semanticarts.com/), Eccenca GmbH (https://eccenca.com/) and many more helping other companies in applying KGs to become data centric (I also have anecdotal evidence from my consulting work for big companies). It seems to me that Shift Left and Data-Centric are two ways of describing the same solution.
I recommend two books by Dave McComb in this area:
- "Software Wasteland: How the Application-Centric Mindset is Hobbling our Enterprises"
- "The Data-Centric Revolution: Restoring Sanity to Enterprise Information Systems"
This is an excellent deep dive into the state of the industry and a well-defined solution!
One key point I'd add: shifting left helps solve problems at the source, but it doesn’t guarantee that downstream teams won’t break things. Data folks don’t always follow SWE best practices, and for shift left to work, both sides need to learn from each other. If engineers adopt a data mindset, shouldn't data folks also adopt an engineering mindset?
Great article, especially the part that defines the organizational context behind an architectural decision. One nudging thought: what if shift left does not “end” with the code? As you write in the article code quality (and consequently data quality) issues could arise from unclear expectations regarding what the code should do. As a data management professional working mainly between business and IT I see further space left from coding where business sets the direction / expectations. Either way, it is in line with your arguments, e.g.: a business glossary could be a starting point to help business to formulate what they want in a language which can be understood by developers.
This is one of the most compelling articulations of why data must shift left and not just how. At Saras, we’ve learned the hard way that downstream data firefighting is a signal of upstream ownership gaps, not engineering failure. What resonated most: “data quality problems are code quality problems.” Embedding contracts, lineage, and compliance into CI/CD isn’t just about better governance it’s about creating shared accountability, speaking the language engineers already know, and scaling trust across the entire data ecosystem. Looking forward to the next post, Chad
- We solved the security review by including defence-in-depth static & dynamic vulnerabilty & pattern scanners into the pipeline and shifting responsibility to the team
- We solved the handoff to Release Management by empowering teams to deploy to production and shifting responsibility to the team
- We solved User Acceptance Testing by adding the 'ambassador user' into the mix and shifting responsibility to the team
- We solved the handoff to Acceptance Testing by including automated test suites into the pipeline and shifting responsibility to the team
- We solved the handoff to Quality Assurance by including developer-in-test (dit) and and shifting responsibility to the team
-...
We can solve for data management by providing the tools for effective (day1 AND day2) governance into pipelines & platforms, and shifting responsibilty to the team
this is basically DevDataOps* (or something).
we know this works, we know it will work.
*- apologies.
#yes
Does moving data engineering upstream mean exploding the engineering->data->business paradigm? You mention how DevOps resulted in more cross functional teams. My experience as a consultant rarely sees the data folks talking to engineering during development but maybe engineers are once again working alongside SWEs and analysts are…somewhere? Everywhere?