Great article and awesome to see you guys on Substack. I ended up implementing Data Contracts at my robotics start up role, leveraging GCP. It was a great/challenging experience.
To your point, it required huge amounts of socio-technical work. Namely breaking through SWE's skepticism on why we needed it. Thankfully I had 3 schema drift nightmares I could put on paper for them.
I just read the article AND IT'S SO GOOD! I wish I had known about this earlier; I would have loved to feature it as a case study in the book.
I'll connect with you on LinkedIn, but I would love to find a way to highlight this implementation. Specifically, how you are working with hardware is super interesting to me!
I love that, let's do it! The hardware aspect is what made that implementation so valuable. I honestly left out so many anecdotes about the problems it solved. I literally swear by that type of implementation and critique all others based on it.
I keep trying to bring up Data Contracts on LinkedIn to get the wheels turning for Data Engineers et al. This one yesterday finally worked with a meme LOL:
While I would love to say data contracts are bulletproof, the reality is that data contract checks sit within CI/CD workflows. It's very possible for such jobs to fail, similar to other tests in CI/CD. In addition, contracts only enforce the expectations you place; thus, this can also be a failure point similar to other testing. With that said, I argue that the value in data contracts lies not solely in expectation enforcement, but in providing a version-controlled form of change management for data that aligns stakeholders and provides a space to document expectations.
Great article and awesome to see you guys on Substack. I ended up implementing Data Contracts at my robotics start up role, leveraging GCP. It was a great/challenging experience.
To your point, it required huge amounts of socio-technical work. Namely breaking through SWE's skepticism on why we needed it. Thankfully I had 3 schema drift nightmares I could put on paper for them.
Put it all here: https://thefulldatastack.substack.com/p/proposing-and-implementing-data-contracts
I just read the article AND IT'S SO GOOD! I wish I had known about this earlier; I would have loved to feature it as a case study in the book.
I'll connect with you on LinkedIn, but I would love to find a way to highlight this implementation. Specifically, how you are working with hardware is super interesting to me!
I love that, let's do it! The hardware aspect is what made that implementation so valuable. I honestly left out so many anecdotes about the problems it solved. I literally swear by that type of implementation and critique all others based on it.
I keep trying to bring up Data Contracts on LinkedIn to get the wheels turning for Data Engineers et al. This one yesterday finally worked with a meme LOL:
https://www.linkedin.com/feed/update/urn:li:activity:7381732154483019778/
Sounds like data contracts can never fail, they can only be failed?
While I would love to say data contracts are bulletproof, the reality is that data contract checks sit within CI/CD workflows. It's very possible for such jobs to fail, similar to other tests in CI/CD. In addition, contracts only enforce the expectations you place; thus, this can also be a failure point similar to other testing. With that said, I argue that the value in data contracts lies not solely in expectation enforcement, but in providing a version-controlled form of change management for data that aligns stakeholders and provides a space to document expectations.