Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
021 - Demystifying Data Contracts with Andrew Jones
160 Plays1 year ago
Data Contracts have gained a huge amount of traction over recent years as Data teams strive to increase the quality of their offering.
This week on Stacked Data podcast I had the pleasure of taking a deep dive into the world of data contracts with none other than Andrew Jones, the Principal Engineer at GoCardless and the pioneer behind this transformative concept.
If you haven’t heard of Data Contracts they bring several significant benefits to a data team and their data management practices, including:
Improved Data Quality
Standardisation and Consistency
Scalability
Data Governance
Facilitates Data Monetization
🔍 Episode Highlights:
- Introduction to Data Contracts: Andrew kicks off our episode with a basic understanding of what data contracts are and how they revolutionise GoCardless approach to data management.
- Why Data Contracts?: Discover the crucial role data contracts play in today's complex data landscape and why Andrew felt compelled to develop them.
- Benefits Unveiled: Learn about the substantial advantages data contracts bring to data governance, integrity, and management practices.
- Real-World Impact: Andrew shares compelling examples from his experiences at GoCardless, illustrating the significant improvements data contracts have facilitated.
- Addressing Common Data Challenges: Find out how data contracts can solve prevalent issues like data inconsistency and quality concerns.
- Implementation Insights: Gain practical strategies and tips for integrating data contracts into your data management processes, alongside the best tools and cultural shifts required for success.
- Overcoming Obstacles: Hear about the common challenges organizations face when adopting data contracts and the solutions to overcome them.
- Future Perspectives: What does the future hold for data contracts? Andrew shares his vision for adapting to upcoming data management trends and opportunities.
- Final Thoughts: Before we wrap up, Andrew imparts some invaluable advice for anyone considering data contracts as part of their data strategy.
Recommended
Transcript
Introduction to Stacked Podcast and Guest
00:00:02
Speaker
Hello and welcome to the Stacked podcast brought to you by Cognify, the recruitment partner for modern data teams hosted by me, Harry Golop. Stacked with incredible content from the most influential and successful data teams, interviewing industry experts who share their invaluable journeys, groundbreaking projects, and most importantly, their key learnings. So get ready to join us as we uncover the dynamic world of modern data.
00:00:34
Speaker
Hello, everyone. Welcome to another episode of the Stacked Data Podcast. This week, I'm joined by a very special guest, Andrew Jones, the creator of Data Contracts. Andrew is a principal engineer at GoCardless. Andrew, it's great to have you on the show. Really excited to dive into your philosophy around data contracts and really dissect what
00:00:58
Speaker
you've come up with what the benefits are and how people maybe can look to implement them into their own organization. So thanks for joining us. How are you doing today? Yeah, I'm doing very well, thank you. Yes, great to be here. I'm looking forward to it. I've been looking forward to this podcast for a while. Brilliant. Well, look, let's start off, Andrew, with I suppose just letting the audience get to know a bit more about yourself.
From Software to Data Engineering
00:01:22
Speaker
It'd be great to get an overview of your career today.
00:01:25
Speaker
Yeah, so I started off mainly in software engineering. I moved to data engineering a bit later on in my career. There's people even about the last eight years or so have been working in data. That's mainly on their platform side. So not necessarily on building data transformations and doing building pipelines directly by building tools and platforms to support other people creating data and create their pipelines and providing data to their users.
00:01:51
Speaker
For the last six years, I've been doing that at GoCarless. Actually, when I joined GoCarless, I was the first of Data Engineering, Data Platform hire. I had a great opportunity here with my team to build out Data Platform Scratch, which is quite a rare opportunity, but it's been a lot of fun. Yeah, I've learned a lot doing that, including eventually what became Data Contracts.
00:02:11
Speaker
Amazing. It must have been very exciting. I know co-card lists are quite the monster of an organization now, and I know a very, very well-respected data team, but to be there when it first started as the first engineer, what was it like back then, I suppose, as to what
Building a Data Platform at GoCardless
00:02:27
Speaker
it's like now? Could you just maybe hear a bit more about that journey before we dive into data contracts?
00:02:33
Speaker
When I joined, we didn't really have any data platform as such. Our data platform consisted of basically a replica of our main database, and we built on top of that. We were just starting to move to Google Cloud, which is what we're on now. Just starting to bring in tools like Airflow. Airflow just introduced when I joined. But I started from that, there was nothing really. I was just starting to do a big career as well. I came in and
00:02:56
Speaker
The first thing I did really was try to build a, what I thought was like a traditional data platform, following like the bottom data stack. We built a change data capture service, extract all the data out, checked into warehouse, and from that's what we built. That's what our data teams built all the analytics on, like back in those days. After a few years of doing that, we kind of
00:03:17
Speaker
Starting again, all the kind of problems that other teams have and other companies have around what happens with building on top of a something like a change date capture stream or an ELT kind of copy of database. I will dive into that a bit more in broadcast later, but like that's when we start realizing those problems we had, that's when we started thinking about upgrades through and that eventually led to date contracts.
00:03:39
Speaker
The idea started around four years ago, so after doing our traditional data platform for a couple of years, then we started thinking about maybe a different way to be doing things to achieve what we want to achieve with Data Toccardis.
00:03:51
Speaker
Amazing. You touched upon it there, Andrew, but what led you to develop this concept of data contracts and why is it crucial in today's landscape for many organizations, but particularly for Go
Development and Benefits of Data Contracts
00:04:07
Speaker
Cardless? It'd be great to hear about where the whole concept comes from.
00:04:11
Speaker
Yeah, initially, I had regular meetings with people consuming data. So, I had data analysts, I had BI engineers, data engineers, data scientists. And I kept hearing about the same problems, like re-kinging the data really, particularly things changing upstream, breaking things downstream. Of course, it's not being very good, it's being very hard to find data, there's no documentation. All those kind of problems that many data engineers have when they're trying to build upon data
00:04:39
Speaker
that typically come from an engineering function, a product engineering function. And after hearing the same problems like we can work out, I've ended up wondering, well, why is this the case? Why are we building things in this way? Is it not a better way of doing this? It doesn't seem to be the best way of doing things if we keep having the same problems over and over again.
00:04:57
Speaker
So I started thinking about that, and kind of leaning on my software engineering background, I started thinking, well, what we're doing here is we're building on top of someone else's database, and that is going to change, and that is going to change. If I was going to be a software engineer, I wanted to build something on top of a database, I would never go to a database that really, I'd always ask a team to create an API, some sort of interface that's attracted away from the database.
00:05:22
Speaker
And that idea, I asked that question to myself, like, why are we not doing that? Why is there no interface around data? Why is there no change management around that interface? And I kept asking myself that question. I couldn't really see a good reason why, apart from it's just not the way we have been doing it over the last few years, over the last couple of decades, really.
00:05:39
Speaker
And basically, that's kind of where it started. What if we had an interface? What if we had a bit more change management? What if we had something a bit more explicit about data generation rather than just sucking data out of the database? What if we changed some of those assumptions we've been having in terms of how we're building data pipelines today? And yes, it's really those kind of problems that are trying to solve the problems of having the same problems over and over again causing issues, same kind of issues each time.
00:06:06
Speaker
Moreover, we were also starting to use data for more important things regardless. It wasn't just analytics. Analytics is important, but it's not as important as some of the processes we have, some key business processes we have internally. That might be around managing our risk, forward detection, things like that, or even using data to drive product features and our product, whether that's directly or through ML,
00:06:30
Speaker
So we're starting to use data for more and more important things, and yet we weren't confident in the quality of data and the dependability of our data. So trying to align with that as a company goal as well, and think about how we could meet that company goal, it didn't seem like that you was going to achieve that, just building the same thing you've been building over the last few years.
00:06:49
Speaker
So, yeah, I mean, I know there's many data teams and engineers that are going through the same patting points day in, day out, week in, week out. You saw the problem and decided to come up with a solution that broke the traditional mold and also gave much more integrity, which, yeah, sounds great. And I think something that I know many data teams are probably going through and bashing their heads against the wall at.
Data Governance and Producer Ownership
00:07:17
Speaker
Let's jump in. Let's start with the basics, Andrew. What exactly are data contracts and how do they differ from the traditional approach of managing data?
00:07:28
Speaker
Yeah, I think at a simple layer, at a simple level, the easiest way to describe that construct is basically interface-free data. So interfaces are everywhere in software gene, APIs or interface. You've got an interface in your software libraries, in your pandas, whatever you're using there. Those are all interfaces. And it sounds quite simple, just interface-free data as well, but actually interfaces are very powerful. Like once you decide you want data for an interface,
00:07:56
Speaker
you're already forcing a more explicit generation of data. You're not just getting data out of a database anymore. You're saying, I want you to explicitly provide data through this interface. So you're already asking data producers to consider how they structure the data they're producing.
00:08:12
Speaker
who is it for, what requirements we're trying to meet with that data, who the consumers are, stakeholders are, those kind of things. Essentially, you restart and treat data as more a feature or a product, as well as become software engineering product they're building. So it's something they then provide to other teams in the organization.
00:08:30
Speaker
You also have a path to doing, but you're starting to really shift things left in terms of ownership as well. You're saying, as software engineers, you provide APIs. I've seen some building new on top of your services. You also need to provide date contracts so we can build just a lot of data you're producing.
00:08:46
Speaker
And that just becomes something they do. And as they start taking ownership of the interface, they start taking some of that accountability and responsibility for the production of data, make sure it's of the right quality, make sure it meets the needs of the users they have been updated for. You can also use the interface for enforcing things like change management, for doing some day college checks, and things like that as well, day governance as well. There's a lot you can do just by starting to think about finding data through an interface and describing that interface through
00:09:15
Speaker
my day contract. It's all about efficiency and I think the other key word that I took away from there was ownership and accountability. Would you say of the producer of the data ensuring that it remains working?
00:09:31
Speaker
Yeah, one of the goals from day one, really, was not just to think of different ways to move data around, but still have it owned by the same people, but by really being stronger on who owns this data. And in our opinion, it was, well, people producing data must be the owners of the data. Because only if they know things in and out of the data, they have full context on the data, they know how it's changed before, how it might change in the future. It's kind of fair data.
00:10:00
Speaker
So why were they not owners of the data? Why was they thinking the GM team owners of that data? And they had less context about data. They didn't know how it was going to change over time. Had no control over how it changed over time. So ownership and shifting left ownership was already, from day one was a goal of their contracts. Nice. And what are the key principles of data contracts and how do they contribute to a more efficient data ecosystem? Yeah, I think
00:10:28
Speaker
In terms of principles, it's really about what do you actually want to get out of our data contracts process and why are we introducing data contracts. So far as it was about shipping after ownership, it was about being explicit about how we generate data, it was about reducing the distance between data generators and data consumers.
00:10:47
Speaker
And the idea between that was, like, if they are much closer together, they're talking more often, they're collaborating more often, then naturally they all work together to improve quality of data. Because by producing data, they now start to understand the value of data they're producing. They talk to the consumers directly, they can see how it's leading to maybe a particular product feature that ultimately drives revenue.
00:11:08
Speaker
or how it's reducing risk in a certain risk area of ours, or whether it might be. So they become more incentivised to generate quality data, more incentivised to own that data, because they have a greater understanding of what that data is used for. And also you mentioned efficiencies, like one of the benefits of creating better quality data upstream or right at the source is that you have to do much fewer transformations of data downstream.
00:11:32
Speaker
And that saves a lot of time, it saves a lot of cost. Doing transformations on data warehouses is not particularly cheap, not particularly when you're using. And maintenance of virus are quite expensive. Particularly for doing a lot of work to clean up the data. A lot of transformations, a lot of implementation of business logic. A lot of that is a lot cheaper to do upstream as the data is being generated. So it becomes a more efficient way of generating and using data as well.
00:11:59
Speaker
Nice. There's a financial sort of optimization part to it as well. I suppose that question leads nicely into the next one.
Implementing Data Contracts: Cultural Shift
00:12:09
Speaker
I suppose there are many data professionals that listen to this show. They're keen to obviously understand a bit more about data contracts, but
00:12:18
Speaker
Often in data, you have to be able to put a business case together to implement something new. What are the primary benefits of implementing data contracts on top of what we've mentioned already? If there are any, and how do they enhance data governance, integrity, and overall data management?
00:12:39
Speaker
Yeah, I think the prior benefits of ones that we kind of spoke about already, and very initial reasons why we wanted to take contracts, really shifting ownership left, bringing producers and companies so that they look closer together so that we can
00:12:55
Speaker
get greater value from our data ultimately, and the outcomes from our data are greater. So we can provide reliable product features based on our data, based on AI maybe as well. And we could do that because we got good quality data from the source and we could do that cheaply and we could do that quickly. So I think there's a prime benefit.
00:13:15
Speaker
But as we started thinking about date contracts in more detail, thinking about how we're going to implement it, we saw a number of other benefits as well with that. And you mentioned governance, and governance is an interesting one as well. Because from day one, date contracts always described the data, and we knew we'd have things in there like the schema, but also some descriptions, documentation,
00:13:37
Speaker
We start thinking, well, we also need to categorize our data, particularly personal data. And if we can do that consistently within date contract across all our data sets, it then becomes very easy to build tooling about user categorization to automate away a lot of the date governance activities that are currently quite manual. So things like deciding on data retention rules or anonymizing data in response to maybe due to the transition crests or things like that.
00:14:04
Speaker
A lot of those can be quite manual processes, but if you have a standard way of documenting data, categorizing data, it's quite easy to be all talking about automating a lot of governance activities away completely. So by the time we've been doing it a lot, we can't say we're embedding date governance in date contracts.
00:14:23
Speaker
So you don't have to become an expert in GDPR, for example, to manage your own data. You just need to tell us a bit about your data and we're talking to take care of the rest. Amazing. So one thing I'm keen to explore, Andrew, is
00:14:39
Speaker
Some of the implementation strategies for data contracts, there's clearly many benefits as we've spoken about financial optimization, integrity, data governance, ownership. But yeah, how does an organization, how does the data team go about actually implementing this data contract strategy effectively?
00:15:01
Speaker
to support a successful migration to a more data contract like, well, you need to talk to a lot of people in the organization. You're kind of looking at more of a culture change rather than just providing some tooling. Because what you want to do is you want to, if your goals are to shift things left, to assign greater initial data, to assign, to improve quality of data at the source, which I think is where you get most benefit from data contracts.
00:15:29
Speaker
then you can't just build tooling and expect people just to start doing that. You need to do a lot of talking to people with brains and benefits of that. Why is it important to the organization? Why support to them? Why should I do a bit of extra work now? What's the benefits? Maybe there's cost implications. Why is it cheaper for the organization? All of that involves a lot of talking to people, particularly with data producers, which in our case for most of our important data are software engineers.
00:15:56
Speaker
So we spend a lot of time talking to those, a lot of time working with them on what the solution of date contracts should be, where should I be implemented, what should it look like. One thing I've seen other organizations struggle with date contracts is, the date team says they want to build date contracts on input in a way for the date team for themselves really.
00:16:17
Speaker
But software engineers aren't going to use that. They're not going to go somewhere else and categorize it in a younger document somewhere else or something like that. There's just too much friction for them. So we really, really focused on implementing the tooling where the developers were, which in our case were software engineers, and therefore we built it kind of out of platform layer. So whether to find the infrastructure, to find the APIs, or to find the contracts, all where they would expect them to be.
00:16:41
Speaker
And that really helped with adoption because, yeah, it's exactly what Ray wanted it to be. They weren't going, you know, filling in a spreadsheet somewhere or a download document or a web UI or something like that. It was what Ray expected it to be. So, really, we focused a lot on the experience for the people who wanted to use their contracts or to go with their contracts and that, in our case, for software engineers.
00:17:04
Speaker
Amazing. I know that's often one of the biggest challenges in data, isn't it? It's not the tooling, it is the people aspect. But before I go on to a few more questions around that, you mentioned tooling. Are there specific tools or technologies that align well when implementing data contracts? Or is it more around this sort of establishing this culture and procedural shift within the organization?
00:17:32
Speaker
Yeah, I think it's certainly more around the culture and kind of procedural shift really. Like I think there's, you can maintain contracts in a few different ways. And I think in some ways, I mean, it doesn't matter like how you implement it, but how you implement it needs to be tailored to your organization. I don't think there's anything off the shelf that you can necessarily buy and drop into your organization that's going to work perfectly. I think you need to understand how the touring supports the people in your organization and the behaviors you want them to take.
00:17:59
Speaker
So it's really that's the most important thing. What do you want people to do? How do you want them to do that? And how does this holding enable them to do those things and do those things easily and in a way that you want them to do that?
00:18:14
Speaker
But really, it's about people. You could build or buy the best date contract tooling you want. But if the people in the version aren't brought in to doing this, particularly date producers, if they are not understanding why they have suddenly been asked to take a bit more into the data, why they've been asked to fill in date contracts and produce different data, they're not going to do it, or they're not going to do it well. Even if you get like,
00:18:39
Speaker
a mandate from up higher saying they have to do that, what you're buying from them, they'll do the minimum effort, clear that out and understand the reasons why. It'll be a tick-box exercise. What you really want is them to be really brought into it, understand why that's important, understand why the outcome is important for organisations, what aligns with organisation goals, and then they will use hopefully your good tooling to go and do that.
00:19:03
Speaker
Amazing. I mean, I think that's very much how you framed it. You need to show us why is it going to be valuable to them in their world? Because I know from experience, if you're going to go and ask someone to do something that's outside of their normal remit, why most people will turn around and say that that's not what I have to do. So it's interesting to hear about how you've framed that conversation. Because I know that's conversations that many, I think, struggle to implement and to have
00:19:30
Speaker
have effective conversations to actually influence them people. So yeah, I think on that point, Andrew, is that are there any other, you know, tricks or examples of conversations that you're able to share, which were able for you to help, I suppose, sell and get that buy in on this philosophy? Because I think that's clearly one of the sticking points to this being success. I'd really love to hear any other examples that you've got. Yeah, I think
00:19:58
Speaker
I really spent a lot of time really speaking to people, particularly myself and my PM at the time, because she was a really great PM as well and she really helped a lot. And we've had a message from different audiences. So, are we talking to software engineers? How do we communicate to them? How do we show them the tooling and make sure the tooling is what they would be happy using? We're talking to sort of other product managers and then how we talk about prioritization and things like that.
00:20:23
Speaker
I was talking to leaders and like, why is this important to your organisation? They're really telling a message to different audiences and just over communicating as well, just talking and talking really to different people, different audiences, different stories, really trying to articulate the impact of that.
00:20:39
Speaker
I think that was important. I think the other thing we did well to get up to start was we started quite small. So we had like, we started working with particular teams who had a, have done a proof of concept. But before we bought a tooling reliever, we found a team, it wasn't necessarily the most important data set in our organization.
00:20:56
Speaker
But it was one where we knew we had good data sets with team already. That team were already quite data literate. They used data themselves. They had data people already come and better that team. They come and see the benefits of good data already because it helped them. And we started thinking about like, okay, what data sets can you improve with data contracts? What kind of tooling would you like to support that? How would you like to use it? And that was important for a few reasons.
00:21:24
Speaker
One, because they felt involved in the solution as well as the problem, they got kind of a sense of ownership of the solution as well. They're like, yes, we understand what I thought of this and we're giving you feedback and you're listening to it. It's something we're happy to use. Even though it wasn't perfect, it's still a kind of proof of concept. It was quite minimal at the time. They felt like they, they felt brought into it.
00:21:45
Speaker
Also, by starting small, once we proved the concept, proved the value, we could then replicate it to other teams and keep going from there. And you can't repeat this process. And eventually, you're starting to demonstrate the value. More broadly, and that's when you start thinking about, how do I get the foundation involved? How do I make this more? I think that we just do what we're just doing in certain places. So yeah, starting small was definitely a good idea. We had our start.
00:22:10
Speaker
I've heard this type of strategy before. I think it's one of the most effective ways for implementing new technology and new process. You, as you say, start with your small use case, really show the value and get that use case working and then look to roll it out.
Data Contracts and Data Mesh
00:22:28
Speaker
You can sometimes create sort of FOMO, particularly maybe with new tooling where other teams sort of start seeing the benefits. Whereas when you try and have one big splash, it can almost be lost. So I can really
00:22:40
Speaker
really see that. Interesting that you obviously talk about how many conversations you're having. One of the early episodes on the podcast was with Sandy Metta from Dojo talking about data mesh and how he said he used to wear a data mesh t-shirt or felt like he would wear a data mesh t-shirt into the office every day to shout about it.
00:23:00
Speaker
It sounds like having producers of the data links very closely to data mesh and data contracts. It'd be great to hear how they intersect in your eyes and how they complement each other.
00:23:16
Speaker
Yeah, so I can't mention that four years ago, something I started thinking about a contract and that was around the same sort of time that date mesh articles, original ones from Jamec came out on camera nine. And they were a great inspiration to me. I really liked the ideas that were in there. I felt like she was trying to get into the same problems we had and the constitutions were roughly aligned with what I was thinking.
00:23:39
Speaker
I remember thinking of a time that it just felt a bit big for what I wanted to do. I was a bit worried about bringing such a big idea into the organisation and starting to argue for organisational changes and all of those things as well. It felt like too much of a leap for me.
00:23:57
Speaker
So I felt like Date Contract was a way to start smaller on this and start working out a gradually towards organization that looks a bit like the identity mesh organization without having to do all the changes up front. So that's kind of how it started. And even now, really, I don't talk about date mesh internally at Go Cardless, really, ever. Because I still think it's one of those ideas that kind of scare people because of the size of it and the scope of it.
00:24:21
Speaker
But what we have done with data contracts is we have achieved a lot in terms of shifting things left, in terms of treating data as a product. A lot of the core concepts of data mesh, we have achieved through data contracts on its own. And that's probably as far as we need to go, really. I don't know if we need to go much further. So I think, yeah, they're certainly complementary ideas. They certainly have much of the same goals.
00:24:44
Speaker
I like to think of it as like, if you do date contracts, you can take it to a point, and then you can decide if you want to keep going further and create yourself an organization that looks exactly the same date mesh. And if you want to do that, I think that's great. But you might also decide to stop there. And that's as far as you need to go for now. And that's also great. And that's kind of where we are today.
00:25:02
Speaker
Yes, it's about tailoring the solution to what your problem and to what you're trying to achieve. I think that's the key message there and liking data. There's no one size fits all and great to hear that you obviously had that inspiration from the data mesh concept.
00:25:19
Speaker
Looking into the future, Andrew, as the data landscape continues to evolve, how do you foresee the philosophy of data contracts adapting, meeting new challenges?
Future of Data Contracts and Governance
00:25:32
Speaker
How is it going to pan out? Is there anything that you think can be developed further? You started this idea four years ago. What's next?
00:25:41
Speaker
as a concept or as a pattern, I think it's likely to remain relevant no matter what happens. Because really it's quite a simple idea. It's like we're going to describe our data and we're going to use that description to generate interfaces or automate their governments or build tooling that kind of uses that metadata, that contract to help us manage and provide data. So I think as an idea, it's likely to remain relevant.
00:26:07
Speaker
I mentioned at the start, it's kind of based on interfaces, and interfaces are everywhere, software, API, software libraries, for example, and they have been since forever, and they always will be, I think, interfaces, and date contracts just have an interface. So I think the idea is sound, and I think going forward, what we found is that we can just keep building on top of date contracts,
00:26:29
Speaker
Even from the start, we had this idea of what if we could build a data platform around data contracts? We call it this contract-driven data platform. Along with people who describe their data, it doesn't really matter what data is structured, what it actually contains. It might be quite a wide structure. It might be quite a deep structure. It doesn't really matter. We don't really care. We need them to describe the data. We can build tooling that
00:26:50
Speaker
helps them produce that data, helps them automate the governance, helps them produce the interface, helps them get the data out of the database. Any kind of tooling we think we could build as a platform team, we think we could build on top-of-the-day contracts.
00:27:02
Speaker
And I thought of four years of doing that, but that's proven to be the case, really. Like we use that contract to generate what could be the answer to the interface, which in reality is like BigQuery table or PubSub topic with schema. Whereas Google Cloud could be a Kafka topic, could be a Snowflake table, it doesn't really matter. We use that to backup data, we use it to anonymize data, we use it to
00:27:24
Speaker
you don't have to stay to attention, how long should you keep data? All of these kind of processes around data that you would expect the platform to handle, all handled through data contracts. And as a producer, you're not saying, I want backups. You're not only configuring back yourself, not because I want to become a GDPR expert and data retention expert and work out policies and things like that. You just describe it later and we will take care of the rest.
00:27:51
Speaker
I think this idea of you don't have to manage the infrastructure yourself. You just tell us what you intend to happen. I want this data. I want it provided through BigQuery. And I want maybe a library to help me produce that data in my particular language of choice. That's what you want. You don't have to build it all yourself or describe it yourself or invent it all yourself. You just have to tell us what you want and the platform can do the rest.
00:28:14
Speaker
So I think this is a really powerful idea that I think we're starting to see more of this idea of intention as code, I think something we've called it. And that's an idea that's quite exciting, not just for data tooling, but for platform tooling in general. So that's something I am looking into the future. I'm thinking it could be quite a powerful idea.
00:28:34
Speaker
Amazing. So looking back, Andrew, is there anything that you wish you'd done differently?
Lessons Learned and Human Element
00:28:41
Speaker
And I suppose what's your advice to anyone that's looking to implement or is now going to implement data contracts? That hindsight perspective is always a valuable one.
00:28:55
Speaker
Yeah, I mean, we certainly didn't do it all perfectly over the last few years, and it'd be weird if we did. One of the mistakes we made was trying to go a bit too quick with date contracts roll out. I mentioned we did start small, we did kind of start a curve series, but once we kind of proved it to Leash, it kind of got a bit curative about, okay, we want all data on date contracts, we want to do it in this time frame, we want to turn off our change date capture by this particular date.
00:29:19
Speaker
But that was too much. The value for many of those assets wasn't there. And the KPIs which host monitor migration, they didn't incentivize the right behaviors you wanted to incentivize. So if we set a team, we want 20% of data on date contracts by the end of a particular month. They might do that, but the 20% might choose it with the easiest ones and not the most impactful ones. So we certainly went a bit too far that
00:29:47
Speaker
What we should have done is really identified most viable data and where it makes more sense to have real data contracts and use KPIs that said something like, okay, we have noticed that we've had this many incidents over the last six months caused by change management data that has affected this particular business process or this production service. And we think moving data contracts will reduce that to risk number. And that's why we want to do that.
00:30:14
Speaker
We make cases like that, it's much easier to get prioritized just through kind of standard way that prioritization happens in a company or any company of a certain size. Like you might have OKRs and whatever else you might have to kind of do prioritization. There's no reason why date contracts or data in general should be outside that.
00:30:31
Speaker
The trick is making sure that you can not take out your value to get the other teams aligned and the value aligned to make that move to their contracts or make that move to improve their quality or whatever it is that you're trying to do through their contracts. Link it to the business outcome, the value that comes from business outcome.
00:30:50
Speaker
That's a really interesting point. You need to be tracking them, metrics, tracking them, breakages, checking where their problems are to be able to identify where data contracts may be useful. As you say, they might not be useful in every process. If you're not doing that already, I think that's maybe the first step to try and identify where it's going to be most effective.
00:31:12
Speaker
That's brilliant, Andrew. I think we've covered a lot of ground there. Are there any other bits of advice or insights that you'd be keen to share with the audience about data contracts in general, or have we covered everything?
00:31:29
Speaker
I think we covered anything, but I think it is still worth reiterating, but it's really, at least for ARCD contracts, it's really all about people side of the culture side. It's like, what do you want to achieve? What behaviors do you want to take through data? And why is that important to the company?
00:31:44
Speaker
And then date contracts and the tools and processes around date contracts, they are there to support those people, those behaviours that you want from these people to drive the outcomes you want for a business. I think often the date contracts see people online talking about it and often they focus too much on tooling and enforcement and what they could do with that, but really it's the people that are most important.
00:32:08
Speaker
And that's hard, right? It's hard. It's hard talking to people. It's hard getting alignment. It's hard trying to bring this new idea to an organisation. It's hard trying to indicate value. But if we can't do that well, we're not going to improve date quality no matter what we try and do, whether it's through date contracts or through anything else. So, yeah, really start with people and understand. We're talking about just support for people and processes.
00:32:33
Speaker
Amazing, amazing. Well, if any of our listeners want to learn more about those contracts, I believe Andrew has a book out. Yeah, I don't think he knew I was going to plug it, but I'm going to plug it for him. We can put a link in the description, but definitely worth a read if you're looking for more detail.
00:32:50
Speaker
I don't think I shared these questions with you beforehand, but at the end of each podcast, we ask all the guests just three quick fire rounds of questions, basically to help with the listeners with typical challenges in their careers.
Career Advice and Personal Growth
00:33:03
Speaker
The first one is, how do you assess whether a job opportunity is right for you and how do you know it's the right job?
00:33:12
Speaker
That's a good question. I guess you never really know 100%, but I like to think about the culture of a company. Does it seem like one where I can have good impact, I can grow, I can learn? I like to think about the culture of a company.
00:33:26
Speaker
That's probably the most important thing for me. It's the work I'm doing. Is it impactful? Is it impactful for the company? Is that clear from the description from the interview process? Because if I'm going to be working on something that's not impactful, that's going to probably go and limit my own growth. The impact of the projects and work I'm doing and the culture of the company are most important to me.
00:33:46
Speaker
I think they're all intertwined then ones, aren't they? The culture of the company, the value of what you're doing, I think that intrinsically links to a company's appetite for data and that also adds to, I'm sure, to the culture. So, yeah, great point. The next one, Andrew, is what is your best advice slash tip for interviewing?
00:34:07
Speaker
I mean, to be honest, really, it's best advice. When I'm interviewing people, I do quite a few interviews, and it's quite easy to tell if someone is not being completely honest, but they kind of don't know the answer, they're just trying to work very through that answer without really knowing the answer. That's fine. It's okay not to know the interview. I'm not looking for 100% pass-by on all the questions. What I'm looking for is understanding your experience and what knowledge you do have.
00:34:34
Speaker
So, and if I'm not sure, people said like, I just don't know the answer to this. These are some things I might consider roughly in the area. And these are thinking I might go and look up and Google or read my book about, but I don't know the answer. I think I'd rather hear that answer when someone tried to explain the tunnels of Kafka or something, but I don't really know it.
00:34:51
Speaker
I think that's a great one. I think that's what advice I often give to people. If you don't know the answer, say, I don't know the answer, but this is what I do know. This is if I was to try and piece this together. And as you said, this is how I go and find out that answer. No one ever knows the answer to everything you never do in data. It's moving way too quickly. So great advice. And final question, Andrew.
00:35:16
Speaker
What is the best resource for upskilling yourself within the industry? I think that depends on people have different ways of kind of learning. I think the best way to upskill really is to do it in practice. That's not always possible particularly interviewing for a job that's maybe a bit different from what you're currently doing. Maybe you're trying to do a slight change or move to
00:35:38
Speaker
different kind of move up on the ladder a bit. In which case, I think, for me, at least I do a lot of reading, I try to engage a lot of people, attend meetups, things like that, big people on LinkedIn, people who might be friendly on LinkedIn. If you reach out to them, I have questions, particularly if you can help them as well with something and kind of share knowledge as well. So I think
00:35:59
Speaker
Yeah, that's another great skill is by building a network, by talking to people who are doing a job that you want to be doing. If you can't, if not kind of doing that, you can't learn it on a job and that's the next best thing probably. I 100% agree. I think, yeah, obviously doing is great. The next best thing I think is speaking to people. Not everyone's that scary. And I know reaching out to people that you maybe don't know can be a scary thing. But you'll be surprised that many people are keen to
00:36:27
Speaker
To help if they can, and yeah, going to meetups is a great way. Many people, that's why they're called meetups. People are there to share and engage in conversation with other like-minded people. I know we've met many times at the analytics engineering meetup, which is one that I'm very fond of, but there's many in each area of data that are across London. So yeah, jump on Meetup and get involved.
00:36:52
Speaker
Yeah, because I'm quite introverted. I'm not particularly great at going to meet up with people, but like I said, people are generally friendly. Well, everyone's friendly. People are expecting to be helpful, and they want to be helpful, and they want to talk to you. It's actually the same when you think about
00:37:08
Speaker
maybe try and work out that contract and speak to software engineers. You may think that software engineers, they don't care about data, they don't understand why, they're not interested. Again, when I speak to them, they do always want to help, they just don't understand how, they don't understand the impact, the problems they're having, you're having in data. But you just have to go and talk to people, you just have to, yeah, people are always helpful.
00:37:28
Speaker
Yeah, I think putting yourself out there is, you know, people are scared to do it, but only ever positive things that often that you see. So, Andrew, thank you ever so much for joining us on the show. It's been a really insightful conversation, lots for the audience tour to unpick. Thanks again. And yeah, best of luck with the rest of the build and the journey at Go Cardless. Cool. Thank you very much. Thanks, Harry. Bye, everyone. See you next week.
00:37:57
Speaker
Well, that's it for this week. Thank you so, so much for tuning in. I really hope you've learned something. I know I have. The Stack podcast aims to share real journeys and lessons that empower you and the entire community. Together, we aim to unlock new perspectives and overcome challenges in the ever evolving landscape of modern data.
00:38:18
Speaker
Today's episode was brought to you by Cognify, the recruitment partner for modern data teams. If you've enjoyed today's episode, hit that follow button to stay updated with our latest releases. More importantly, if you believe this episode could benefit someone you know, please share it with them. We're always on the lookout for new guests who have inspiring stories and valuable lessons to share with our community.
00:38:41
Speaker
If you or someone you know fits that pill, please don't hesitate to reach out. I've been Harry Gollop from Cognify, your host and guide on this data-driven journey. Until next time, over and out.