About Cazoo #

Cazoo is transforming the car buying and selling experience across the UK. Consumers can simply and seamlessly buy or finance a used car entirely online for delivery or collection in as little as 72 hours.

Benefits from Synq

• Unified alerts from multiple warehouses routed to relevant teams
• End-end monitoring of data controls across Redshift, BigQuery and Looker
• Reduced ongoing data issues by 75%

About the interviewee

Weiting is the Head of Product Analytics at Cazoo. As employee number 12, she’s a founding member of the data team at Cazoo and has been on the journey all the way from setting up the initial data stack to seeing the company go public.

The challenge #

With 1500 dbt models spread across Redshift and BigQuery projects, 1200 Looker dashboards and 40 people building new data assets, Cazoo wanted to make sure they had full observability into issues across all systems in their complex stack.

The team had identified three problems they wanted to solve

1. Centralise management of alerts with the ability to assign ownership
2. Ability to quickly assess root causes of issues and implement error monitoring to reduce the number of ongoing issues
3. Automated lineage between dbt projects across BigQuery and Redshift to Looker

The solution #

The new tool had to centralise alerting, issue resolution workflows and error management, and have deep integrations with dbt, Looker, Redshift and BigQuery. To reduce friction in adoption and eliminate any maintenance requirements the tool needed to be 100% maintainable without any bespoke data engineering work.

“Synq plays a big part in reducing overhead for us in managing the complexity across our existing stack. Despite the complexity of our setup we were up and running without any custom engineering work”

Unified alerting & ownership #

Cazoo’s most pressing challenge was getting on top of the daily inflow of issues across their warehouses. The team had tried making ownership of issues more explicit through a static Notion page but hadn’t seen much business impact from it. With Synq, they’ve put ownership into action by tightly integrating it into their alerting workflows – and now have one centralised place for all alerts with the relevant owner automatically being tagged.

“We invested heavily in clearing up dozens of failing dbt tests that created a lot of noise in our alerting channel. We used Synq to get notifications across all our systems into one channel so we can easily see if issues originate from Redshift or BigQuery, who its owner is, potential root causes and downstream impact.”

Redshift alerts - dbt test alert from the Redshift project with owner, importance, root cause and impact

“We’re writing more tests now because it’s more manageable to monitor them and we can get more granular on what we test for to catch edge cases”

Building scalable workflows for managing test errors #

Since adopting Synq, Cazoo has a single source of truth for monitoring and actioning data issues across the organisation. For starters, Synq gives the team the ability to group failing assets by tag, warehouse and owner. This helps people across teams always be on top of ongoing issues and how long they’ve been failing so they never lose track. And crucially, with more than 20 active dbt users, Synq has become a shared reference for how to monitor, debug and prevent data issues.

“We tag known issues as under-investigation in dbt. We use Synq to monitor these issues separately and can quickly see if they are getting worse over time–without mixing them up with new and more urgent issues”

Custom views to track ongoing issues - Cazoo uses Synq views to track ongoing test errors across warehouses and owners

By making reporting and pipeline observability available company-wide, Synq has helped Cazoo improve communication and collaboration among teams. There’s no longer any siloing when it comes to monitoring data quality.

“Every data practitioner can now understand all issues, see who the owner may be and immediate steps to resolve or escalate them which means we no longer have an unmanageable backlog of data issues”

Automated warehouse to BI lineage #

Although the team had gotten in control of most ongoing data issues, they still frequently faced the challenge of having to assess the downstream impact across potentially thousands of Looker assets. They had tried manually maintaining exposures in dbt but quickly realised that the scale meant that this was not a feasible solution. In turn, end-users too frequently experienced incorrect or stale data and with hundreds of active Looker users this was not sustainable.

With Synq, they’ve automated this and can instantly jump from an alert in Slack to seeing the downstream impact and understand if a core company-wide dashboard is being impacted–and then act accordingly.

“Before Synq, our visibility stopped at the mart models in dbt. Now we can go beyond dbt and see the full impact of issues and document where individual data points are coming from. This has been a huge benefit when onboarding new joiners”

Sharing ownership of data quality with producers #

Next step for the team was to extend ownership of data issues to upstream engineering teams as a large part of issues originated from them. With Synq, they’ve set up different rules to automatically route issues from specific domains to the relevant upstream teams. The data team largely works embedded across multiple domains and a key criteria was that the analysts in each domain could set up and define the alerts without needing to involve the central team.

Data-engineering-pricing-alerts - this rule automatically notifies upstream engineers in the pricing domain of issues on models where their data is used as the source

“…It’s allowed the pricing engineers to jump on an issue straight away which we may have otherwise only caught later down the line. As it flows directly to their alerts channel it means they can monitor it themselves and are not reliant on us to raise any flags”

What’s next #

Three months into using Synq, the team at Cazoo have end-end observability spanning from engineering all the way to BI dashboards and have gotten in control of data issues across dbt projects on multiple warehouses. Next up for the team is extending the ownership model to other domains to bring in the engineering team in the data quality journey company-wide.

As part of a company goal of reaching profitability, the team is also planning to invest in monitoring and reduction of cost of their data warehouses as well as cleaning up unneeded assets across dbt, BigQuery, Redshift and Looker.

“As we’re making progress as a data team we’ll be using the analytics in Synq to help us set internal quality targets and monitor our performance against them”