With over 40,000 companies relying on dbt and a community of more than 100,000 members, dbt has rapidly become a central tool for data teams to transform and test their data. The top reason behind dbt’s fast adoption curve is that it supercharges the way data people work–so much that it warranted the creation of a new role – the analytics engineer.
At this year’s Coalesce conference, dbt reinforced their ambition to sit in the middle of data workflows by becoming the control pane for data and AI. This sparked two questions we get asked over and over again that we’ll answer in this post:
- When to use a data reliability platform alongside dbt
- Factors to consider when using a data reliability platform alongside dbt
When to use a data reliability platform alongside dbt
We have collaborated with and consulted over 1,000 data teams, ranging from single-person teams to large-scale Fortune 500 companies. Consistently, two factors predict how likely they are to see the expected benefit from bringing on a data reliability platform: 1) how important data is to the business and 2) the complexity of data.
Fit matrix for bringing on a data reliability platform
Derived from interviews with over 1,000 data teams across diverse industries and sizes
The data maturity and complexity matrix indicates when it might be the right time to bring on a data reliability platform.
Low fit–smaller data teams handling non-critical data can rely on dbt’s native features without needing a separate data reliability platform. Everyone in the data team still has the context of most data assets, ownership is not yet an issue, and when things break, it’s most often okay for it to be resolved within a few days. You’ll likely only see a limited benefit of onboarding a data reliability platform.
Medium fit–data teams with medium data maturity and simple to moderate complexity frequently use data to drive key business processes and decisions. Uncaught issues have a meaningful impact on decision-making and trust in the data team. You’ll likely see a meaningful benefit of onboarding a data reliability platform such as speeding up resolution times and reducing issues caught by stakeholders in dashboards.
High fit–data teams with business-critical use cases and high complexity. Data is used for AI/ML models, regulatory reporting, or core business processes. Uncaught data issues can mean $100,000 in lost revenue, directly impact customers, or lead to increased regulatory scrutiny. You’re very likely to see a significant impact from onboarding a data reliability platform such as detecting and preventing critical issues in near real-time.
For teams managing complex, business-critical data, integrating a data reliability platform with dbt creates a supercharged stack—combining dbt’s strengths in transformation and testing with advanced monitoring, ownership activation, incident management, and quality analytics.
The supercharged data reliability workflow – dbt enhanced with a data reliability platform
For more, see the article: How data observability fits into the different stages in the data pipeline
A data reliability platform has a significant impact on complex, business-critical data teams to deliver reliable data. But too often, companies compromise their well-established dbt workflows when bringing on a platform. This slows developer productivity and adoption.
In the remainder of this post, we’ll look into key considerations data teams on dbt had when they turned to SYNQ as their data reliability platform.
Factors to consider when using a data reliability platform alongside dbt
The challenge: disjoint reliability workflows impact productivity
Workflows in dbt and the expectations from a data reliability platform should be closely linked. For example, you may define static tests in dbt but use the data reliability platform for more advanced anomaly monitoring. You should put high expectations on your data reliability platform to fit into existing dbt workflows.
We’ve seen first-hand how bad it can get if you start having multiple, disjointed workflows across different tools such as dbt, a data reliability platform, and a data catalog.
One team was forced to define the same tests multiple times, both in the data reliability platform’s UI and in dbt to be able to make use of the ownership functionality and get unified alerting across data and business teams.
Another team had to keep ownership definitions updated in a dispersed set of tools and dbt making it prone to inconsistencies and definitions getting out of sync.
A third team analyzed uptime and the health of their data without the ability to look at both dbt tests and anomaly monitors uptime giving only a partially complete picture.
“In our previous data reliability platform we couldn’t always link data assets to trace back if an anomaly monitor and a dbt test failing indicate the same issue” – Stijn, Data Engineering Manager at Aiven
Below, we’ll look at five factors to consider when bringing on a data reliability platform to best fit existing dbt workflows.
#1: Coverage of all dbt concepts
If you’re a data practitioner serious about dbt, chances are you will want to use the latest functionality such as dbt groups, public and private models, or data contracts. We’ve seen again and again that if these are not supported, it leads to disjointed workflows and compromises.
For example, you may have defined model ownership using dbt groups and want to use this to determine which teams are alerted about issues. If your data reliability platform doesn’t support defining alerting routing based on dbt groups, you’ll have to apply patchy workarounds.
Aiven uses dbt groups combined with dbt tags and folder structures to define data products in SYNQ
If you use ephemeral dbt models and these are not supported in the data reliability platform’s lineage, it creates black boxes making it difficult to confidently assess the full downstream impact or trace back data quality issues to upstream models.
If you use model contracts in dbt to prevent models from being built if a contract is breached, you’ll want data contract breaches to fit your existing workflow, and for example, be routed to relevant upstream owners through the ownership activation you’ve set up in your data reliability platform.
A big part of why we moved to SYNQ is that they have the platform most invested in dbt, and we can count on them to invest in and build support for new functionality.” – Stijn, Data Engineering Manager at Aiven. Read more about how Aiven uses dbt alongside SYNQ for unified observability.
#2: Control the entire data stack with dbt metadata and assets
When you onboard a data reliability platform you’ll want to configure data ownership, asset severity, data product definitions, and anomaly monitor deployment. Managing this from dbt metadata has several benefits – it fits into existing workflows for managing configuration through code in YML files, offers a traceable history of all changes, allows for PR reviews, and reduces vendor lock-in.
Your data reliability tool should support metadata configurations from dbt so that all key configurations can be controlled through code. Support shouldn’t stop with simply letting you use dbt tags for monitoring and ownership deployment. Here are a few examples of how we’ve seen our customers use metadata and structures in more advanced ways
- Use dbt asset types such as dbt sources to programmatically place freshness and volume monitors upstream of all data asset set to importance: high in dbt
- Use dbt group definitions to create data products and manage ownership
- Use folder structures in dbt to manage ownership based on data warehouse schemas
Aiven uses SYNQ’s query engine to programmatically place volume monitors on all sources upstream of dbt models
#3: Centralized overview of data quality
We’re starting to see the best data teams take a calculated approach to data quality. They’re able to track and analyze their data product uptime over time, by different owner groups and by data control (e.g., type of dbt test, anomaly monitor). This helps them systematically improve important areas, communicate data quality to the rest of the organization, and monitor low signal-to-income data controls to reduce alert overload.
Your data reliability platform should give you a comprehensive overview of data quality controls across dbt and anomaly monitors to give you the complete picture of the health of your stack. Consider if you’re able to segment the data in ways that make the insights actionable, such as by data control and owner, as well as any custom dbt tags that are relevant to your business.
As a minimum, you should be able to answer questions such as:
- Percentage of dbt not null dbt tests that failed by area
- Data quality metrics segmented by owner domain and data product using dbt tags
- Compare issue-to-incident ratio across dbt tests and anomaly monitors to identify low signal-to-noise ratio data controls
Every three months, we meet with the chief risk officer, chief technology officer, and bank CEO to update them on the latest developments, risks, and opportunities. This helps everyone contribute to and have a stake in our data quality – Bjørn, Data Manager
Watch the video of how Lunar delivers data quality KPIs to the C-level across dbt and SYNQ 
#4: Consistency of workflows around ownership
Ownership is nuanced and should allow for some flexibility–after all, there’s no one-fit-all workflow when it comes to ownership. One of the biggest pitfalls we’ve seen our customers have when they migrate to us is when they have a dispersed set of tools where ownership has been defined–some in dbt metadata, some in a data catalog, and others in the data reliability platform.
For many, dbt is an excellent choice for a centralized repository of ownership definitions. You can see a history of all changes, use packages such as dbt pre-commit to enforce that an owner tag exists on certain models, or easily migrate between tools without updating ownership definitions.
As such, your data reliability platform should support your current definitions of ownership – whether you use dbt meta owner tag, folder structure, dbt groups, or any other logic. It’s also important to consider how issues such as a failing dbt test fit into existing workflows. For example, if a data platform team is responsible for ingesting data, they may want a dbt source freshness error to be sent directly to PagerDuty to prioritize it alongside other production failures.
We rely on dbt tests and anomaly monitors in SYNQ to notify the data team and key business stakeholders so we’re always on the forefront of issues. Our model risk governance team uses quality analytics across all data controls to notify us if any control areas are falling behind our risk tolerance – Maria, Data Platform Engineering Lead at Ebury
#5: Cross-tool lineage across multiple dbt projects
Not having the full lineage can create a false sense of security – you may falsely assume that an issue didn’t impact a dashboard downstream if you don’t have a BI integration, or you may conclude that an issue doesn’t originate in the data stack if you don’t have visibility into upstream dependencies from other dbt projects.
We recently faced a complex issue where data discrepancies appeared in our BI reports. With SYNQ, we could quickly trace the issue back to its source across multiple dbt projects, saving us many hours of manual investigation and significantly reducing the resolution time
Even if you’re not currently using multiple projects in dbt, it can be a good idea to consider support for it as you may expand to a multi-project setup in the future. Cross-project lineage significantly enhances visibility, allowing teams to trace the flow of data from raw ingestion to final reports or data products. A unified view that includes the status of nodes from dbt, the flow of logic across dbt code, and the precise impacts of changes in the data stack. This simplifies issue resolution—cutting down hours of manual investigation—but also enables better collaboration and faster insights.
Summary
When to use a data reliability platform alongside dbt–supplementing dbt with a data reliability platform becomes essential for teams managing critical data (e.g., AI/ML models, regulatory compliance) due to the high stakes involved in data quality. However, for smaller teams handling less critical data, dbt’s built-in features are most often sufficient.
Factors to consider when using a data reliability platform alongside dbt
- Coverage of dbt concepts–ensure the platform supports dbt’s latest features (e.g., groups, model contracts).
- Control with dbt metadata–you should be able to use dbt as a single source of truth for ownership, severity, and data product definitions and have the data reliability platform be configured based on these.
- Centralized data quality overview–a complete overview of data quality helps you systematically improve. These metrics should include both dbt tests and assets as well as monitors from your data reliability platform.
- Consistency in ownership workflows–centralize ownership definitions in dbt to avoid fragmented management across tools.
- Cross-project lineage–the data lineage should support multiple dbt projects to be able to fully track data flow and troubleshoot issues across projects.
Sign up for our webinar on Nov 5th to learn how Aiven and Shalion use dbt and SYNQ to deliver reliable data products