Use case
Questions over company data
Managers ask questions about sales, margin or operations in plain language and get figures from governed metric definitions, with the query shown so an analyst can check it.
The problem
How the work goes today
A manager wants to know why gross margin dropped in one region last quarter. They file a request with the data team, wait a few days, and get a spreadsheet that answers a slightly different question.
Different teams calculate revenue, churn and utilisation in different ways, so two reports for the same month disagree and the meeting is spent reconciling them. Nobody is sure which definition the board pack used.
The monthly management pack is put together by hand from exports, with commentary written late the night before. The numbers are rarely wrong, but the explanations are often guesses.
21.3%
Share of 632 real-world enterprise text-to-SQL tasks solved by an agent built on o1-preview, against 91.2% on the older, simpler Spider 1.0 benchmark. Leaderboard scores have risen since, but messy enterprise schemas remain hard for raw text-to-SQL.[1]
How it works
Step by step
Highlighted steps are where a person decides. Everything else runs on its own and is logged.
Systems involved
- Data warehouse (Snowflake, BigQuery, Databricks)
- Semantic layer (dbt Semantic Layer, LookML, Cube)
- BI tools (Power BI, Tableau, Looker)
- ERP and finance systems (NetSuite, SAP, Dynamics)
- Data pipelines (Fivetran, Airbyte)
- Chat surfaces (Microsoft Teams, Slack)
- 01Person
Agree metric definitions
Finance and metric owners define revenue, margin, churn and other measures once in a semantic layer (dbt, LookML or Cube), with grain and filters.
- 02Person
Ask a question
A manager types a question in the reporting app, Teams or Slack.
- 03Agent
Map to metrics
The agent maps the question to governed metrics, dimensions and a time range, and asks a follow-up when the question is ambiguous.
- 04Agent
Build the query
It generates a query against the semantic layer, or SQL with schema context when no metric fits.
- 05System
Validate and run
The query is checked as read-only with row and cost limits, then run in Snowflake, BigQuery or Databricks under the user's own row- and column-level permissions.
- 06Agent
Answer with workings
The answer comes back as a chart, a table and a short explanation, with the metric definitions and query shown.
- 07Agent
Draft the management pack
Each month the agent drafts pack commentary from the same governed queries, keeping computed changes apart from possible causes.
- 08Person
Owner approves
Finance or the business-unit owner reviews figures and commentary before anything goes to the board, investors or outside parties.
Measurement
What we measure
Agreed before anything is built, with a baseline taken from how the work runs today.
- Execution accuracy
- Share of a fixed set of benchmark questions where the returned result matches the answer an analyst has validated, not just whether the query ran.
- Semantic-layer coverage
- Share of real user questions over 30 days that could be answered from governed metrics without falling back to free-form SQL.
- Wrong-assumption rate
- Share of sampled answers where the system silently picked a definition or time range the user did not mean, reported next to how often it asked for clarification.
- Reconciliation rate
- Share of AI-produced figures that tie exactly to the official reported numbers for the same period and definition.
- Time to trusted answer
- Median time from question to an answer the requester accepts, compared with the turnaround on data-team tickets before launch.
Pitfalls
What usually goes wrong
- Plausible numbers that are wrong
- Answer from a semantic layer, run a set of golden questions on every change, show the query and definitions, and keep free-form SQL for analysts.
- Nobody agrees what 'revenue' means
- Make the system ask which definition is meant (booked, billed or recognised) and state the one it used in every answer.
- Data shown to the wrong people
- Run every query under the user's own identity so warehouse row- and column-level security applies, never through a shared service account.
- Runaway warehouse costs
- Set per-query cost and row limits and cache frequent questions.
- Invented explanations
- Separate what changed, which is computed, from possible drivers, which are labelled as hypotheses for the owner to confirm.
Build or buy
When a ready-made tool is enough
Custom work isn't always the answer. Here's how we'd decide.
- Power BI Copilot, Tableau Pulse, ThoughtSpot Spotter, Looker with Gemini, Snowflake Cortex Analyst and Databricks AI/BI Genie work well when your data already sits in that platform with a modelled semantic layer.
- We found no independent study of business outcomes from natural-language analytics, only vendor case studies, so we set targets on accuracy and turnaround rather than promise savings.
- Custom work pays off when data is spread across several warehouses or ERPs with no semantic layer. In that case the data infrastructure is the real project and the question interface comes after it.
- It also makes sense when you need a private model, when management packs must follow your own templates and approval steps, or when answers have to combine figures with documents such as board papers and contracts.
Industries
Where this applies
Sources
Figures are as reported by each source on its publication date. Examples are public industry cases, not AI Sewa clients.
Next step