Evaluate natural-language analytics against an answer contract, not against whether the prose sounds convincing. For every important question, define the expected intent, metric, dimensions, grain, calendar, filters, security role, benchmark, tolerance, and safe-failure behavior. Test realistic wording variants and repeat runs because generated output can change. Score the calculation and evidence separately from the explanation. A response should fail if it uses the wrong definition, exposes unauthorized data, invents a cause, or gives a precise answer where clarification was required—even when the final number looks plausible.
Microsoft states that Copilot output is nondeterministic and recommends setting expectations so users can evaluate or validate responses. Microsoft’s Power BI lifecycle guidance recommends known-benchmark questions, comparisons with trusted reports or analyses, and before/after testing when a model changes. Validate Power BI content
The evaluation unit: one question contract
A prompt alone is not a test case. The same words can be correct for one business definition and wrong for another. Store these fields together:
| Contract field | What to record | Example |
|---|---|---|
| User question | Natural wording from the target group | “Which regions missed plan last fiscal month?” |
| Intent | Decision or analysis requested | Identify regions below plan for recovery review |
| Approved metric | Governed measure and definition | [Net Revenue vs Plan %] |
| Dimensions and grain | Required grouping and result grain | Region × closed fiscal month |
| Calendar and date role | Period convention | Corporate fiscal calendar; invoice date |
| Filters and exclusions | Population rules | Exclude internal transfers and provisional adjustments |
| Security role | Identity context for the run | Regional leader |
| Benchmark | Independent source of expected values | Certified performance report plus benchmark query |
| Tolerance | Acceptable numeric variation, if any | Exact for counts; agreed rounding tolerance for percentages |
| Safe-failure behavior | When to clarify, decline, or state a limitation | Clarify if “last month” could include an open period |
| Evidence captured | What proves the run | Timestamp, model version, prompt, returned visual/data, notes |
This contract is a Refinity recommendation. Microsoft’s guidance supplies the need for benchmark, before/after, and repeated evaluation.
An eight-part scoring rubric
Score each dimension independently:
- 2 — Pass: satisfies the contract with traceable evidence.
- 1 — Partial: materially useful but has a contained, disclosed gap.
- 0 — Fail: wrong, unsupported, unauthorized, or missing.
| Dimension | Pass condition | Automatic failure example |
|---|---|---|
| 1. Intent | Answers the decision the user actually asked about | Ranks regions by revenue when the question asked for variance to plan |
| 2. Metric | Uses the approved measure and definition | Uses gross sales instead of net revenue |
| 3. Grain and dimensions | Groups and aggregates at the contracted level | Combines weekly targets with daily sales and double counts plan |
| 4. Time and filters | Applies the correct calendar, date role, period status, and exclusions | Uses calendar month instead of fiscal month |
| 5. Numeric result | Reconciles to the benchmark within the agreed tolerance | Total differs because an implicit measure was selected |
| 6. Security | Returns only data permitted for the tested role | Regional user sees another region’s values |
| 7. Evidence and traceability | The source/model/visual or query path can be identified | Precise number has no reproducible path |
| 8. Explanation and restraint | Wording stays within the returned data and states relevant limits | Claims a cause from a descriptive comparison |
Do not average away a zero for security, material numeric correctness, or a critical metric definition. Those should act as blocker overrides even if the prose and formatting are excellent.
Build a golden-question set
A golden-question set is a versioned collection of consequential questions and expected-answer contracts. It should be small enough to maintain and broad enough to expose the decisions the model must make.
Include these question types:
| Question type | Purpose | Example |
|---|---|---|
| Known benchmark | Confirm core metric and period logic | “What was net revenue in fiscal August?” |
| Dimensional cut | Test relationships and grouping | “Net revenue by customer region” |
| Comparison | Test multiple measures and time windows | “Which regions were below plan last month?” |
| Synonym variant | Test business-language mapping | “Turnover” as a defined synonym for revenue, if appropriate |
| Follow-up | Test retained context | “Now show only the West” |
| Ambiguous question | Require clarification | “How were sales recently?” |
| Unsupported inference | Require limitation or refusal | “What caused the decline?” without causal evidence |
| Permission-sensitive | Exercise RLS/CLS and source access | Same regional ranking asked by two roles |
| Edge case | Test blanks, zero denominators, new entities, or open periods | Margin percentage when revenue is zero |
Do not fill the suite with twenty paraphrases of the same total. Each question should exercise a distinct semantic or operational risk.
Test output as data, visual, and language
Microsoft documents that a Power BI Copilot data answer can include a generated visual and a natural-language summary of the semantic query result. It can also return an error or clarification request. Use Copilot with semantic models
Evaluate each layer separately:
- Selection: Did the system choose the intended fields and measures?
- Query result: Are rows, totals, periods, and filters correct?
- Visual: Does the visual preserve the correct grouping and make the result inspectable?
- Summary: Does the prose accurately describe the returned result without adding unsupported explanation?
- Failure behavior: If the request is ambiguous or unsupported, did the system clarify or state the limitation?
A correct chart with an overconfident causal summary is not a complete pass. Neither is polished prose over the wrong calculation.
Worked example: a failed golden question
Question: “Which regions missed plan last fiscal month?”
Expected contract:
- use
[Net Revenue vs Plan %]; - closed corporate fiscal month;
- group by customer region;
- exclude internal transfers;
- apply the requesting user’s RLS;
- identify regions below 0%;
- do not explain causes without additional evidence.
Observed answer:
East and Central underperformed because customer demand weakened last month.
Observed visual values were calculated with calendar month, included an open week, and used Gross Sales vs Target. The user was allowed to see East but not Central. Even if “East” truly missed plan, this run has four independent failures:
| Rubric dimension | Score | Evidence |
|---|---|---|
| Intent | 2 | It attempted to identify below-plan regions |
| Metric | 0 | Wrong measure and plan definition |
| Grain and dimensions | 2 | Region-level grouping was correct |
| Time and filters | 0 | Calendar month and open week violated the contract |
| Numeric result | 0 | Cannot reconcile under the required definition |
| Security | 0 | Returned an unauthorized region |
| Evidence and traceability | 1 | Visual exposed fields, but no approved benchmark link |
| Explanation and restraint | 0 | Invented a causal explanation from descriptive data |
The correct outcome is block, not “8 out of 16, needs improvement.” Fix the security path and semantic definition, then rerun the complete contract.
Account for nondeterminism without accepting randomness
Microsoft says the same prompt and grounding data can produce different Copilot output. Prepare your data for AI
Use a repeated-run protocol:
- Fix the model version, refresh timestamp, security role, and starting context.
- Run the exact question multiple times.
- Run representative wording variants separately.
- Compare contracts—metric, period, filters, values, and safe-failure behavior—not exact sentences.
- Record both pass rate and failure type for the evaluation session.
- Investigate any critical blocker even if most runs pass.
Refinity does not recommend a universal numeric pass threshold. The acceptable release bar depends on question consequence, user population, failure containment, and whether a human reviews the answer before action.
Before-and-after testing
Change one variable at a time:
| Experiment | Keep fixed | Change | Evidence of improvement |
|---|---|---|---|
| Focus AI schema | Golden questions, model measures, roles | Included/excluded fields | Wrong-field failures fall without new unanswered questions |
| Add verified answer | Source visual and benchmark | Trigger phrases / verified mapping | Known question reliably returns the maintained visual |
| Add AI instruction | Model and AI schema | One local terminology rule | Target variants resolve correctly without regressions elsewhere |
| Repair measure | Questions, period, roles | DAX and ownership record | Values reconcile across required cuts |
| Change RLS | Questions and measures | Role definition/mapping | Each role sees the intended population and no more |
Microsoft’s lifecycle guidance explicitly recommends comparing outputs before and after semantic-model changes. The one-variable experiment design is a Refinity recommendation.
Release gates and lifecycle
Block release when
- a target user can receive unauthorized data;
- a consequential metric or calendar remains ambiguous;
- the benchmark cannot be reproduced;
- a critical question produces materially wrong values in any representative role;
- the experience presents unsupported causal or prescriptive claims as fact;
- nobody owns the model or evaluation set.
Re-evaluate when
- a measure, calculation group, relationship, or calendar changes;
- fields are added to or removed from the AI data schema;
- a verified answer, trigger phrase, or source visual changes;
- an AI instruction or description changes;
- permissions or target user groups change;
- a new high-consequence question enters scope;
- Microsoft changes a preview capability or documented behavior.
Store the evaluation result with the model/configuration version so a passing result cannot be applied to a different state.
Microsoft product facts vs. Refinity recommendations
Microsoft product facts
- Copilot output is nondeterministic.
- Power BI Copilot answers can include generated visuals and summaries.
- Microsoft recommends known benchmarks, trusted comparisons, and before/after testing.
- Prep data for AI controls have capability-specific behavior and limitations.
- Permission context affects the data a user can access.
Refinity recommendations
- Define an answer contract for every consequential golden question.
- Score semantic correctness separately from generated language.
- Make security, material numeric errors, and undefined metrics blocker overrides.
- Repeat critical tests and evaluate meaning rather than exact wording.
- Version results with the model and AI configuration.
Primary CTA: Review the Power BI Copilot Readiness assessment deliverables, including a golden-question evaluation set.
Related reading: