Marketing and Sales teams can now upload data and ask AI questions such as:
“Which Channel grew fastest?”
“Which Customer Segment has the highest Retention?”
“Did this Promotion increase Sales?”
‍
The speed is valuable.
The challenge is that an incorrect answer does not necessarily look incorrect.
An AI system may produce a clean table, use the right terminology and provide a confident explanation even after selecting the wrong Column, using the wrong Base or inventing a reason that does not exist in the data. NIST specifically notes that Generative AI may generate plausible reasoning that supports an answer even when the answer itself is wrong.

The question after AI analysis should therefore not be only: “Does this result look reasonable?”
It should be: “If we had to sign off on this number, could we trace it back to the Source Data?”

Do not only check whether AI calculated correctly, check whether it calculated the right thing
AI errors in Data Analysis go beyond arithmetic mistakes.
An output may fail because the AI:
• Invents a Fact or Source
• Uses the wrong Column or Definition
• Selects the wrong Denominator
• Applies an incorrect Formula
• Calculates correctly but interprets incorrectly
• Converts Correlation into Causation
• Generalizes from one Segment to the entire Customer Base
• Produces a plausible Explanation that the data never established
NIST uses the term Confabulation for Generative AI outputs that confidently present erroneous or false content, including what is commonly called Hallucination. NIST also notes that models can generate misleading logic or citations that appear to justify an incorrect answer. A useful AI QA sequence is: Source → Definition → Calculation → Reconciliation → Interpretation → Decision

First, separate Hallucination from Calculation Error

A Hallucination occurs when AI generates a Fact, Explanation, Reference or Claim that is not supported by the available evidence or source.
A Calculation Error occurs when the numerical operation is wrong, such as incorrect arithmetic, Formula use or aggregation.

Business Analysis introduces another important category: Interpretation Error
For example:
Data:
Campaign A Conversion Rate = 5.2%
Campaign B Conversion Rate = 4.6%

AI concludes: “Campaign A performed better because its Creative was stronger.”
The rates may be calculated correctly.
But “because its Creative was stronger” is an Explanation requiring additional evidence.
Therefore: Correct Number ≠ Correct Interpretation

AI can be confidently wrong

OpenAI notes that Language Models can generate incorrect or misleading answers while sounding confident, including fabricated Quotes, Studies, Facts or Citations.
Do not treat confident language as confidence in the evidence.
Statements such as:
“The data clearly proves...”
“The main cause is...”
“This strongly indicates...”
‍
do not make the underlying Claim stronger.
Instead ask:
Where did the number come from?
How was it calculated?
What evidence supports the Explanation?
What Alternative Explanations remain possible?

Start QA with Source Grounding before checking the Formula

Before checking arithmetic, confirm that the AI analyzed the right data.
Suppose you upload 2025–2026 Sales Data and ask: “How much did Q2 2026 Sales grow year-on-year?”

Verify that the system compared: Q2 2026 with Q2 2025
not: Q2 2026 with Q1 2026
or: H1 2026 with Q2 2025
Check:

• Correct File
• Correct Sheet
• Complete Rows
• Correct Filters
• Correct Period
• Correct Currency and Units
• Correct Column Definitions

A perfect Formula applied to the wrong subset is still a wrong answer.

Check the Denominator whenever you see a Percentage or Rate

One of the most dangerous errors is not dividing incorrectly.
It is dividing correctly by the wrong Base.
Repeat Purchase Rate could mean: Returning Customers / All Customers
or: Customers making a Second Purchase / New Customers eligible to repurchase
or: Repeat Transactions / All Transactions

These are different Metrics answering different questions.
Whenever AI reports:
Retention = 68%
Conversion = 4.2%
Complaint Rate = 3.5%
ask:
What is the Numerator?
What is the Denominator?
What is the Time Window?
Who is Included and Excluded?
What is the Metric Definition?
If those questions cannot be answered, the number is not ready for a Dashboard.

Do not let AI silently invent Business Definitions

Suppose a Dataset contains:
Registered User
Active User
Paid User
Returning User

The Prompt asks: “Are real users increasing?”
The AI may decide that “real users” means Active Users.
But the term may not yet have an agreed Business Definition.
A stronger approach is: Active User = User with at least one Completed Transaction within 30 days

Definitions should be explicit for Metrics such as:
Active Customer
Churn
Retention
Engagement
High-value Customer
Qualified Lead
Conversion
A calculation can be mathematically correct and still answer the wrong Business Question when its Definition is wrong.

Ask AI to expose the Formula, then verify it independently

If AI reports: Revenue Growth = 18.4%
ask it to show:
Current Revenue
Previous Revenue
Formula
Result

For example: Revenue Growth = (Current Revenue − Previous Revenue) / Previous Revenue × 100
Then independently recalculate the result using Excel, a Calculator, SQL or another computation method.

OpenAI notes that Data Analysis and Code Execution tools can support more accurate calculations and structured reasoning than relying only on generated prose, while still recommending verification of important information.
The core principle is: AI-generated calculations should be reproducible.
If the number cannot be recreated from the Source Data, it has not passed QA.

Use Reconciliation Checks to detect errors that individual numbers can hide

Suppose AI reports Sales by Channel:
Website = 4.2 million baht
Marketplace = 3.1 million baht
Store = 2.5 million baht
Total = 9.8 million baht

But the Source Report says Total Revenue = 10.4 million baht.
Stop before interpreting performance.
The gap may come from:

• Missing Channel
• Refund Treatment
• Tax
• Duplicate Removal
• Filtering
• Currency Conversion
• Unallocated Sales

A simple rule is: Breakdown Total should reconcile to the Master Total under the same Definition.

Spot-check real rows instead of reviewing every record manually

A Dataset with 100,000 Rows does not require manual review of every row.
Choose cases that are useful for detecting errors:

• Highest-value Transaction
• Negative Transaction
• Highest-frequency Customer
• Missing-value Cases
• First and last date in the Period
• Segment AI identifies as unusual
• Outliers

If AI says Customer C1082 purchased 17 times, check the records for C1082.
If the source contains only nine purchases, the problem may involve a Join, Duplicate or Customer ID Mapping issue.
A Spot Check does not prove that everything is correct.
It helps identify whether the Analysis Process contains a serious Red Flag.

Watch for explanations that AI adds to patterns

Suppose the Data shows: Sales declined in April.
AI writes: “Sales declined because customers became more cautious about spending.”

Does the Dataset contain Consumer Confidence or Spending Sentiment Data?
If not, this is a HYPOTHESIS, not a FACT.

A safer version is:
FACT: Sales in April were 12% lower than in March within this Dataset.
BEE INTERPRETATION: The decline should be examined by Customer, Product and Channel.
HYPOTHESIS: Possible explanations include Seasonality, Promotion Timing, Traffic, Stock Availability or Consumer Spending.
UNKNOWN: The available data does not establish the Cause.
This separation makes AI-generated analysis safer for Business Decision use.

Correlation requires extra QA

Suppose AI finds: Higher Customer Satisfaction is associated with higher Spend.
It then concludes: “Increasing Satisfaction will increase Customer Spend.”
That conclusion goes beyond the Evidence.
‍
Association ≠ Causation
‍
Alternative Explanations include:
High-spending customers receive different service.
High-value Customers use different Products.
A third Segment characteristic drives both Satisfaction and Spend.
The direction of the relationship may differ from the assumed explanation.
A useful QA instruction is: “Separate what the data demonstrates from causal explanations that remain unproven, and list plausible Alternative Explanations.”

Check Segments before trusting the Average

AI reports: Average Satisfaction = 8.1
and concludes that Customer Experience is strong.
‍
But segmented results show:
New Customers = 6.4
Returning Customers = 8.8
Premium Customers = 9.0
The average is mathematically correct. The conclusion may still be too broad.
Before turning an Average into a Business Action, check:

• Customer Segment
• Channel
• Product
• Region
• Customer Type
• Time Period
• Promotion vs Full Price

Check Units and Scales separately from Formulas

Common errors include:
0.18 vs 18%
Baht vs Thousand Baht
Revenue vs Gross Profit
Monthly vs Annual
Per Customer vs Per Transaction
1–5 Scale vs 0–10 Scale
Seconds vs Minutes

A calculation may process the values exactly as provided while the underlying Unit has been misunderstood.
Formula Check and Unit Check should therefore be separate QA steps.

Asking the same AI “Are you sure?” is not independent verification

A common QA approach is: “Are you sure?”
The AI replies: “Yes, I have verified the calculation.”
That is not an Independent Check.

The same system may reproduce the original mistake or generate a new explanation for it.
NIST explicitly warns that Generative AI can generate plausible logic that appears to justify an incorrect output.
Stronger checks include:
AI Calculation → Excel Recalculation
AI Summary → Source Row Check
AI Claim → Original Source
AI Percentage → Numerator / Denominator Reconstruction
AI Trend → Raw Time Series
AI Correlation → Statistical Output and Design Review

AI can help review its own work, but it should be treated as a QA Assistant rather than the final Auditor

AI remains useful during Review.
For example: “Review this analysis for Wrong Denominators, Duplicates, Missing Data, Unit Errors, Percentage Errors, Causal Claims and Unsupported Explanations. Identify issues before proposing corrections.”
Or: “For every conclusion, show the Source Column, Formula, Population, Time Period and Evidence Limitation.”
Or: “Identify every statement that cannot be traced to the supplied data.”

This changes the role of AI from: Answer Generator to: Analysis + Critique Assistant
Final accountability should remain with the person or process responsible for the decision, particularly for consequential actions. NIST and Microsoft's Responsible AI guidance emphasize Reliability, Transparency and appropriate Accountability in AI use.

Example: AI says a Promotion increased Sales by 24%

Suppose AI compares:
Sales during Promotion = 1.24 million baht
Sales before Promotion = 1.00 million baht
and concludes: “The Promotion increased Sales by 24%.”

The calculation: (1.24 − 1.00) / 1.00 = 24% is correct.
But the Business Conclusion has not yet passed QA.
Check:

• Are the Periods the same length?
• Are the number of selling days comparable?
• Is Seasonality similar?
• Is Store / Channel Coverage comparable?
• Did Stock Availability change?
• Were other Campaigns running?
• Did Customer Mix change?
• What happened to Gross Margin?
• What would Sales have been without the Promotion?

A better evidence-calibrated conclusion is:
FACT: Sales during the Promotion Period were 24% higher than during the comparison period.
UNKNOWN: This Before-After Comparison alone does not establish that the Promotion caused a 24% Incremental Sales Lift.
A calculation can be correct while the causal conclusion is wrong.

Example: AI says Segment A is the best customer group

AI identifies Segment A as having: The highest Average Spend
and recommends: “Allocate more Marketing Budget to Segment A.”

Before accepting the Recommendation, check:
Sample Size
Margin
Purchase Frequency
Retention
Cost-to-Serve
Promotion Dependency
Acquisition Cost
Growth Potential

If Average Spend is driven by only three unusually large customers, the decision may change substantially.
AI QA therefore does not end when the number is correct.
Ask whether the Metric is sufficient for the Decision.

Match QA intensity to Decision Risk

Low-risk Decision
Example: Internal Draft Summary or Initial Pattern Exploration
Source, Formula and Spot Checks may be sufficient.

Medium-risk Decision
Example: Campaign Selection, Budget Allocation or Customer Targeting
Add Independent Recalculation, Reconciliation, Segment Checks and Human Review.

High-risk or Hard-to-reverse Decision
Example: Major Investment, Contract, Pricing Change or Decisions that significantly affect people
Use stronger Verification, Independent Data or Code Review, clear approval responsibility and an Audit Trail of Data, Definitions and Assumptions.

NIST notes that Confabulation Risk is particularly important to monitor when Generative AI is used in consequential decision-making contexts.
The principle is: Higher Decision Risk → Higher Evidence and QA Standard

AI Analysis Checklist before sending results to management

1. Source Check
   Can every important number be traced to Source Data?
2. Definition Check
   Does every Metric have an explicit Definition?
3. Base Check
   What are the Numerator and Denominator behind each Rate?
4. Calculation Check
   Can important numbers be independently reproduced?
5. Reconciliation Check
   Do Subtotals reconcile to the Total?
6. Segment Check
   Is an Average hiding an important subgroup difference?
7. Interpretation Check
   Has an Association been written as a Cause?
8. Evidence Check
   Did AI invent any Fact, Benchmark, Source or Explanation outside the Input?
9. Limitation Check
   What can this Dataset not answer?
10. Decision Check
    If the conclusion is wrong, how large is the consequence and does the QA standard need to increase?

The takeaway: Use AI for speed, but separate Speed from Trust

Generative AI can accelerate many analysis tasks, from Summarization and Coding to Table Creation and Initial Pattern Detection.
Speed does not make the output verified evidence.

NIST documents that Generative AI can produce confident Confabulations, including incorrect logic and citations, while OpenAI advises users to verify important information rather than treating confident model output as automatically reliable.
A practical Data Analysis rule is: Trace → Recalculate → Reconcile → Challenge → Label → Decide

Trace the output to its Source.
Recalculate important numbers.
Reconcile Breakdowns with Totals.
Challenge Explanations and Causal Claims.

Label FACT, BEE INTERPRETATION, HYPOTHESIS and UNKNOWN.
Then make the Decision.
AI does not need to be excluded from Data Analysis.
But it should not be allowed to skip QA simply because its answer looks professional.