An AI product analyst grounded in analytics evidence

An LLM can summarize patterns and propose investigations, but only if the analytics system gives it a clear evidence contract. Sending a screenshot or an unrestricted database connection creates two opposite risks: the model may miss the definitions behind the chart, or it may receive far more sensitive data than the question requires.

MetricFold exposes bounded JSON reports and MCP tools for acquisition, funnels, friction, lifecycle, revenue, performance and traffic quality. Each report carries its time window, filters, counts and relevant caveats. The model can reason over evidence while the application retains authorization and query control.

What the model needs to know

A useful AI report includes more than metric names. It should define the denominator, comparison window, minimum sample, identity boundary and trusted-event status. “Checkout conversion fell 20%” is ambiguous without knowing whether conversion is per visitor or session, whether the comparison is seasonally meaningful, and whether the purchase came from a verified webhook.

MetricFold's insight builder returns structured findings with severity, evidence, affected segment and suggested next check. It can surface examples such as:

• a CTA with substantial exposure and unusually low activation;
• a funnel step with both high loss and increased errors;
• a device cohort with degraded Core Web Vitals and lower conversion;
• failed payments with little recovery;
• a subscriber cohort with rising churn;
• trial starts that are not reaching the activation event;
• a source that creates traffic but very little retained value.

These are leads for investigation. The system does not claim an observed correlation is causal.

Friction requires several kinds of evidence

UX friction is rarely represented by one event. A useful model combines repeated clicks, stalled actions, validation failures, JavaScript errors, slow interactions, excessive backtracking, support openings and abandonment around a meaningful step. It should distinguish an intentional exit from a failed attempt where the user repeatedly tried to progress.

MetricFold avoids unrestricted session replay by default. Replay can be helpful for some teams, but it carries a larger privacy and security surface and is difficult for an LLM to search reliably. Structured friction signals are smaller, comparable across releases and safer to include in automated analysis. A team that needs replay can connect a specialist system and keep it outside the default collection boundary.

CRO recommendations need outcome depth

A landing-page change that increases clicks but reduces qualified activation is not an improvement. CRO evidence should follow the journey far enough to show signup quality, time to value, retained use and trusted revenue. The AI can compare source and landing-page cohorts across those downstream outcomes and suggest the smallest experiment that resolves an uncertainty.

A strong recommendation has five parts: observation, evidence, plausible explanations, proposed change and success guardrail. For example, “Mobile pricing CTA click-through is lower” is an observation. “The CTA has 4,200 qualified impressions, a 1.8% click rate versus 4.9% desktop, and mobile interaction latency is poor” is evidence. The model can then suggest testing layout or performance while guarding completed signup and paid conversion.

MCP and API boundaries

MetricFold's MCP surface can register a site, retrieve an installation manifest and read authorized summaries. It never gives the model raw credentials, connector tokens or an unrestricted SQL tool. API keys are workspace-scoped, hashed at rest, permissioned by ability and revocable. A mobile client can use the same REST contracts without needing model-specific access.

Write actions need a separate approval boundary. The AI may draft a tracking-plan change, report filter or experiment hypothesis. It should not publish content, change billing, message customers or alter a live collector without an authorized application action.

Protecting against prompt injection in analytics data

Event properties, URLs and external connector labels are untrusted data. They may contain text that looks like an instruction. Treat them as data fields, not model messages. Use fixed system instructions, structured schemas, length limits and field allowlists. Exclude DOM text, form input and raw support messages from the default analytics prompt.

The same principle applies to connected APIs. MetricFold extracts only a configured numeric JSON path from custom metric endpoints and never passes the remote document to the dashboard or model. Social connectors normalize numeric metrics before storage.

A practical AI analysis loop

1. Select one business question and reporting window.
2. Request the smallest relevant reports.
3. Check data quality, sample and definition warnings.
4. Rank opportunities by affected users, commercial importance and confidence.
5. Ask the model for competing explanations, not only one story.
6. Turn the best uncertainty into an experiment or instrumentation repair.
7. Record the decision and compare the same metric contract after the change.

MetricFold's role is to make this loop reproducible. The evidence can be exported as JSON or CSV, streamed to an application, or read through MCP. The dashboard and model use the same underlying report definitions, so a human can verify what the AI saw.

The best AI analytics feature is not a chat box attached to every chart. It is a disciplined interface between definitions, trusted evidence and a model that can help a team form better tests.

Design the evidence packet before the prompt

The reliable unit of AI analysis is a versioned evidence packet, not a long natural-language request. It should contain the business question, metric definitions, window, comparison, filters, sample sizes, truncation flags and a compact set of ranked rows. The Model Context Protocol specification provides a standard way to expose bounded tools, but the product still owns authorization and the meaning of every returned field.

Separate facts, calculations and hypotheses

Return observed counts as facts. Return conversion, lift or percentile values with their formulas and denominators. Put model suggestions in a different field with confidence and competing explanations. This structure lets a human reject an interpretation without disputing the underlying data.

For a funnel finding, include entrants, completions, per-step actors, loss, comparison values and the exact step definition. For a friction finding, include affected sessions, eligible sessions, device/page concentration and overlapping errors or performance evidence. The funnel analytics guide and friction analytics guide define those evidence contracts in more detail.

Protect against silent query expansion

A model should not turn “why did checkout fall?” into a full data export. Give each tool a maximum reporting window, allowed dimensions, maximum ranked rows and workspace/site boundary. Return an explicit truncation flag when a query reaches a limit. Log tool name, scoped key, timing and result size without logging the sensitive payload.

OpenAI's structured outputs guidance illustrates why schema-constrained responses are easier to validate than prose. The same principle applies before the model call: provide a typed analytics schema so the application can reject an impossible window, unknown metric or unsupported filter.

Evaluate recommendations like a product feature

Create a fixed set of historical analytics cases with known evidence and acceptable responses. Include low-sample funnels, seasonality, bot spikes, delayed webhook events, Simpson's paradox across segments, a performance regression and a result where no action is justified. Score whether the model preserves the denominator, notices uncertainty, cites the supplied evidence, avoids causal language and proposes a reversible next check.

Use holdouts and human adjudication

Do not train on every accepted recommendation. Operators may accept a suggestion because it sounds plausible, not because it improved an outcome. Retain the evidence packet, human decision and later experiment result as separate records. Use adjudicated cases for regression testing and model changes; use successful experiments only as examples of a good decision process, not universal rules.

Monitor drift in tool selection, token use, unsupported claims and the rate of “insufficient evidence” answers. A model that always produces a confident recommendation is less trustworthy than one that stops when the packet is sparse.

Frequently asked questions

Can the AI change a live tracking plan automatically?

No. It can draft a proposal with missing evidence and expected impact. An authorized human or application workflow validates the schema and publishes the change. Billing, messaging, content publication and other irreversible actions remain outside the read-only analyst boundary.

Does the model receive raw visitor journeys?

MetricFold returns bounded event sequences and aggregate reports authorized for the site. It does not send DOM contents, form values, raw IP addresses, connector credentials or an unrestricted event table. Product teams can further restrict tool abilities per key.

Which model should be used?

Choose based on the evaluated task, latency, data policy and cost rather than a universal ranking. Keep the analytics tool contracts model-neutral so a new model can be tested against the same cases before it is enabled.

Where should a team begin?

Start with one recurring question such as activation loss or payment recovery, define its evidence packet, and run the answer alongside the human review. The broader product analytics system should be reliable before adding more AI surface area.