ALHENA RESEARCH LAB · METHODOLOGY
What the scores measure.
New: shopping across the storefront
shopping-journey-v2 adds six shopping outcome tasks and a separate measure of usable shopping reach across chat, product-page Q&A, recommendations, search and cart assistance. More distinct, verified usable interfaces earn reach credit; duplicate widgets do not. Outcomes and reach remain separate scores.
The framework is published, with no results under this version yet. Automated capture across these interfaces is not enabled. Current submissions and published scores retain their existing method; support scoring is unchanged.
Explore the shopping journey frameworkThe current research library uses policy-resolution-v1. Shopping and support composite scores combine three distinct measures: policy-compliant resolution, answer quality and full-answer speed. Studies test one form factor, the AI agent in a storefront’s on-site chat widget; helpdesks, email, SMS, voice, search, product-page content and AI visibility are outside their scope.
Policy-compliant resolution
Did the assistant give a correct answer or the actionable next step required by the merchant’s published policy? A required handoff can earn credit when the response fulfills the request under that policy.
Answer quality
The original 26 published quality criteria, with fixed weights: 16 for shopping and 10 for support. Quality is a separate component, not the composite score.
Full-answer speed
Time until the full answer is complete. The speed score ranges from 100 at three seconds to zero at 22 seconds, bounded between zero and 100.
Evidence and audit
Every assessed resolution checkpoint receives a primary judgment and a separate blind audit. Both must award valid evidence-backed attainment for verified credit. Disagreements and exclusions remain visible.
How the composite is calculated
| Lane | Policy-compliant resolution | Answer quality | Speed |
|---|---|---|---|
| Shopping | 40% | 35% | 25% |
| Support | 50% | 40% | 10% |
Overall is the equal mean of the two unrounded lane composites, available only when both lanes meet the published coverage requirements.
Sample and coverage
Each provider has at least 5 registered storefronts and ten core themes per storefront. Headline lane composites require at least 15 eligible conversations and 3 storefronts with resolution observations. Each report shows the actual included and excluded sample.
These are selected public storefront sessions. Resolution can include a policy-prescribed next step; it does not establish a completed refund, account change or human-resolved case. Alhena commissions and operates this research.
Current submissions and the historical pilot
The earlier quality-pilot-v1 reports retain their original scores and dates in the historical archive. They cover three storefronts and two question themes per tool, measuring quality only. Their scores are not combined with the current study’s composites.
New “Analyze your tool” submissions use the current five-storefront policy-resolution method. After approval, the runner captures all ten core themes and a separate guardrail conversation per storefront. Validated results update the current library automatically; older quality-pilot jobs retain their frozen scope.