All evaluated tools

LATEST APPROVED STUDY · policy-resolution-v1

Alhena

In its latest study (captured Sep 21–22, 2026), Alhena scored 68.4 for shopping and 81.4 for support, as composite scores out of 100. Full answers arrived in 11.8 s for shopping and 8.7 s for support on average.

alhena.ai

Captures Sep 21, 2026 to Sep 22, 2026. Published Sep 22, 2026. 5 registered storefronts. Selected by capture end date, then publication date.

Public-session scope

Policy-compliant resolution includes a verified policy-required next step. These scores do not prove completed refunds, account actions or human-resolved orders. Coverage differs by lane and provider; this is an Alhena-commissioned study, not independent certification or a universal vendor ranking.

Read the study and evidence Read the methodology

Source: Alhena vs. Gorgias: which AI agent gets shoppers the right answer?. Summaries are public; detailed evidence requires a verified work email.

Beyond the chat widget

These scores cover Alhena’s AI agent in the on-site chat widget only. Alhena also publicly lists the following, which the Lab has not tested (checked ):

Shopping

Composite

68.4 / 100

40% policy-compliant resolution, 35% quality, 25% speed.

Policy-compliant resolution · public-session scope

64.8 / 100

Verified correct answer or policy-prescribed next step, averaged equally by conversation and storefront. This does not measure downstream case completion.

Answer quality

83 / 100

The original fixed answer-quality rubric. Unchanged captures retain their original judgments; repaired captures receive a fresh judgment and audit.

Full-answer speed score

53.7 / 100

Full-answer completion 11.8 seconds; 100 at 3 seconds, zero at 22 seconds, bounded between 0 and 100.

Support

Composite

81.4 / 100

50% policy-compliant resolution, 40% quality, 10% speed.

Policy-compliant resolution · public-session scope

72.8 / 100

Verified correct answer or policy-prescribed next step, averaged equally by conversation and storefront. This does not measure downstream case completion.

Answer quality

95 / 100

The original fixed answer-quality rubric. Unchanged captures retain their original judgments; repaired captures receive a fresh judgment and audit.

Full-answer speed score

70 / 100

Full-answer completion 8.7 seconds; 100 at 3 seconds, zero at 22 seconds, bounded between 0 and 100.

Overall composite

Overall composite

74.9 / 100

Equal mean of the two unrounded lane composites; only available when both lanes meet coverage floors.

Evidence coverage

Registered storefronts describe the cohort. Included storefronts and assessed checkpoints describe usable policy-resolution evidence. Unknown and unsent checkpoints are not presumed successes or failures.

LaneCheckpoints: planned / attempted / observedRecorded submitted / assessed / unassessableAttained / unverifiedContexts: included / excludedStorefronts: included / registeredQuality eligible: contexts / storefrontsCaptures: original / repaired
Shopping250 / 242 / 242242 / 242 / 0158 / 6425 / 05 / 525 / 523 / 2
Support250 / 250 / 250250 / 250 / 0182 / 6325 / 05 / 525 / 519 / 6
Study scope and limitations

Alhena commissioned and operates this study. Selected storefront deployments do not establish a universal provider ranking or independent certification.

This methodology was developed after reviewing the original automation results, then frozen before the new policy-resolution judgments.

The sessions were logged out. No real order, refund, account change or completed human resolution was independently verified.

Unknown delivery or actor attribution remains unassessable. Unsent questions are not successes or failures. Conditional scores must be read with planned and assessed coverage.

All ten JSHealth core contexts were excluded because a generative actor and delivery could not be confirmed. Two additional support contexts stopped at login gates.

Two retained dryrobe shopping contexts have only six measured replies. Twelve cause-selected capture repairs contain all 120 submitted questions; every repaired outcome is retained.

Merchant policy conflicts and gaps are retained. Product efficacy, review authenticity and downstream cart or refund completion were not independently established.

The repaired captures have gaps in live resource telemetry. Terminal state and artifact hashes were verified; continuous resource observations are not claimed.

Private cart, session or customer links are redacted from displayed evidence and quotations. Judgments used the unchanged original captures; these display redactions do not change scores.

Audit scope

Every assessed policy-resolution checkpoint receives a primary and a fresh blind audit. Both must award valid evidence-backed attainment. Attainment disagreements receive zero verified credit and remain visible.

Provider names were masked where practicable; full anonymity is not claimed.

Retained original quality judgments have separate historical audit coverage. The repair-quality audit sees the primary quality verdicts and full captured replies; it is distinct from the blind policy-resolution audit.

One initial scoring attempt failed in the response decoder with zero accepted decisions; upstream completion is unknown. The output was unavailable, and the documented infrastructure repair did not select among verdicts.

The first blind policy-resolution audit exhausted its 8,000-token per-model-turn output allowance without a structured verdict. Its unusable output was preserved and the identical input was audited once more under the reviewed 16,384-token per-model-turn allowance.

A later blind audit failed after about 602 seconds without a returned verdict. This is consistent with the original 600-second local timeout, but the inner error was not retained and provider completion is unknown. The unusable attempt was preserved and the identical input was retried once under a uniform 1,200-second future execution limit with a 1,250-second outer transport limit. All 36 previously successful stage outputs remain unchanged under their original timeout contracts.

The exact successful initial primary judgment for 30 checkpoints is retained with an 8,000-token per-model-turn allowance. Every subsequent policy-resolution and repair-quality call uses 16,384 per model turn, with the same frozen prompts, model, effort and criteria. New calls use fresh independent sessions under reviewed pool releases with 2 and 4 active slots; 4 distinct isolated slots were used. A CLI session can contain multiple internal steps, so its total output can exceed the per-turn allowance.

The original quality audit sampled only Alhena records. New policy-resolution audits cover both providers, but this does not retroactively expand historical quality-audit coverage.

Historical quality-pilot archive · Sep 20, 2026

This earlier sample retains its original quality scores and dates. Its quality-only scores are not the latest study composites or quality components.

HISTORICAL QUALITY PILOT · quality-pilot-v1

Alhena quality pilot

Shopping and support quality across three selected customer storefronts, using the same 26 published criteria.

alhena.ai
Refresh needed

SHOPPING QUALITY

96/ 100

Mean of three storefront conversations.

SUPPORT QUALITY

100/ 100

Mean of three storefront conversations.

3Storefronts
6Conversations
60Captured turns
26Published criteria

Original capture dates: Sep 20, 2026. These results remain available as historical evidence. A fresh evaluation is needed before creating new comparisons.

Summaries are public. Detailed conversations and scoring decisions require a verified work email.

BEHIND THE TOOL SCORE

Three deployments. Every result visible.

Customer storefrontShopping /100Support /100Captured
Tatcha98100Sep 20, 2026
Paula's Choice95100Sep 20, 2026
Benchmade95100Sep 20, 2026

HISTORICAL COMPARISONS

Alhena quality-pilot reports

Reports retain their original capture dates and source evaluations.

QUALITY COMPARISONPublished Sep 20, 2026

Alhena vs. Gorgias

Shopping and support quality across six live storefronts, scored against Gorgias’s published criteria.

6 storefronts120 captured turns26 criteria
Explore the report

Original capture dates and limitations are retained. Read the evidence before interpreting the scores.

QUALITY SCORES / 100

Shopping

Alhena96
Gorgias76.7

Support

Alhena100
Gorgias82.7
Scope and limitations

Operated and commissioned by Alhena Research Lab. Scores describe this selected sample, not universal tool performance or an overall vendor ranking.

Small selected sample: three stores per provider and one conversation per lane/store; not a randomized or matched-store study.

Different stores, catalogs, configurations, browser sessions and current dates limit causal vendor comparisons.

Only two fixed themes were run, not all five per lane or the separate guardrail battery.

Observed response completion substitutes for timed-answer eligibility for quality-only grading. No reliable live latency, automation/speed composite, statistical superiority or official rank is claimed.

Three conversations per vendor/lane are below the published 15-judged-conversation threshold.

Alhena lanes used New Chat in a shared profile. Paula’s Choice claimed preferences not supplied by the script; the record cannot distinguish retained context from unsupported assumptions. Cold isolation was not established.

Store/vendor masking was limited; URLs and product names could reveal identity.

Primary and audit passes were AI agents, not external independent researchers. Audit used full judging packets rather than upstream 450-character tails.

Policy and link checks are separately disclosed; no extra scoring criteria or penalties were introduced. A 100 quality score does not certify factual perfection.

Historical current-rubric regrading cannot reconstruct an unrecorded historical model/prompt version and does not prove manipulation.

Some conversations are reused unchanged from published reports. Their original capture dates and source limitations remain applicable; they were not recaptured or rejudged for this comparison.

Historical reused captures inherit the original published source attribution. Per-turn AI-author proof was not recorded by that source and has not been invented or newly verified.

Read the scoring rubric and methodology