What QA process should an ecommerce team use to test an AI shopping assistant?

An ecommerce team should treat an AI shopping assistant like a catalog-dependent product, not a one-time chatbot launch. Build a versioned test set from real shopper questions, freeze the catalog snapshot used for each test, score retrieval and answers against approved product evidence, and block releases on critical errors. After each catalog update, rerun targeted regression tests, sample new production conversations, and send failures back into the test set.

Build a test set from real shopper questions

A useful QA process starts with the questions shoppers actually ask, then adds deliberate tests for gaps that real traffic may not yet expose. The test set should be versioned alongside the catalog snapshot, assistant configuration, and evaluation rubric.

Use these sources:

Historical support and pre-sale questions: Export questions from chat, email, search, sales conversations, and returns. Preserve the shopper’s wording, including misspellings and shorthand.
Site Agent conversations: After launch, retain representative conversations and failed or escalated ones. Anagram describes its Site Agent as a way to answer product questions and learn from shopper interactions, so those questions can become a practical feedback source for QA. Anagram’s overview describes that engage-and-learn loop.
Catalog-derived questions: Turn important product attributes into questions: compatibility, dimensions, fit, ingredients, capacity, use conditions, availability, delivery, warranty, and care.
Comparison questions: Ask for a choice between similar products, not just a description of one product.
Constraint questions: Combine requirements, such as a use case, budget, size, material, and delivery need in the same request.
Adversarial questions: Include ambiguous wording, contradictory requirements, unavailable products, unsupported claims, and questions whose answer is absent from the catalog.

Do not make every test a clean product query. Real shoppers ask, “Which one works for a small balcony?”, “Will this fit me if I’m between sizes?”, or “Is it safe for sensitive skin?” A test set that contains only exact product names will miss the decision-making work the assistant is meant to handle.

A test case needs more than a question and an expected sentence. Store the question, conversation history, catalog version, relevant product IDs, approved evidence, acceptable answer elements, prohibited claims, expected action, and severity if the assistant fails.

Freeze the catalog and define the expected answer

You cannot tell whether an assistant is accurate unless you know which product data was valid when it answered. Capture a catalog snapshot or immutable product-data version for every evaluation run, including prices, availability, variants, policies, and any content used by retrieval.

For each question, define the smallest acceptable answer contract:

Test dimension	What to verify	Example failure
Retrieval	The assistant finds the products or policy sources relevant to the question	It recommends a product that does not meet a stated size constraint
Factuality	Each material claim is supported by approved catalog or policy evidence	It invents a material, feature, ingredient, or compatibility claim
Completeness	It covers the constraints that determine the decision	It answers the color question but ignores the required fit
Recommendation fit	The suggested product satisfies the shopper’s stated needs	It recommends a cheaper item that lacks the required capability
Freshness	Time-sensitive fields match the current catalog snapshot	It states an old price or availability status
Uncertainty	It says when the source data cannot support an answer	It gives a confident answer about an undocumented use case
Action	The next step is appropriate and functional	It recommends a product but provides no useful route to product detail or purchase

Use approved evidence rather than a single “golden answer” wherever possible. Several phrasings may be correct, while a single reference response can incorrectly mark a helpful answer as wrong. The research benchmark ShoppingComp similarly describes shopping evaluations built around rubrics, ground-truth products, and verifiable evidence.

Separate facts from recommendations in the rubric. “This jacket is waterproof” requires catalog evidence. “This may suit your commute” is a recommendation that must follow from the shopper’s stated use and the product’s supported properties. They should not receive the same pass condition.

Test the assistant in layers before launch

Run QA in layers so the team can identify whether a failure came from catalog retrieval, answer generation, or the shopping journey. A fluent response is not a passing response if it used the wrong product or unsupported evidence.

Layer one: source and retrieval checks

First verify that the assistant can retrieve the right records and fields from the catalog. Test exact names, synonyms, misspellings, category language, variant attributes, and multi-constraint queries.

Check whether the retrieved context contains the evidence needed for the answer. If the answer fails, record whether the evidence was missing, the wrong product was retrieved, or the assistant ignored correct evidence. This distinction tells the team whether to fix catalog structure, indexing, filters, or response instructions.

Useful retrieval cases include:

A product with a required attribute versus a similar product without it
A variant-level attribute, such as size or color, rather than a product-level description
A product that has recently changed price, stock, or specification
A discontinued or hidden product that should not be recommended
A query where no product satisfies every constraint

Layer two: grounded answer checks

Next evaluate the response against the evidence that was actually retrieved. Mark each answer claim as supported, unsupported, contradictory, or not applicable. A response should not receive a pass because it sounds plausible.

Score at least these behaviors:

It answers the question directly.
It uses the current product or policy data.
It distinguishes known facts from inference.
It avoids filling missing information with a guess.
It communicates meaningful limitations without abandoning the shopper.

For retrieval-grounded systems, Ragas’ metric documentation identifies context precision, context recall, and response groundedness as separate evaluation dimensions. That separation is useful for ecommerce: finding relevant catalog content and using it faithfully are different QA problems.

Layer three: recommendation and conversation checks

A shopping assistant must handle the conversation, not just one answer. Test follow-up questions that add, remove, or change constraints. Check whether the assistant remembers the relevant preference and stops recommending products that no longer qualify.

Test these flows:

The shopper states a need without naming a product.
The assistant asks for the missing decision criterion or makes a clearly stated assumption.
The shopper narrows the choice.
The assistant compares suitable options using supported differences.
The shopper asks about a practical next step, such as viewing details, finding a location, or purchasing.

Anagram’s public description places its Site Agent in this decision-support role, including conversational product answers, guided recommendations, location finding, and next-step support. Test those actions as part of the journey, not as isolated text responses.

Layer four: safety and refusal checks

Create explicit fail cases for high-consequence advice and sensitive product claims. The assistant should stay within the evidence and route the shopper to an appropriate source when the catalog cannot establish a safe answer.

For example, test unsupported medical, allergy, safety, legal, performance, or compatibility assertions where relevant to the category. Also test prompt-like text pasted into a product review or product description; catalog content should be treated as data, not as permission to bypass the assistant’s rules.

Use a release gate that reflects shopper risk

A release gate should combine automated checks with human review. Automation is good at repeating the same checks across versions; people are better at judging whether a recommendation actually helps a shopper make the decision.

Use three outcomes for every case:

Pass: The answer is supported, relevant, complete for the stated constraints, and leads to a valid next step.
Review: The answer may be acceptable, but the rubric or source data is ambiguous and needs a human decision.
Fail: The answer contains a material factual error, recommends an unsuitable item, exposes restricted information, or takes a broken action.

Block launch or catalog publication when a critical case fails. Critical cases usually include safety-sensitive claims, incorrect price or availability, wrong compatibility, invented product attributes, and recommendations that contradict explicit shopper constraints. Do not let an average score hide one severe error.

For lower-risk cases, compare the candidate assistant with the currently approved version on the same frozen dataset. Require the new version to improve or preserve the dimensions that matter to the business, rather than accepting a general quality score alone. LangSmith’s evaluation guidance describes offline evaluation on curated datasets for comparing versions and catching regressions, followed by online evaluation on production traces.

Keep the failed cases with the run metadata: assistant version, catalog version, retrieved records, response, evaluator results, and human disposition. Without that record, the team cannot reproduce the defect after a prompt, model, feed, or merchandising change.

Rerun targeted tests after every catalog update

After a catalog update, do not rerun only a broad random sample. Identify what changed and run the tests that depend on those fields, products, categories, policies, and actions.

A practical post-update sequence is:

Detect the change: Compare product IDs, titles, descriptions, attributes, variants, price, inventory, policies, and eligibility state.
Select impacted cases: Pull every test that references a changed product or field, plus comparison and category cases that could be affected by the change.
Run critical regressions: Check facts, recommendations, exclusions, availability, and next-step links against the new catalog snapshot.
Review new behavior: Sample real conversations after publication, prioritizing low-confidence, escalated, abandoned, or negatively rated interactions.
Promote failures: Add confirmed defects and newly observed question types to the versioned test set.

A changed product description can affect retrieval even when the product ID stays the same. A changed variant, policy, or inventory state can also change the correct answer without any change to the assistant’s prompt. That is why catalog versioning belongs in the test record.

Monitor real shopper questions after launch

Production QA should look for failure patterns that offline tests did not predict. Sample conversations by category, product, intent, outcome, and change window instead of reviewing only the most recent or most visible chats.

Track operational signals alongside answer quality:

Questions with no retrieved product or evidence
Answers that cite or rely on stale fields
Recommendations rejected by the shopper
Repeated follow-up questions that indicate an incomplete answer
Escalations and requests for a human
Broken product, location, or purchase actions
Questions that reveal missing catalog information

Use a human-reviewed sample to calibrate automated graders. An LLM-based grader can help scale checks for relevance or groundedness, but it should use a clear rubric and be checked against human judgments. Microsoft’s evaluation guidance recommends scenario-specific, human-friendly grading rubrics and running multiple evaluators across a dataset.

Feed production failures into the next offline run. This closes the loop between what shoppers really ask and what the team protects against before the next release. LangSmith’s documented workflow also recommends adding failing production traces to a dataset, creating targeted evaluators, validating fixes offline, and redeploying.

A lean operating model for ecommerce teams

A lean team does not need to test every possible question. It needs a defensible set that covers revenue-critical products, changing data, common decision criteria, and high-severity failure modes.

Assign ownership clearly:

Ecommerce or merchandising: approves product facts, comparisons, eligibility, and recommendation rules.
Customer experience: supplies real questions and reviews whether answers resolve shopper intent.
Digital product or engineering: records versions, runs evaluations, and verifies actions and integrations.
Marketing or content: fixes recurring information gaps revealed by questions.

The durable process is simple: preserve real questions, tie each case to a catalog version, evaluate evidence and behavior separately, gate critical failures, rerun impacted cases after updates, and turn production failures into new tests. That gives an ecommerce team proof that an AI shopping assistant is ready for the catalog shoppers will actually see—not merely that it can produce convincing text.