Find where search actually breaks.
A poor result page is the end of a chain, not the diagnosis. This framework helps ecommerce teams locate the earliest failure, gather the right evidence and choose a fix that addresses the cause.
Vendor-agnostic. Designed for catalogue search across markets, languages and search platforms.
Fix the earliest broken stage. Every downstream layer inherits what the stage before it misunderstood, excluded or distorted.
Query understanding
The system builds the wrong meaning before retrieval begins.
A query is not merely a string. It can contain a product type, attributes, a brand, a use case and constraints. When those parts are flattened or mapped to the wrong fields, the engine retrieves a plausible but fundamentally wrong candidate set.
black waterproof hiking jacket
Weak readingMatches any product containing several of the words, including black hiking trousers and water-resistant casual jackets.
Useful readingTreats jacket as the product type; black and waterproof as constraints; hiking as activity context.
Questions to ask
- Which parts of the query describe the product, and which constrain it?
- Are customer terms mapped to the catalogue language used in this market?
- Does the interpretation change when word order, spelling or language changes?
Evidence to inspect
- Parsed-query output, applied filters and rewritten terms
- Searches reformulated immediately after the first results page
- Results grouped by query class: product, attribute, brand and use case
What to try
- Create a labelled query set before changing synonyms or boosts
- Map high-value concepts to explicit attributes where the data supports it
- Test the same intent across spelling, inflection and local vocabulary
Ranking
Relevant products are present, but the order does not reflect the shopper’s intent.
Ranking decides which eligible products earn attention. Problems appear when broad text matches outweigh exact product-type matches, business signals dominate relevance, or one global formula is expected to serve every query class equally well.
linen shirt
Weak readingA linen-blend scarf and a cotton shirt with ‘linen look’ copy outrank an in-stock linen shirt.
Useful readingExact product type and material matches lead, with availability and business signals used as controlled tie-breakers.
Questions to ask
- Is the first result defensible for this query, not merely relevant somewhere?
- Which signals determine the top positions for exact, broad and category queries?
- Do out-of-stock, low-quality or duplicate products crowd out stronger options?
Evidence to inspect
- Top 10 results with matched fields and score contributions
- Clicks, add-to-basket events and reformulations by query class
- Before-and-after judgements from a fixed relevance set
What to try
- Separate exact product and attribute matches from loose description matches
- Tune by query class rather than relying on one universal weighting
- Introduce business signals gradually and test their relevance cost
Coverage and retrieval
Products that should qualify never become candidates.
Coverage is the boundary between the catalogue and the searchable index. Missing attributes, stale feeds, aggressive filters, field configuration and locale-specific gaps can all make a product invisible even though it exists and is available to buy.
size 38 trail shoes
Weak readingReturns zero results because sizes are stored as variant strings that the search index does not expose.
Useful readingRetrieves trail shoes with an available size-38 variant, then ranks them using product and activity relevance.
Questions to ask
- Does the product exist, is it indexed, and is it eligible in this market?
- Which filter or field removed it from the candidate set?
- Are product and variant availability represented consistently?
Evidence to inspect
- Catalogue-to-index record counts and freshness timestamps
- Field completeness by category, market and variant
- Zero- and low-result queries replayed with filters removed one at a time
What to try
- Trace representative SKUs from source catalogue to final index
- Measure attribute completeness before building rules around an attribute
- Add monitoring for feed failures, stale records and unexpected count shifts
Evaluation
The team cannot tell whether a change improved search or moved the failure elsewhere.
Search quality can look better in hand-picked examples while worsening across long-tail, multilingual or high-volume query groups. Evaluation fails when teams lack a stable query set, explicit judgements, segmented behavioural measures and safeguards for commercial side effects.
A synonym change fixes ten visible searches
Weak readingThe change ships because the demo looks better, without checking false matches or other markets.
Useful readingThe same change is replayed against labelled query groups, inspected for regressions, then measured in real use where traffic permits.
Questions to ask
- What specific behaviour should improve, for which query group?
- Which known-good searches must not regress?
- Can the event tracking distinguish a useful result from a desperate click?
Evidence to inspect
- A versioned judgement set with query intent and expected products
- Behavioural metrics segmented by query class, device and market
- Change logs connecting configuration releases to metric movement
What to try
- Define the hypothesis and guardrails before changing configuration
- Combine offline relevance review with behavioural evidence
- Keep a regression set built from real failures and important queries
Merchandising distortion
Commercial rules override the intent that brought the shopper to search.
Boosts, pinned products, campaigns and stock priorities are legitimate inputs. They become harmful when they are opaque, permanent or broad enough to outrank clearly better matches. The goal is not relevance without commerce; it is commercial influence with explicit limits.
blue wool coat
Weak readingA campaign pins discounted polyester jackets above exact wool coats for every coat-related query.
Useful readingExact blue wool coats remain the core result set; eligible campaign products receive a bounded boost within that set.
Questions to ask
- Would this product rank highly without the commercial rule?
- What query scope, market and dates should limit the rule?
- Who owns expiry and review when the campaign ends?
Evidence to inspect
- Active rules with owners, scopes, start dates and expiry dates
- Rank positions with commercial boosts switched on and off
- Engagement and reformulation for queries touched by each rule
What to try
- Apply boosts only after basic eligibility and relevance are satisfied
- Give every temporary rule an owner and automatic expiry
- Audit rules by reach so the broadest interventions are reviewed first
Operational drift
Individually reasonable changes accumulate into an unowned system.
Search is a living product. Catalogue changes, platform upgrades, new markets, undocumented rules and one-off fixes gradually separate the current behaviour from the original design. Drift is usually slow enough to escape incident processes and broad enough to resist one tuning session.
The same search, six months later
Weak readingNew attributes are not indexed, old synonyms remain active and overlapping rules produce an unexplained order.
Useful readingConfiguration, data health and relevance sets are reviewed on a cadence, with ownership and rollback information.
Questions to ask
- Who can explain the current configuration and approve a change?
- Which rules, synonyms or fields no longer match the catalogue?
- What changed in search behaviour after the last platform or feed release?
Evidence to inspect
- Configuration history, release notes and rule ownership
- Trend lines for coverage, zero results and repeated reformulation
- Scheduled replays of the relevance and data-quality sets
What to try
- Treat search configuration as versioned product logic
- Assign ownership for data, relevance, merchandising and measurement
- Review the highest-reach rules and weakest query groups regularly
Turn the framework into a working review.
The useful unit is not a single bad screenshot. It is a repeated failure pattern connected to evidence, an owner and a measurable next action.
- 1
Build a representative query set
Sample high-volume, high-value, zero-result and reformulated searches. Add deliberate probes for products, attributes, brands, use cases, ambiguity and local language.
Output: A query set labelled by intent, market and business importance.
- 2
Replay and classify the failure
For each query, inspect the interpretation, candidate set, order and applied rules. Assign the earliest stage where behaviour diverges from the expected result.
Output: A failure map that separates root causes from visible symptoms.
- 3
Prioritise patterns, not anecdotes
Group failures by cause and estimate reach using query frequency, affected catalogue areas, market importance and severity for the shopper.
Output: A short backlog ordered by customer reach, confidence and effort.
- 4
Change one layer and measure
State the hypothesis, expected query groups and guardrails. Replay the fixed set before release and use real behavioural testing when volume and instrumentation allow.
Output: A decision to keep, adjust or reverse—with the evidence attached.
A small worksheet is enough to begin.
Capture one row per query. The discipline is separating what the customer asked for, what the system did, and what evidence would justify a change.
| Field | What to capture | Why it matters |
|---|---|---|
| Query and market | Raw query, language, device and market | Meaning and expected results vary by context. |
| Expected intent | Product type, attributes, brand, use case and constraints | Makes the relevance judgement explicit. |
| Observed behaviour | Interpretation, candidate count, top results and applied rules | Locates the earliest stage that failed. |
| Failure class | Interpretation, ranking, coverage, evaluation, merchandising or drift | Prevents a visible symptom becoming the diagnosis. |
| Evidence and action | Supporting data, proposed change, measure and guardrail | Turns the finding into a testable improvement. |