How to Build a Production-Grade Multimodal Product Matching Engine for Retail Catalogs
Product matching answers a deceptively difficult question: do two listings represent the same sellable product?
For a retailer, that decision affects search, comparison pages, pricing, inventory, reviews, advertising, and analytics. A false merge can attach the price or reviews of one variant to another. A missed match can split demand across duplicate catalog pages. The problem therefore needs more than an LLM prompt or a single image-similarity score.
In this tutorial, we will build a complete two-stage matching service. It retrieves plausible candidates using text, image, brand, and category signals; reranks the candidates with a supervised classifier; and converts model probabilities into three operational outcomes:
auto_match for high-confidence exact matches;
human_review for uncertain pairs;
non_match for rejected pairs.
The companion repository includes synthetic retail data, generated product images, training and evaluation scripts, a FastAPI service, automated tests, Docker packaging, GitHub Actions, a model card, and a production-readiness checklist.
What You Will Build
By the end, you will have:
a written policy for exact product identity;
normalized title, brand, category, model, color, quantity, and unit fields;
lightweight text and image feature encoders that work fully offline;
a high-recall candidate-retrieval stage;
a supervised pair classifier trained with hard negatives;
separate automatic-match and human-review thresholds;
candidate-level and pair-level evaluation;
an explainable FastAPI matching endpoint;
a non-root Docker image and CI workflow.
The implementation is intentionally lightweight so anyone can run it without a GPU, customer data, paid APIs, or downloaded model weights. The interfaces are designed so the encoders and in-memory search can later be replaced with fine-tuned deep models and an approximate-nearest-neighbor index.
Why Retail Product Matching Is Hard
Two product titles can be different strings but describe the same item:
Northstar M310 Wireless Mouse, Black
M310-BK Cordless Optical Mouse by Northstar - Black
Conversely, two almost identical titles can represent different sellable products:
Northstar M310 Wireless Mouse, Black
Northstar M310 Wireless Mouse, White
The second pair may belong to the same product family, but whether it is an exact match depends on the retailer's variant policy. Pack size is even less forgiving: one ink cartridge and a two-pack are not the same offer even when their product images look similar.
Real catalogs add abbreviated titles, missing identifiers, inconsistent units, multilingual data, reused images, seller errors, taxonomy drift, and new products with no historical labels. Research systems such as MAPS combine modalities because neither text nor images are consistently sufficient by themselves. Industry work on end-to-end multimodal product matching likewise treats the problem as a learned matching system rather than a collection of string rules.
Step 1: Define Product Identity Before Training a Model
A model cannot learn a stable target if the organization has not defined what “same product” means. Begin with a label policy reviewed by catalog, merchandising, and business stakeholders.
Relationship | Example | Exact-match action |
Exact identity | Same mouse model, color, and unit quantity | Merge or link automatically when confidence is high |
Variant | Same chair model, different color | Keep separate unless the catalog deliberately groups variants |
Family | Same printer series, different model number | Do not merge |
Pack-size difference | One cartridge versus a two-pack | Do not merge |
Substitute | Compatible item from another brand | Recommendation relationship, not identity |
Uncertain | Missing model number with similar text and image | Route to human review |
This tutorial labels exact identity only. The training data includes hard negatives that share a brand, product family, image style, or most title tokens but differ in a decisive attribute.
Step 2: Understand the Two-Stage Architecture
The two stages solve different problems:
Candidate retrieval optimizes recall. It reduces a large catalog to a small set that probably contains the correct match.
Pair reranking optimizes decision quality. It compares the query with each candidate using richer cross-product features.
This separation is important at retail scale. Running an expensive pair model against every catalog item is wasteful. Returning only the nearest vector without a pairwise decision, however, can silently merge visually similar variants.
In production, version the label policy, taxonomy, encoders, classifier, thresholds, and vector index together. An index created by one encoder version must not be queried with another without an explicit compatibility check.
'
Step 3: Set Up the Reference Project
The complete project is available in examples/retail-multimodal-product-matching.
You need Python 3.11 or newer. Docker is optional for the local Python workflow.
cd examples\retail-multimodal-product-matching
python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --requirement requirements-dev.txt
Generate the synthetic image fixtures, train the classifier, and evaluate it:
python scripts/generate_sample_images.py
python scripts/train.py
python scripts/evaluate.py
python -m pytest
The repository keeps training, calibration, and evaluation pairs in separate CSV files. For real data, split by canonical product and time not only by pair row so the same product identity cannot leak into both training and evaluation.
Step 4: Normalize Identity-Bearing Fields
Normalization should remove meaningless formatting differences without erasing business meaning.
For example:
M310-BK, M310 BK, and m310-bk can share a canonical model-code representation;
1 kg and 1000 g can be comparable after unit conversion;
1 count and 2 count must remain different;
white and black must not disappear just because the titles otherwise match.
The reference implementation normalizes text and model codes and computes quantity similarity in normalization.py. Keep the raw source values alongside normalized fields so every decision remains auditable.
Avoid one universal rule set. Category-specific attributes matter: dimensions and paper weight may be decisive for office paper, while connectivity and model number matter more for headsets.
Step 5: Create Multimodal Features
The offline profile uses two deterministic feature extractors:
signed feature hashing over normalized word and character n-grams;
a compact image descriptor built from per-channel color histograms, means, and standard deviations.
Each candidate pair then receives eight features:
FEATURE_NAMES = [
"text_cosine",
"image_cosine",
"brand_exact",
"category_exact",
"model_exact",
"color_exact",
"quantity_similarity",
"title_jaccard",
]
This is enough to exercise the full architecture offline, but a color histogram is not semantic computer vision. A production system should evaluate domain-trained text and visual or vision-language encoders on the retailer's categories and failure modes. Catalog Phrase Grounding research also shows the value of associating textual attributes with their visual evidence rather than treating the entire image as one undifferentiated vector (Amazon Science).
If the catalog contains image-only attributes, plan a governed attribute-extraction pipeline as a separate capability. Large-scale multimodal extraction has been studied for noisy marketplace data, but extracted attributes still need validation before they become identity constraints (ACL Anthology).
Step 6: Retrieve a Broad Candidate Set
The tutorial retrieval score combines text, image, brand, and category evidence:
def retrieval_score(left, right, root):
features = pair_features(left, right, root)
return float(
0.60 * features[0] +
0.20 * features[1] +
0.10 * features[2] +
0.10 * features[3]
)
For 25 products, an in-memory scan is fine. For millions of offers, encode products in batches and store normalized vectors in an index such as FAISS or an approved managed vector service. Apply safe filters such as market, category, or language before or during retrieval, but measure whether those filters remove valid matches.
The primary retrieval metric is candidate recall@k:
queries whose true match appears in the first k candidates
----------------------------------------------------------
queries that have a known true match
If the correct product never reaches the candidate pool, no reranker can recover it. This is why one blended “matching accuracy” number is insufficient.
For neural retrieval, a bi-encoder is efficient because catalog representations can be precomputed. A cross-encoder or other pair model can then rerank the shortlist; the Sentence Transformers documentation describes this retrieve-and-rerank pattern.
Step 7: Train the Pair Classifier with Hard Negatives
The reranker takes the eight pair features and predicts the probability of exact identity. The tutorial uses NumPy logistic regression so the training mechanics remain inspectable and dependency-light.
Hard negatives teach the model where retail errors actually occur. The sample data includes:
the same mouse family with a different model number;
the same model with a different color;
the same ink number with a different pack quantity;
a similar headset with a different connection type;
nearly identical paper titles with a different size or weight.
Random negatives from unrelated categories are useful initially but quickly become too easy. Production training should continuously mine near-neighbor false positives, reviewer rejections, and newly observed edge cases. Sample carefully so large brands and popular categories do not dominate the learning objective.
Step 8: Calibrate Operational Thresholds
A probability is not yet an automation policy. We need two boundaries:
probability >= auto threshold -> auto_match
review threshold <= probability < auto threshold -> human_review
probability < review threshold -> non_match
The training script selects an automatic threshold from a separate calibration set with a requested minimum precision of 0.95. The resulting tutorial thresholds are:
auto-match threshold: 0.9258
human-review threshold: 0.5092
In a real system, calibrate per category, seller segment, and error cost where the data supports it. Reliability diagrams and calibration metrics help determine whether estimated probabilities correspond to observed outcome frequencies; see the scikit-learn calibration guide.
The objective is not to maximize automation at any cost. High-risk categories may accept lower automatic recall to protect precision, while the review band retains uncertain positive cases for adjudication.
Step 9: Evaluate the Entire Decision System
Run:
python scripts/evaluate.py
The generated report is stored in artifacts/evaluation.json.
The local held-out results were:
Metric | Result | What it means |
Candidate recall@3 | 1.0000 | Every eligible query found a true duplicate among its first three retrieved candidates. |
Auto-match precision | 1.0000 | No held-out negative crossed the conservative automatic threshold. |
Auto-match recall | 0.4000 | Two of five true matches were accepted automatically. |
False merges | 0 | No negative pair was automatically merged. |
Human-review rate | 0.2143 | Three of fourteen evaluated pairs entered review. |
Positive recall including review | 1.0000 | All five positives were either auto-matched or routed to review. |
These results reveal an intentional trade-off. The system is conservative: it protects automatic precision but gives up automatic recall. That is often a safer starting point than silently over-merging a catalog.
Do not compare this miniature result with a production benchmark. A valid production evaluation needs representative categories, sellers, countries, image quality, missing fields, recent products, multilingual data, and adjudicated edge cases. Use precision, recall, F-scores, confusion matrices, and threshold curves appropriate to the operating decision; the scikit-learn model-evaluation guide provides the standard definitions.
Step 10: Return Evidence, Not Just a Score
Start the API:
$env:PYTHONPATH = "$PWD\src"
uvicorn retail_matcher.api:app --host 0.0.0.0 --port 8080
Open http://localhost:8080/docs, or send a request:
$body = @{ product_id = "P1001"; top_k = 4 } | ConvertTo-Json
Invoke-RestMethod `
-Method Post `
-Uri http://localhost:8080/v1/matches `
-ContentType application/json `
-Body $body
The same query produces different decisions for different candidates:
The first returned candidate is another listing of the same synthetic M310-BK mouse:
{
"product_id": "P1002",
"match_probability": 0.9408,
"decision": "auto_match",
"evidence": {
"text_cosine": 0.8488,
"image_cosine": 1.0,
"brand_exact": 1.0,
"category_exact": 1.0,
"model_exact": 1.0,
"color_exact": 1.0,
"quantity_similarity": 1.0,
"title_jaccard": 0.2727
}
}
Returning structured evidence makes debugging, audit, and reviewer tooling possible. A generated natural-language explanation may be added for convenience, but it should not redefine the matching policy or invent missing evidence.
The API also exposes:
GET /healthz for process health;
GET /readyz for catalog and model readiness;
GET /version for service and artifact visibility;
POST /v1/matches for ranked matches and evidence.
Step 11: Test the Failure Paths
The test suite checks both positive and negative behavior:
python -m pytest
Verified local result:
8 passed
The important tests are not only “a duplicate was found.” They also verify that:
a different model number is not automatically merged;
a different quantity is not automatically merged;
an unknown product ID returns HTTP 404;
readiness loads the catalog and model artifacts;
normalization preserves identity-bearing distinctions.
For production, add replay tests from confirmed incidents, modality-ablation tests, corrupted-image tests, latency and load tests, index/model compatibility tests, and rollback validation.
Step 12: Containerize and Add CI
Build the image after training so the versioned artifact exists:
docker build -t catalogmatch-ai:local .
docker run --rm -p 8080:8080 catalogmatch-ai:local
The supplied multi-stage Dockerfile:
pins the Python base image;
installs runtime dependencies in a builder stage;
copies only source, sample data, and versioned model artifacts;
runs as UID/GID 10001 rather than root;
defines a /healthz health check;
exposes port 8080.
The GitHub Actions workflow rebuilds the tutorial fixtures and artifacts, evaluates the model, runs the test suite, and builds an image tagged with the commit SHA. For a real release, add dependency and container scanning, signed images, a model-evaluation gate, immutable registry tags, workload identity, environment promotion, and post-deployment smoke tests.
Production Architecture and Controls
The reference code shows the skeleton. A production system needs additional controls across data, modeling, serving, and operations.
Data and Label Governance
Publish an identity-policy document with category-specific examples.
Track annotator agreement and adjudicate ambiguous cases.
Preserve data lineage from the source offer through normalized fields and labels.
Split datasets by canonical identity, seller, and time to prevent leakage.
Govern reviewer decisions before feeding them back into training.
The WDC Products dataset can support research and pipeline experimentation, but acceptance testing must reflect the target retailer's distribution and policy.
Model and Retrieval Quality
Track candidate recall independently from reranker metrics.
Evaluate by category, seller, locale, modality availability, and product age.
Use hard-negative mining and long-tail sampling.
Measure calibration, false merges, false splits, and review workload.
Compare text-only, image-only, attribute-only, and fused systems.
Require statistically justified promotion criteria for new versions.
Reliability and Scale
Generate embeddings asynchronously and incrementally.
Use a sharded or managed index with explicit version aliases.
Keep the previous model and index available for rollback.
Make batch writes idempotent and checkpoint long catalog jobs.
Define timeouts and fallback behavior when an image or encoder is unavailable.
Separate online query latency targets from offline full-catalog reconciliation.
Security and Privacy
Authenticate both online and batch entry points.
Apply least-privilege access to catalogs, images, labels, and artifacts.
Scan uploaded images and restrict supported formats and sizes.
Encrypt data in transit and at rest; define retention rules for seller data.
Avoid writing raw sensitive fields into logs or model-debug payloads.
Record who approved model, threshold, and identity-policy changes.
Monitoring
Monitor the business decision, not only CPU and request latency:
candidate recall on newly adjudicated samples;
confirmed false-merge rate;
match, review, and reject rates by category;
reviewer override rate and reason;
feature and score drift;
missing-image and missing-identifier rates;
index freshness and encoder/index version compatibility;
p50, p95, and p99 latency plus error rate.
A rising review rate may indicate catalog drift even when the API remains healthy. A sudden precision change in one category may be caused by a taxonomy or supplier-format change rather than the classifier itself.
Moving from the Tutorial Encoders to Deep Models
Keep the service boundary and replace components incrementally:
Fine-tune a text bi-encoder using exact matches and mined hard negatives.
Fine-tune a vision or vision-language encoder on catalog images.
Store normalized embeddings in a versioned ANN index.
Retrieve broadly with category and locale constraints.
Rerank using a cross-encoder, multimodal network, gradient-boosted model, or calibrated ensemble.
Retain deterministic checks for model number, quantity, compatibility, and regulated attributes.
Calibrate on an independent, recent dataset.
Shadow the new system, review disagreements, then increase automation gradually.
This approach avoids tying production orchestration to a particular foundation model. It also makes controlled experiments and rollbacks practical.
Known Limitations of This Tutorial
The 25 products and their images are synthetic.
The visual descriptor captures color distribution, not semantic shape.
Retrieval uses an in-memory scan rather than an ANN index.
The classifier is deliberately small and is not a deep multimodal model.
The service matches one catalog product against the catalog; it does not implement distributed full-catalog clustering.
Multilingual text, online index updates, reviewer UI, authentication, and cloud deployment are outside this example.
No production accuracy, scale, security, or compliance claim is made.
These limitations are deliberate: the repository stays runnable while exposing exactly which pieces must be replaced or hardened for a commercial deployment.
How Codersarts Can Help
Codersarts helps retail and commerce teams move from an ambiguous matching requirement to a measurable, production-ready system. Engagements can include:
product-identity policy and error-cost workshops;
catalog data assessment and label-program design;
multimodal retrieval, reranking, and attribute-extraction proofs of concept;
offline evaluation, threshold calibration, and human-review design;
production APIs, batch pipelines, MLOps, monitoring, and cloud deployment;
dedicated AI, ML, data, and cloud engineering expertise.
If your team is evaluating product matching, catalog deduplication, offer comparison, or catalog intelligence, contact contact@codersarts.com to discuss a focused consultation or implementation engagement.
You can also explore the Codersarts Identity Verification API for another example of production-oriented AI API design.
Conclusion
A production-grade product matcher is a decision system, not just an embedding model. It starts with a precise definition of identity, retrieves candidates with high recall, reranks them using multimodal and structured evidence, calibrates confidence against business risk, and sends uncertainty to people. It also versions its data and artifacts, tests failure paths, and monitors real decision outcomes after release.
The included project gives you a working, inspectable baseline. Replace the lightweight encoders with validated domain models, train on representative labeled catalog data, and preserve the evaluation, review, and operational controls around them.
Comments