Two-Tower Recommendation Models for Large-Scale Candidate Retrieval
A recommendation surface cannot score 80 million products with an expensive ranking model every time a user opens the application. Even at one millisecond per item, exhaustive scoring would take more than 22 hours. The system needs a fast first stage that can reduce millions of eligible items to hundreds or thousands of plausible candidates without throwing away the few items the ranker would have chosen.
That is the problem a two-tower recommendation model is designed to solve.
One neural network converts the user, session, or request context into a query embedding. A second neural network converts each item into a candidate embedding in the same vector space. Because the item embeddings can be computed before the request, an approximate nearest-neighbor index can retrieve high-scoring items in milliseconds. A separate ranker then applies richer cross-features, business objectives, and policy constraints to the reduced set.
The architecture is elegant. Productionizing it is not. Training examples inherit exposure bias. Negative sampling defines much of the decision boundary. A small difference between training and serving features can rotate the embedding space. Index recall competes with latency and memory. New items need embeddings before they have behavior. Model and index versions must move together. Offline metrics can look excellent while candidate recall, catalog coverage, or business outcomes deteriorate.
Practical verdict: use a two-tower model when the candidate corpus is too large for exhaustive online scoring and the request-item score can be factorized into separately computed embeddings. Treat it as a high-recall candidate generator, not the final recommender. Build full-catalog evaluation, sampling correction, index validation, eligibility enforcement, and coordinated model-index deployment into the first production design.
Executive Answer: What Is a Two-Tower Recommendation Model?
A two-tower recommendation model, also called a dual encoder or factorized retrieval model, learns two functions:
q=fquery(xu,xs,xc)q=fquery(xu,xs,xc)
vi=fitem(xi)vi=fitem(xi)
The query tower uses user, session, and request-context features to produce query embedding qq. The candidate tower uses item identity, metadata, content, or behavioral features to produce candidate embedding vivi. Their retrieval score is deliberately cheap:
s(q,i)=qTvis(q,i)=qTvi
or, when normalized embeddings are used:
s(q,i)=cosine(q,vi)s(q,i)=cosine(q,vi)
The model is trained so observed or relevant query-item pairs score higher than alternatives. At serving time, item vectors are precomputed and indexed. Only the query tower normally runs online; the index returns the nearest candidate vectors.
This factorization is the key property. A model that must jointly process every user-item pair through attention, cross-features, or a multilayer perceptron cannot precompute item scores independently and therefore cannot retrieve directly from a standard vector index.
The architecture has roots in semantic retrieval. The Deep Structured Semantic Model projected queries and documents into a common low-dimensional space and calculated relevance by distance. Large recommendation systems adapted the same idea to users, sessions, and items. Google’s Neural Deep Retrieval research explicitly describes a two-tower framework for retrieving from a large item corpus.
The Retrieval Contract: Reduce Millions to Hundreds Without Losing the Winners
Before choosing tower layers or embedding dimensions, define the contract between candidate generation and ranking.
Contract field | Example | Why it matters |
eligible corpus | 42 million active, in-region products | defines the actual search space, not the database total |
request types | home feed, search continuation, product detail, email | determines which query context is available |
retrieval output | 1,500 candidates per source | gives the ranker enough recall and controls cost |
recall target | at least 95% of known relevant items in top 1,500 | expresses the candidate generator’s primary job |
latency budget | p95 under 35 ms for all retrieval work | makes index and network trade-offs explicit |
freshness | new item searchable within 10 minutes | governs embedding and index-update architecture |
policy | tenant, entitlement, geography, safety, inventory | prevents invalid items entering the ranking pool |
diversity expectation | minimum source/category coverage | stops retrieval collapse before ranking |
fallback | contextual popularity plus curated inventory | keeps the product available during low confidence or failure |
observability | source, score, model/index version, filter outcome | enables diagnosis and safe experimentation |
The retrieval objective is usually high recall under bounded latency and cost. Precision matters because irrelevant candidates waste ranker capacity, but the ranker can remove weak items. It cannot recover a relevant item the retrieval stage omitted.
This is why optimizing only the two-tower training loss is inadequate. The true production contract spans learned relevance, full-corpus retrieval, ANN approximation, policy filters, source blending, and operational freshness.
Why Two Towers Scale
Assume there are NN eligible items and the final ranking model costs CrCr per query-item pair. Exhaustive ranking costs approximately:
Costexhaustive=N×CrCostexhaustive=N×Cr
A two-stage system uses a cheap query encoder, an ANN lookup, and then ranks only KK retrieved items:
Costtwo−stage=Cq+CANN(N,d,K)+K×CrCosttwo−stage=Cq+CANN(N,d,K)+K×Cr
where dd is embedding dimension and K≪NK≪N.
The item tower can run offline because vivi does not depend on the current request. The expensive work processing descriptions, categorical features, image embeddings, or item history happens during index construction. Online serving computes qq once and uses a dot-product-compatible index.
The production advantage is not merely “neural embeddings are fast.” It is separability:
candidate vectors are materialized once and reused across requests;
vector indexes avoid scanning the entire corpus;
query inference can be batched, cached, or accelerated independently;
item ingestion and user-serving capacity scale separately; and
the ranker sees a manageable set where richer interactions become affordable.
The YouTube recommendation paper describes this classic two-stage division: candidate generation first, then a separate ranking model. The specific models have evolved, but the systems principle remains central to large-catalog recommendation.
Candidate Generation and Ranking Solve Different Problems
Teams often weaken two-tower retrieval by asking it to perform every final-ranking task. Candidate generation and ranking need different inductive biases.
Candidate generation
Candidate generation must:
search the entire eligible corpus;
produce broad, relevant coverage;
run within a small, predictable latency budget;
work with a simple decomposable score;
include new and long-tail items where appropriate; and
retain enough provenance for downstream controls.
Ranking
Ranking can:
evaluate hundreds or thousands of candidates rather than millions;
use request-item cross-features;
incorporate price fit, position, inventory, margin, quality, risk, or long-term value;
apply sequence attention between the request and each item;
estimate multiple outcomes such as click, purchase, return, or completion; and
optimize the final slate for diversity and constraints.
For example, the candidate generator can learn that a user interested in trail running should see outdoor footwear and equipment. The ranker can determine whether a particular shoe is available in the user’s size, fits the current price range, has acceptable return risk, and adds diversity relative to other slate items.
If a feature cannot be factorized “distance between this user’s preferred price and this item’s current price,” for example it usually belongs in the ranker or must be approximated by separate query and item representations. Do not quietly introduce pairwise computation into retrieval and assume ANN serving will still work.
Designing the Query Tower
The query is not always a durable user. It can represent a user, an anonymous session, a search context, a seed item, an account, a household, or a combination.
User identity and stable attributes
User IDs can capture repeated behavioral preference through learned embeddings. They also create limitations:
unseen users have no trained ID representation;
infrequent users receive poorly estimated vectors;
embeddings can memorize historical exposure;
privacy and deletion requirements extend to derived parameters; and
an ID alone cannot respond quickly to new intent.
Use identity as one signal, not the entire query.
Stable features may include declared interests, account type, subscription tier, organization, or preferred language. Exclude protected or sensitive attributes unless there is a lawful, justified, tested use—and evaluate proxies that may encode them indirectly.
Interaction history
Represent recent and long-term activity through:
weighted mean pooling of item embeddings;
recurrent or transformer sequence encoders;
attention over past items;
separate event-type pools;
time-decayed summaries; or
multiple interest vectors.
A simple pooled history is a strong baseline:
hu=∑j∈Huwjvj∑j∈Hu∣wj∣+ϵhu=∑j∈Hu∣wj∣+ϵ∑j∈Huwjvj
The weight wjwj can reflect event strength, recency, completion, and confidence. Reusing candidate item embeddings inside the history can align the space, but it also couples query serving to the item-embedding version. Version that dependency explicitly.
Session and request context
Current intent often dominates long-term taste. Useful query features include:
the current page or seed item;
recent search terms;
the last few interactions;
device and surface;
locale, time, or season;
referral source; and
permitted market or tenant context.
Do not include context at training time if it cannot be reproduced before retrieval in production. Position, future actions, final purchase totals, or post-interaction attributes create leakage.
Multiple interests
A single vector compresses all current intent into one point. It may blur independent interests, shared-account behavior, or a session that diverges from long-term history. Options include:
separate short-term and long-term query vectors;
category-specific heads;
several learned interest vectors with max-sim retrieval;
retrieval once per recent seed and source fusion; or
separate models for materially different surfaces.
Multiple query vectors increase retrieval calls and deduplication work. Adopt them only when single-vector error analysis shows mode collapse.
Designing the Candidate Tower
The candidate tower determines whether the system can generalize beyond item popularity and IDs.
Item identity
A learned item-ID embedding captures behavioral relationships efficiently. Mature, frequently exposed items often benefit. New and rare items receive weak or absent embeddings, and deleted IDs consume vocabulary until cleaned up.
Structured metadata
Category, brand, creator, topic, format, language, price band, and technical attributes help cold-start coverage. Structured fields should have controlled vocabularies, missing-value handling, and versioned preprocessing.
Text, image, and multimodal content
Pretrained or task-tuned content embeddings can represent descriptions, documents, images, audio, or video. They help new items enter a useful area of the vector space before interaction data accumulates. The detailed guide to content-based recommendation systems explains representation contracts, sparse baselines, multimodal fusion, and content-quality risks.
Content features do not eliminate behavioral bias when the model is trained on exposed clicks. The model can learn to use content as a shortcut for historically popular inventory. Evaluate new-item and low-exposure cohorts separately.
Lifecycle and eligibility features
Some fields change too quickly to bake into a slowly refreshed vector. Real-time inventory, entitlement, legal status, or market availability should usually be filters or ranking features rather than latent dimensions. If price is embedded, an index refresh is required whenever price changes enough to matter.
Classify features by update rate:
Feature class | Examples | Recommended handling |
static or slow | category, creator, product family | candidate tower and index |
medium-rate | description, quality score, popularity bucket | candidate tower if refresh SLA supports it; otherwise ranker |
high-rate | inventory, live price, current availability | retrieval filter or online ranker |
request-specific | user-item distance, current promotion eligibility | ranker or policy layer |
The Shared Embedding Space Is an Interface
The query and candidate towers may have different input features and internal architectures, but their outputs must share dimension, metric, normalization, and semantic version.
Define a representation specification:
space_id: rec-home-v12
dimension: 256
score: dot_product
normalize: false
query_model: query-tower-v12.3
candidate_model: item-tower-v12.3
history_item_space: rec-home-v12
feature_contract: features-2026-08-18
training_cutoff: 2026-08-15T00:00:00Z
index_version: catalog-2026-08-20-04
Never mix query vectors from one shared space with candidate vectors from another, even when dimensions match. The coordinates have no stable meaning across independent model trainings.
Embedding dimension
Higher dimensions increase expressive capacity, index memory, compute, and sometimes overfitting. Lower dimensions improve efficiency but may compress distinct interests or items together. Select dimension from a Pareto curve of full-catalog recall, segment quality, ANN performance, memory, and query latency not from convention.
Norm and popularity
With dot-product scoring, vector norms can influence rank. The model may encode item popularity or confidence in norm. That may be useful or may cause head-item domination. Inspect norm distributions by popularity, age, category, and metadata completeness. If using cosine similarity, normalize in both training and serving.
Temperature
For a softmax objective, temperature ττ controls score sharpness:
P(i+∣q)=exp(qTvi+/τ)∑j∈Cexp(qTvj/τ)P(i+∣q)=∑j∈Cexp(qTvj/τ)exp(qTvi+/τ)
A smaller temperature makes the distribution sharper and gradients focus on close alternatives. Tune it with the negative strategy and embedding norms; the same value behaves differently when vectors are normalized.
Training Data: The Logging Policy Is in the Dataset
Implicit feedback is not a random sample of preference. A user can interact only with items the previous system exposed. Position, popularity, inventory, interface design, notifications, and marketing affect the label.
Define the positive event
Clicks provide volume but can reflect curiosity or misleading presentation. Purchases, completions, saves, long dwell, qualified leads, or successful resolutions may better express value but are sparser and delayed.
Possible approaches include:
train on a high-volume event and use outcome weights;
use multiple tasks or event heads;
filter low-confidence events;
require dwell or completion thresholds;
build separate retrieval models by surface; or
optimize a downstream event directly when scale permits.
Document what one training row means:
query state available at time t
positive item interacted with after t
event type and confidence
exposure source and position
sampling probability
market, surface, and eligibility snapshot
Build examples at event time
Reconstruct the user history and context as they existed before the positive event. Do not use future interactions, the later state of the catalog, or features updated by the outcome. Split train, validation, and test chronologically to reflect deployment.
Deduplicate correlated events
Ten clicks caused by a refresh loop should not equal ten independent preference observations. Sessionize activity, cap repeated events, remove bots and automation, and identify shared devices or service accounts where relevant.
Record exposure when possible
Unclicked exposed items are not perfect negatives, but they carry different information from random unseen items. Logging request ID, candidate source, rank, item, eligibility, score, and outcome enables better sampling, bias analysis, and online evaluation.
Negative Sampling Is Part of the Model
The full softmax denominator may contain millions of items, so training usually samples alternatives. Those negatives define what the model learns to separate.
In-batch negatives
For a batch of BB positive query-item pairs, each positive item can serve as a negative for the other B−1B−1 queries. This provides many negatives without additional candidate-tower computation.
Benefits:
efficient matrix multiplication;
larger negative sets with bigger batches;
simple distributed training; and
strong baseline performance.
Risks:
popular items appear more often and distort the sampled distribution;
another user’s positive may also be relevant to the current query—a false negative;
duplicated positives create accidental hits; and
batches grouped by time, region, or data pipeline may not represent the corpus.
Google’s sampling-bias-corrected neural retrieval work shows that in-batch sampling can be biased under a power-law item distribution and proposes correction based on item sampling frequency.
Uniform random negatives
Uniform corpus samples improve tail coverage and approximate the broad search space. Most are extremely easy. The model can reduce loss without learning fine distinctions near the decision boundary.
Popularity-weighted negatives
Sampling by popularity exposes the model to realistic competitors and high-exposure inventory. Without correction, it can reinforce popularity and undertrain the tail.
Hard negatives
Hard negatives score highly under the current model or a baseline but are not the observed positive. They teach fine-grained separation:
same category but wrong use case;
exposed but skipped items;
ANN neighbors that violate expert relevance;
items retrieved by the previous production model; or
lexical/content neighbors that are not substitutable.
Hard negatives can be false negatives. A user may have liked them but never had the opportunity to interact. Overusing them can make the model push genuinely relevant alternatives away.
Mixed and cached negatives
The Mixed Negative Sampling paper combines in-batch and uniformly sampled negatives to address selection bias in implicit feedback. Negative Cache research uses cached candidate embeddings to expose retrieval models to larger negative pools under limited memory and compute.
A production recipe often combines:
corrected in-batch negatives for efficiency;
uniform samples for corpus coverage;
popularity or exposure-aware samples for realism;
mined hard negatives for local discrimination; and
explicit masks for known positives, duplicate items, or impossible pairs.
Log the sampling method and probability. Changing the sampler is a model change even when the network architecture is unchanged.
Choosing a Training Objective
Sampled softmax
For one positive and sampled candidate set CC, minimize cross-entropy over dot-product scores. This is common because it maps naturally to retrieval and in-batch negatives.
Correct for nonuniform candidate sampling when necessary. Otherwise, the model may learn the sampler’s frequency distribution rather than the desired retrieval distribution.
Pairwise logistic or BPR-style loss
For positive i+i+ and negative i−i−:
L=−logσ(s(q,i+)−s(q,i−))L=−logσ(s(q,i+)−s(q,i−))
Pairwise objectives directly encourage the positive to outrank a negative. They depend strongly on negative difficulty and can be inefficient if negatives are too easy.
Hinge or margin loss
L=max(0,m−s(q,i+)+s(q,i−))L=max(0,m−s(q,i+)+s(q,i−))
Margin loss stops penalizing a pair once separation exceeds margin mm. It can make the intended gap interpretable but still needs careful mining.
Multi-task objectives
Retrieval can learn from clicks, saves, purchases, dwell, or completion simultaneously. Avoid combining events with arbitrary weights and calling the output “engagement.” Validate whether the shared space benefits all tasks or whether one high-volume outcome dominates.
The best objective is the one that improves full-catalog retrieval for the target outcome under realistic cohorts. Training-loss convergence alone does not answer that question.
From Candidate Embeddings to ANN Retrieval
After training, run the candidate tower over every active item and build a vector index. At request time:
request features
-> query tower
-> query embedding
-> ANN search over current candidate index
-> candidate IDs and retrieval scores
-> policy filters and enrichment
-> final ranker
Exact search is the quality oracle
For a manageable evaluation corpus, calculate exact top-KK dot products. Then compare the ANN result:
ANN Recall@K=∣TopKANN∩TopKexact∣KANN Recall@K=K∣TopKANN∩TopKexact∣
Without this benchmark, teams cannot tell whether quality loss comes from the learned space or the index approximation.
Common index choices
Index family | Strength | Production trade-off |
flat exact search | exact, simple, ideal reference | compute and latency grow with corpus size |
HNSW | strong recall-latency trade-off and flexible queries | memory, build time, and updates/deletions |
IVF | limits search to selected partitions | training, partition balance, and probe count affect recall |
product quantization | compresses vectors for very large corpora | compression can change nearest neighbors |
tree or hashing methods | can work for particular distributions and constraints | quality varies with dimension and geometry |
The HNSW paper describes hierarchical navigable small-world graphs. The FAISS research paper covers large-scale exact, approximate, and compressed-domain vector search. Choose through benchmarks on the actual embeddings, filters, hardware, and traffic distribution.
Metric compatibility
If training uses dot product, the index must support maximum inner-product search or an equivalent transformation. If training normalizes vectors and uses cosine, apply the same normalization during batch embedding, query serving, and exact evaluation. A silent metric mismatch can preserve plausible results while destroying learned ranking.
Filtering and sharding
Tenant, geography, entitlement, inventory, language, safety, and lifecycle constraints may reduce the searchable corpus substantially. Options include:
separate indexes by stable high-level boundary;
metadata filters inside the ANN engine;
oversampling followed by deterministic filtering;
routing to category or locale partitions; and
source-specific fallback indexes.
Measure filtered ANN recall. A globally high-recall index may fail for narrow filters because the nearest eligible items were never explored.
Over-retrieval
Retrieve more than the ranker needs to accommodate filtering, deduplication, source quotas, and failures. If the ranker needs 500 candidates, the ANN layer may retrieve 800 or 1,500 depending on filter-drop rates. Set the multiplier from measured cohorts rather than a fixed guess.
Three Production Planes Must Move Together
A two-tower system has three connected planes.
1. Learning plane
The learning plane builds temporal examples, joins point-in-time features, samples negatives, trains both towers, evaluates full-catalog recall, registers artifacts, and approves a shared embedding space.
2. Indexing plane
The indexing plane reads the approved candidate tower, embeds active items, validates coverage and vector distributions, builds or updates the ANN index, measures ANN recall, and promotes an immutable index version.
3. Serving plane
The serving plane assembles online query features, runs the matching query tower, routes to the compatible index, applies eligibility, blends sources, calls the ranker, and logs exposures and outcomes.
Version compatibility is non-negotiable
Publish the query tower, candidate tower, representation specification, feature definitions, and index manifest as one release set. A safe promotion sequence is:
approve the trained shared space;
build a complete shadow index with the candidate tower;
run coverage, distribution, exact-recall, ANN, and policy tests;
deploy or preload the compatible query tower;
route shadow traffic and compare candidates;
atomically switch model-index routing for a small cohort;
expand while monitoring; and
retain the previous compatible pair for rollback.
Never deploy the new query tower against the old index “temporarily.” Similar coordinate dimensions do not make spaces compatible.
Freshness and Cold Start
New items
An item-ID-only candidate tower cannot represent an unseen item. Add metadata or content features, reserve explicit unknown handling, or train a separate cold-item retriever. The new item must still pass through validation, embedding, and index update before it is discoverable.
Track:
source creation to canonical catalog lag;
catalog to embedding lag;
embedding to searchable-index lag;
percentage of active items in the current space; and
cold-item exposure and outcome quality.
New users
A new user needs session context, onboarding preferences, a seed item, query text, segment defaults, or exploration. An unknown user-ID token alone creates identical recommendations for every new user.
Rapidly changing intent
Precomputing user vectors lowers latency but can make intent stale. Compute the query online from recent events, maintain a near-real-time profile store, or combine a cached long-term vector with an online session vector. Choose based on event velocity and query-tower cost.
Rapidly changing items
Do not rebuild embeddings for every inventory decrement if inventory is a filter. Conversely, when the product description, category, creator, or content changes, stale embeddings can misrepresent the item. Define which fields invalidate the vector and which update only serving metadata.
Feature Parity and Online Latency
The online query tower must see feature values semantically equivalent to training.
Point-in-time correctness
Build historical examples using features available at the event timestamp. Current user aggregates joined onto old events leak future information and usually inflate offline recall.
Default-value behavior
Test missing history, unknown IDs, delayed features, malformed locale, and partial profiles. Default vectors can become high-traffic hot spots in the embedding space. Monitor their nearest neighbors and share of requests.
Latency decomposition
Track the critical path separately:
authentication and routing
+ online feature reads
+ query preprocessing
+ query-tower inference
+ ANN request and search
+ filtering and candidate hydration
+ multi-source merge
+ ranker inference
+ response serialization
Average endpoint latency hides tail amplification. Report p50, p95, and p99 per stage and by cohort, index shard, region, and fallback path.
Cache carefully
Query embeddings can be cached for stable users or repeated seed items, but cache keys must include the representation space, relevant context, profile version, and authorization boundary. A stale cached vector may produce technically valid but contextually outdated candidates.
Hybrid Candidate Generation
One two-tower model rarely covers every retrieval need. Production pools often combine:
two-tower personalized retrieval;
item-based collaborative neighbors;
content or multimodal similarity;
session or sequence retrieval;
query or lexical search;
popularity and trending inventory;
editorial or contractual items; and
controlled exploration.
The companion article on user-based versus item-based collaborative filtering explains interpretable behavioral neighborhoods. The content-based guide explains semantic and cold-item candidates. Preserve the source of every candidate through ranking.
Merge strategies
Quota union: allocate a retrieval budget per source. It guarantees coverage but can waste capacity on a weak source.
Score calibration: transform source scores onto comparable scales. Calibration can drift by category, user state, or source version.
Rank fusion: combine source ranks when raw scores are incomparable. Reciprocal rank fusion is simple and robust but ignores score confidence.
Learned source selection: predict which candidate sources or quotas fit the request. This improves efficiency but adds another failure surface and requires exploration data.
Candidate-source diversity should be measured before final ranking. If 99% of the pool comes from one model, the ranker has little opportunity to recover other intents.
Full-Catalog Evaluation, Not Convenient Sampled Metrics
Evaluation should mirror the actual retrieval decision: find relevant items among the full eligible corpus.
Model retrieval metrics
Measure:
Recall@K;
Hit Rate@K;
Mean Reciprocal Rank;
NDCG@K when graded relevance exists;
coverage of catalog, categories, suppliers, and item-age cohorts;
popularity and exposure concentration;
new-item and low-history recall; and
source contribution in a hybrid pool.
Why sampled evaluation is dangerous
Ranking one positive against 100 random negatives is much easier than ranking it against 40 million items. More importantly, sampled metrics may not preserve the ordering between models. Google’s research on sampled recommendation metrics shows that sampled metrics can be inconsistent with their exact counterparts and recommends avoiding sampling for metric calculation when possible.
If full-catalog evaluation is expensive:
run it less frequently but make it a release gate;
use distributed matrix multiplication or a production-like index;
maintain smaller diagnostic samples for rapid iteration;
label sampled results clearly; and
never compare metrics generated under different sampling schemes as though they were equivalent.
ANN evaluation
Separate learned-model quality from index approximation:
exact retrieval using the candidate vectors;
ANN retrieval using the same vectors;
policy-filtered retrieval;
merged candidate-pool recall; and
final ranker/slate metrics.
This decomposition answers whether a relevant item was lost by the model, the index, a filter, a source quota, or the ranker.
Temporal and cohort evaluation
At minimum, slice by:
new versus established users;
new, tail, and head items;
history length;
market, locale, device, and surface;
item category and supplier;
metadata completeness;
index shard and representation version; and
time since model or index promotion.
Aggregate Recall@K can improve while new-item recall collapses because head items dominate the event volume.
Online experiments
Candidate retrieval changes the options available to ranking. Online evaluation should track the primary product outcome plus:
final recommendation click and conversion;
long-term retention or completion;
hides, returns, cancellations, or complaints;
candidate-source share in displayed slates;
catalog and supply-side coverage;
latency and timeout rate; and
novelty and repeated exposure.
Use a predeclared hypothesis, guardrails, minimum duration, power calculation, and rollback trigger. Do not launch solely because offline Recall@K increased.
Capacity and Cost Planning
Two-tower systems shift cost from online pairwise scoring to embedding generation, index memory, and retrieval infrastructure.
Index memory estimate
Raw vector storage is approximately:
Memoryraw=N×d×bytesPerValueMemoryraw=N×d×bytesPerValue
For 50 million items, 256 dimensions, and 4-byte floats, raw vectors alone require about 51.2 GB before graph edges, identifiers, metadata, replicas, allocator overhead, or caches. Multiple regions and versions multiply that footprint.
Compression, lower precision, or product quantization can reduce memory, but quality must be measured. Include:
active and shadow index versions;
replication for availability;
metadata-filter storage;
item-ID mappings;
graph or partition overhead;
batch-embedding compute;
index-build compute and temporary storage; and
network cost between serving and index layers.
Query capacity
Estimate peak queries per second by surface, query-tower inference cost, feature-store reads, ANN shard fan-out, over-retrieval count, and ranker load. Run load tests with realistic filter selectivity and hot keys. Uniform synthetic queries frequently miss real contention patterns.
Total cost of quality
A smaller embedding or compressed index may save infrastructure and lose tail recall. A larger candidate set may improve ranking opportunity and increase ranker latency. Build a Pareto frontier rather than selecting the maximum offline metric regardless of cost.
Production Monitoring and SLOs
Layer | Metrics | Example failure |
event data | volume, delay, bot rate, event mix, join success | positive examples no longer reflect production behavior |
training | loss, gradient health, norm distribution, sampler composition | collapse, exploding norms, or sampler drift |
representation | query/item norms, centroid, variance, duplicates, neighbor churn | shared space shifted unexpectedly |
candidate coverage | active items embedded, missing/failed vectors, age | catalog is partially absent |
index | build status, searchable lag, ANN Recall@K, memory, shard balance | index is stale or approximate recall degraded |
serving | query inference and ANN p95/p99, errors, timeouts, cache hit | latency or dependency incident |
filtering | candidates before/after filters, empty rate, authorization rejects | retrieval is incompatible with eligibility |
source mix | candidate and displayed share by source | two-tower source overwhelms the pool |
quality | full-corpus recall, cold-item recall, coverage, diversity | offline relevance or catalog health regressed |
outcomes | exposures, clicks, conversions, hides, returns, retention | recommendations do not create value |
Set SLOs for freshness, coverage, latency, availability, and ANN quality. Keep a lightweight exact-recall canary set in recurring monitoring. A healthy endpoint returning stale or incompatible candidates is not a healthy recommender.
The deployment pipeline should use the same controls as other production ML systems: artifact lineage, automated validation, staged promotion, and rollback. See CI/CD for machine learning, continuous training and automated retraining pipelines, and the Codersarts MLOps service.
Security, Privacy, and Governance
User embeddings are personal data when they encode user behavior
Pseudonymous vectors can reveal interests or account patterns. Apply purpose limitation, retention, access control, deletion handling, encryption, and auditing to raw events, features, profiles, training data, checkpoints, and caches.
Deleting a user row from an online store may not remove its influence from an already trained model. Define the legal and operational process for retraining, unlearning where applicable, and artifact expiry with counsel and governance teams.
Prevent cross-tenant retrieval
In B2B platforms, one organization must not retrieve another tenant’s restricted items. Use isolated indexes or enforceable tenant filters, authenticate every request, and deterministically recheck authorization after retrieval. Never expose unauthorized item titles in logs, reason codes, or fallback results.
Govern training labels and objectives
Clicks can optimize addictive or misleading outcomes. Purchases can favor high-price items. Historical exposure can encode discrimination or supplier bias. Document the intended objective, limitations, sensitive cohorts, and supply-side effects. Review not only accuracy but who receives and who supplies recommendations.
Explainability
A dense dot product does not provide a faithful human explanation by itself. Generate reason codes from auditable context recent seed items, shared categories, declared interests, or source type and verify that they are true. Do not infer meaning from individual embedding dimensions.
Worked Example: A B2B Marketplace with 60 Million Listings
Consider a marketplace serving procurement teams across regions. It has 60 million active listings, millions of buyers, strict account entitlements, and a 150-millisecond end-to-end recommendation budget.
The initial problem
The existing system retrieves popular items within the current category, then uses a gradient-boosted ranker. It is fast but repeats head inventory, underperforms for specialized buyers, and cannot search the full catalog with personalized features.
The first two-tower design
The query tower consumes account segment, market, recent categories, weighted recent item embeddings, search context, and surface. The candidate tower consumes item ID, taxonomy, supplier, structured specifications, and text embedding. Both emit 192-dimensional normalized vectors.
The team trains on qualified product-detail views and purchases, constructs histories at event time, and starts with corrected in-batch plus uniform negatives. Exposed-but-skipped items become a separate hard-negative pool after analysts confirm they are eligible and visible.
What testing reveals
An ID-heavy model wins aggregate Recall@500 but performs poorly on listings younger than seven days. Adding metadata and text improves cold-item recall. Increasing hard negatives improves same-category discrimination but reduces substitute coverage because many “negatives” are actually acceptable alternatives. The final sampler uses a smaller hard-negative share and masks known positives across a rolling window.
The ANN team evaluates HNSW and an IVF-compressed design against exact dot-product retrieval. HNSW meets recall but exceeds memory targets at two simultaneously deployed index versions. The compressed alternative meets memory but loses recall on specialized tail categories. The production design routes large general categories to the compressed index and keeps smaller high-value specialized partitions exact or high-recall.
Serving design
The system retrieves 1,200 two-tower candidates, 200 item-neighbor candidates, 100 content candidates, and a small exploration set. Tenant, contract, geography, and availability filters are enforced during retrieval when possible and checked again before ranking. A richer ranker evaluates 800 deduplicated candidates and returns 30 items.
Release gates
The launch requires:
full-catalog Recall@100, @500, and @1,200;
cold-item and specialist-category recall;
ANN recall against exact search;
zero unauthorized results in adversarial tests;
source-to-index freshness within ten minutes;
p95 retrieval and end-to-end latency targets;
catalog and supplier coverage guardrails; and
an online experiment on qualified engagement and procurement outcomes.
The architecture succeeds because the company treats the model, item index, filters, ranker, and measurement loop as one retrieval product—not because it selected a fashionable network shape.
Common Failure Modes and Corrective Actions
Symptom | Likely cause | How to confirm | Corrective action |
popular items dominate every query | in-batch frequency bias, dot-product norm, or labels mirror exposure | recall and norm by popularity decile | sampling correction, norm controls, balanced negatives, coverage objectives |
new items never appear | candidate tower relies on item IDs or index updates are slow | recall/coverage by item age and pipeline lag | add content features, fast-path embedding, exploration |
offline recall is high but online quality falls | sampled evaluation or target mismatch | full-catalog metrics and source-level experiment | evaluate full corpus, revise labels/objective |
ANN results differ sharply from exact | poor index tuning, compression, or metric mismatch | ANN Recall@K by cohort | correct metric, tune probes/search effort, change index or dimension |
restrictive filters return empty pools | global retrieval followed by aggressive filtering | pre/post-filter candidate count | partition, filtered ANN, over-retrieve, safe fallback |
model launch causes random-looking results | new query tower served with old candidate index | version telemetry | atomic model-index routing and rollback |
query recommendations lag current session | stale precomputed user vector | compare online versus cached profile age | combine live session and long-term vectors |
hard-negative mining hurts recall | false negatives or overly narrow boundary | expert review and alternate-positive rate | mask positives, diversify negatives, reduce hard-negative weight |
one interest crowds out others | single-vector query compression | recall by interest cluster and history diversity | multi-vector query or source-per-interest retrieval |
one tenant sees another’s item | filter or cache boundary failure | adversarial authorization test and logs | tenant isolation, scoped cache keys, deterministic authorization |
embedding memory exceeds forecast | graph/index overhead and parallel versions omitted | actual bytes per item by component | capacity model, compression, lower dimension, partitioning |
ranker rarely uses two-tower candidates | weak retrieval or uncalibrated source merging | candidate-to-display survival rate | retrain, recalibrate, change quotas, remove redundant source |
When a Two-Tower Model Is the Right Choice
Use it when:
the eligible catalog is too large for exhaustive pairwise scoring;
candidate generation needs personalization or contextual retrieval;
item embeddings can be computed independently of the live query;
a dot product or cosine score provides useful first-stage recall;
the organization can operate embedding and ANN pipelines;
there is sufficient interaction or relevance data for contrastive training; and
a downstream ranker or policy layer can refine the result.
Typical domains include commerce, media, jobs, advertising, marketplaces, learning platforms, social feeds, enterprise content, and large knowledge or service catalogs.
When Not to Use It or Not Yet
Do not make a two-tower model the default when:
the catalog is small enough for exact ranking within the latency budget;
rules, search, or item neighborhoods already meet the product need;
training labels are too sparse or unreliable to learn the shared space;
most relevance depends on non-factorizable query-item interactions;
strict compatibility can be solved only by deterministic matching;
the team cannot maintain synchronized model and index versions; or
the expected business lift does not justify training and serving cost.
For smaller catalogs, direct ranking may be simpler and higher quality. For explainable co-behavior relationships, item-based collaborative filtering may be a better baseline. For new-item similarity, structured metadata or content embeddings may provide value sooner. Architecture should follow the retrieval contract, not model prestige.
A Production Implementation Roadmap
Phase 1: establish the retrieval baseline
define corpus, surfaces, outcomes, eligibility, latency, and freshness;
create temporal train/validation/test data;
measure popularity, item-based, and content-based candidate sources;
implement full-catalog Recall@K;
audit exposure and item-frequency distributions; and
document the fallback.
Phase 2: train a simple factorized model
use reproducible user/item features;
begin with a modest embedding dimension;
test dot product versus normalized cosine deliberately;
train with in-batch negatives and accidental-hit masking;
add sampling correction and a controlled negative mix; and
evaluate head, tail, cold, and short-history cohorts.
Phase 3: prove serving geometry
export both towers and the representation specification;
batch-embed the complete eligible catalog;
establish exact-search results;
benchmark ANN recall, latency, memory, filtering, and updates;
validate candidate coverage and index freshness; and
load-test realistic traffic.
Phase 4: integrate ranking and operations
blend complementary candidate sources;
preserve provenance through the ranker;
log exposures, outcomes, versions, and filter reasons;
implement shadow indexes, canaries, and atomic routing;
create SLOs and incident runbooks; and
test security, deletion, and tenant isolation.
Phase 5: validate business impact
run a controlled online experiment;
monitor both demand- and supply-side outcomes;
inspect source survival from retrieval to display;
review bad recommendations with domain teams;
promote only within guardrails; and
schedule retraining from measured drift, not habit alone.
Enterprise Readiness Checklist
Retrieval contract
[ ] The eligible corpus and recommendation surfaces are explicit.
[ ] Candidate count, Recall@K, latency, freshness, and fallback targets are approved.
[ ] Candidate generation is separated from ranking and policy enforcement.
[ ] Non-factorizable features have a downstream home.
Data and training
[ ] Positives represent a documented user or business outcome.
[ ] Histories and features are reconstructed at event time.
[ ] Exposure, position, surface, and sampling probability are retained where possible.
[ ] Bots, repeated events, and correlated sessions are controlled.
[ ] Negative sources, false-negative handling, and sampling correction are versioned.
[ ] Temporal full-catalog evaluation is a release gate.
Embedding space
[ ] Query and candidate towers share a versioned dimension, metric, and normalization rule.
[ ] Embedding dimension was chosen from quality-cost measurements.
[ ] Vector norms and neighbor distributions are inspected by cohort.
[ ] Cold-user and cold-item behavior is intentional.
[ ] Sensitive and rapidly changing features are handled appropriately.
Index and serving
[ ] Exact retrieval is the ANN oracle.
[ ] ANN recall is measured by filter, category, locale, and item age.
[ ] Index memory includes overhead, replicas, and parallel versions.
[ ] Model-index compatibility is enforced in routing.
[ ] Authorization and eligibility are rechecked after retrieval.
[ ] Latency is decomposed by feature, inference, ANN, hydration, merge, and ranking.
Operations and governance
[ ] Representation, model, feature, and index lineage is auditable.
[ ] Freshness, coverage, latency, availability, and ANN recall have SLOs.
[ ] Shadow, canary, rollback, and fallback paths are tested.
[ ] User-vector retention and deletion policies are defined.
[ ] Tenant isolation and cache boundaries pass adversarial tests.
[ ] Online experiments include long-term and supply-side guardrails.Frequently Asked Questions
Is a two-tower model the same as matrix factorization?
Both represent users or queries and items in a shared latent space and often score them with a dot product. Matrix factorization typically learns direct user and item latent factors. Two-tower models can use neural networks and rich features such as histories, context, metadata, text, and images, which helps generalization and cold start. A simple matrix-factorization baseline remains valuable.
Why is it called a dual encoder?
The system has two encoders: one for the query side and one for the candidate side. Search and NLP literature often says “dual encoder,” while recommendation literature commonly says “two tower.” The factorized scoring property is the important part.
Can both towers use the same network weights?
They can, but usually do not because query and item features differ. Weight sharing is appropriate when both sides represent the same kind of object, such as item-to-item retrieval or certain semantic matching tasks. The output space must be compatible whether weights are shared or separate.
How many candidates should the model retrieve?
Set KK from the candidate-recall curve, downstream ranker capacity, filter-drop rate, source blending, latency, and cost. Common production values range from hundreds to thousands, but there is no universal number. Plot incremental recall and business value as KK grows.
What embedding dimension should we use?
Start modestly and benchmark several dimensions. Evaluate full-catalog recall, cold and tail cohorts, ANN recall, memory, build time, and serving latency. More dimensions do not guarantee better recommendations.
Are in-batch negatives enough?
They are an efficient baseline, not a universal solution. Correct or account for frequency bias, mask accidental positives, and evaluate a mixture that includes uniform and carefully mined hard negatives. The best composition depends on catalog distribution and outcome labels.
How often should the ANN index be rebuilt?
Use the item-change rate and freshness contract. Some systems incrementally insert new embeddings throughout the day and run periodic clean rebuilds. Major candidate-tower changes require a complete compatible index. Urgent eligibility and deletion changes need a faster policy path than a model rebuild.
Can the two-tower model be the final ranker?
For simple products it can return the final list, but it cannot efficiently express rich request-item cross-features or slate interactions. At enterprise scale, it is normally a retrieval stage feeding a separate ranker and constraint layer.
How do we handle multiple user interests?
Use recent-context features, separate short- and long-term vectors, multiple interest heads, or retrieval from several seed items. Merge and deduplicate candidates before ranking. Validate whether the added retrieval cost improves cohort recall and online outcomes.
How do we know the vector index is not degrading quality?
Compare ANN top-KK with exact top-KK for a recurring representative query set. Track ANN Recall@K alongside latency, memory, filter selectivity, index age, and shard health. Re-run the benchmark after model, dimension, metric, compression, or index-parameter changes.
What should an enterprise proof of concept prove?
It should beat simple retrieval baselines on temporal full-catalog metrics; demonstrate cold, tail, locale, and short-history cohorts; quantify negative-sampling choices; meet exact-versus-ANN recall and latency targets; enforce security filters; and define a realistic model-index deployment path. A notebook trained against sampled negatives is not a production proof.
Build a Retrieval System, Not Merely Two Neural Networks
The two-tower pattern makes personalized search across a massive catalog computationally practical. Its power comes from a strict interface: independently compute query and candidate embeddings, compare them cheaply, retrieve broadly, then let ranking and policy layers make the final decision.
The enterprise work lies around that interface. Training data must represent time and exposure correctly. Negative sampling must be treated as part of the model. Content and identity features must support both mature and cold items. The embedding space needs a versioned contract. ANN quality must be compared with exact retrieval. Query towers and candidate indexes must be promoted atomically. Full-catalog and online evaluation must prove that the system retrieves valuable options rather than merely reproducing historical popularity.
Codersarts helps teams design and implement large-scale recommendation platforms across training data, two-tower models, content and behavioral embeddings, ANN infrastructure, ranking, deployment, evaluation, and monitoring. Explore our machine learning development services, machine learning deployment services, and MLOps services.
Building candidate retrieval for a large catalog or user base? Discuss your recommendation-system architecture with Codersarts.
Primary References
Huang, P.-S., et al. “Learning Deep Structured Semantic Models for Web Search using Clickthrough Data.” CIKM, 2013. Microsoft Research.
Covington, P., Adams, J., and Sargin, E. “Deep Neural Networks for YouTube Recommendations.” RecSys, 2016. Google Research.
Yi, X., et al. “Sampling-Bias-Corrected Neural Modeling for Large Corpus Item Recommendations.” RecSys, 2019. Google Research.
Yang, J., et al. “Mixed Negative Sampling for Learning Two-tower Neural Networks in Recommendations.” The Web Conference, 2020. Google Research.
Lindgren, E., et al. “Efficient Training of Retrieval Models using Negative Cache.” NeurIPS, 2021. Google Research.
Krichene, W., and Rendle, S. “On Sampled Metrics for Item Recommendation.” KDD, 2020. Google Research.
Malkov, Y. A., and Yashunin, D. A. “Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs.” IEEE TPAMI, 2020. IEEE.
Johnson, J., Douze, M., and Jégou, H. “Billion-scale similarity search with GPUs.” 2017. arXiv.
TensorFlow Recommenders. “Factorized Retrieval Task.” Official documentation.
Comments