Recommendation System & A/B Testing
Showing each user the products/content they like, and proving through experiments that a new feature is really better.
23.1 Why is it used?
- Among crores of products a user cannot find things alone - 'Customers also bought' and 'Recommended for you' improve discovery and sales.
- Netflix has publicly said that a large share (about 80%) of what users watch comes from recommendations.
- A personalized home page raises engagement, cart size and retention.
- A/B testing lets you decide with data instead of guesses - does a new algorithm or button colour really raise conversion?
- Feature flags roll a new feature out to 1% of users first, then gradually to everyone - less risk.
23.2 How it works
- Collaborative filtering: 'users like you who took X also took Y' - from a user-item interaction matrix; item-to-item CF is Amazon's famous approach.
- Content-based filtering: matching item attributes (category, brand, genre) to find similar items.
- Matrix factorization / embeddings: convert users and items into vectors, dot product = preference score; two-tower deep models.
- Two-stage pipeline: candidate generation (~500 out of lakhs, ANN vector search) -> ranking model (top 20 via ML) -> business rules (stock, diversity).
- Feature store: features (a user's last 10 clicks, item CTR) are the same in offline training and online serving; no training-serving skew.
- Cold start: a new user/item has no data - start with popular/trending or content-based.
- A/B testing: split users randomly (hash(user_id)) into control A and variant B, compare a metric (CTR, conversion), and check statistical significance.
- Feature flags: deploy code separately from releasing the feature; gradual rollout, kill switch, targeting (city, app version).
- 1.Offline (daily): Orders data - who bought what together
- 2.Offline CF job: item-to-item similarity, phone -> cover, charger, earphones
- 3.KV store / cache: similar items pre-computed
- 4.Online (milliseconds): the user opens the phone's product page
- 5.Reco service: lookup by product_id
- 6.Rules: stock + price + diversity (two-stage ML: ~500 candidates ANN -> ranking top 20 -> business rules)
- 7.Output: 'Customers who bought this also bought': cover, charger, earphones
Cold start: new user -> popular / trending fallback. Feature store: same features in training + serving
- 1.Users: all visitors, hash(user_id) % 100 bucket (random + sticky)
- 2.A: old CF model, 50% of users, control
- 3.B: embedding model, 50% of users, variant
- 4.Metric compare: add-to-cart rate, 2 weeks, significance check
- 5.If B wins: 100% rollout via feature flag (gradual: 1% -> 10% -> 100%)
- 6.If a bug or metric drop: kill switch, all users back to A immediately
Tools: Optimizely A/B, GrowthBook, LaunchDarkly flags, Amazon Personalize, Feast, FAISS / Milvus. Mistake: stopping the test as soon as the result looks good (peeking) - decide the sample size first
23.3 Tools you can use
| Tool | Type | How it is used |
|---|---|---|
| Amazon Personalize / Vertex AI / Azure ML | Managed ML recommendation | Give interaction data, and the managed service trains a recommendation model and serves it through an API. |
| Feast / Tecton / SageMaker Feature Store | Feature store | Offline (warehouse) + online (Redis/DynamoDB) feature store, same feature definitions in training and serving. |
| FAISS / Milvus / OpenSearch kNN | Vector similarity search | Finding nearest-neighbour items from embeddings in milliseconds (candidate generation). |
| Optimizely | SaaS experimentation platform | A/B and multivariate tests, a stats engine, web and server-side experiments. |
| GrowthBook | Open-source feature flags + experiments | Self-host or cloud; analyzes experiments on data in your warehouse (BigQuery/Snowflake). |
| LaunchDarkly | SaaS feature flag management | Enterprise feature flags, gradual rollout, targeting, kill switch; experimentation add-on. |
23.4 Comparison: Optimizely vs GrowthBook vs LaunchDarkly
| Point | Optimizely | GrowthBook | LaunchDarkly |
|---|---|---|---|
| Main focus | Experimentation (A/B testing) | Feature flags + warehouse-native A/B | Feature flag management |
| Hosting | SaaS | Open-source self-host or cloud | SaaS |
| Stats / analysis | Strong built-in stats engine | Analysis with SQL on your own data warehouse | Basic, experimentation add-on |
| Feature flags | Yes | Yes | Best-in-class, enterprise grade |
| Data ownership | Data sits with the vendor | Data stays in your warehouse | Data sits with the vendor |
| Cost | Expensive, enterprise pricing | Free open-source, cheap cloud plans | Expensive, seat/usage based |
| Best for | Heavy experimentation by marketing/product teams | Data teams + companies with a warehouse, low budget | Safe rollouts, large engineering teams |
23.5 When to choose which
- Optimizely: product/marketing teams run lots of experiments and want a ready stats engine.
- GrowthBook: your data is already in Snowflake/BigQuery and you want open-source at low cost.
- LaunchDarkly: the main need is safe release, gradual rollout and a kill switch.
- Amazon Personalize: the ML team is small and you want recommendations quickly on AWS; at larger scale, custom models + a feature store.
- Start with simple 'frequently bought together' (co-occurrence count), then move to ML.
23.6 Real-world example
Real world: Amazon 'Customers who bought this also bought' + A/B test
- Amazon has publicly described item-to-item collaborative filtering (2003 paper) - similarity between items instead of users.
- An offline job calculates from orders data which items were bought together with a 'phone' - cover, charger, earphones.
- The list of similar items is pre-computed for every product and kept in a fast key-value store/cache.
- The user opens a product page, the service fetches the top similar items by product_id and applies stock and price rules.
- The team builds a new embedding-based model; hash(user_id) puts 50% of users on A (old) and 50% on B (new).
- After 2 weeks, B's add-to-cart rate is statistically significantly higher, so it goes to 100% rollout via a feature flag.
- If B shows any bug or drop, the kill switch immediately puts everyone back on A.
23.7 Interview tips
- Common mistake: stopping an A/B test too early (peeking) when the result looks good - false positive; decide the sample size first.
- Common mistake: computing features differently in training and serving - training-serving skew, the model misbehaves. Ignoring cold start is also wrong - a new user gets empty recommendations; keep a popular-items fallback.
- Interview tip: explain the two-stage design of candidate generation -> ranking -> re-ranking, and the offline vs online pipeline.
- Worth quoting: Netflix ~80% of viewing from recommendations; in a two-stage design ~500 candidates out of lakhs, then top 20; a 50/50 split with a 2-week test.