1 of 47

Using a powerful Business-Contextual Image Retrieval

Paul-Louis Nech | Senior ML Engineer @Algolia | M. ScEng @EPITA

2 of 47

Who we are

Powering discovery

3 of 47

Agenda

User expectations

The challenge

Start with Vectors

Giving finer control

1

2

3

4

4 of 47

01

—

User expectations in 2023

Powering discovery

5 of 47

Users expect �instant interactions�

0.5 sec delay causes 20% traffic drop�
Users pick a movie in �1.8 seconds�
100ms delay will result in �1% revenue loss

Powering discovery

6 of 47

Returning results quickly

doesn’t matter

If it’s the wrong ones

Powering discovery

7 of 47

Users expect relevant results

#1 search result gets 27.6% of clicks�
10th result gets 10x less clicks than 1st�
37% of users only engage with first page

~ Backlinko, Google CTR Stats

Powering discovery

8 of 47

On mobile and via chat,�the first result(s)

is all that matters

Powering discovery

9 of 47

Users expect �Understanding�

�See “Like Having a Really bad Personal Assistant: �The Gulf between User Expectation �and Experience of Conversational Agents”,�Luger & Sellen, 2016

Powering discovery

10 of 47

“We find user expectations dramatically out of step �with the operation of the systems, �particularly in terms of �known machine intelligence, �system capability and goals.”

Ewa Luger

Microsoft Research, 2016

Powering discovery

11 of 47

Users expect MultiModality

Pinterest’s PinSage recommender system

based on “visual and annotation embeddings as input features”

Netflix’ artwork personalization system

optimizing CTR & view share per country and per user

Powering discovery

12 of 47

The Challenge:

�Building an API which meets these expectations

Powering discovery

13 of 47

Marketplaces�with short-lived items

Powering discovery

14 of 47

“Find me this”�shopping experience

Powering discovery

15 of 47

Step by step�user journeys

Powering discovery

V1 = v(product['image']) + 0.2 * v(category['title'])

V2 = v(product['image']) + 0.1 * v(previous_product['image'])

V3 = v(product['image']) - 0.3 * v(not_this_product['image'])

Powering discovery

16 of 47

02

—

Baseline: �Image Vectors Retrieval

Powering discovery

17 of 47

From Images to Vectors

Resnet18 | Deep Residual Learning for Image Recognition�https://arxiv.org/abs/1512.03385

Fixed length vectors�(512)

Arbitrary-sized images

img2vec = Img2Vec(cuda=False, model="resnet18", layer_output_size=512)

Powering discovery

18 of 47

From Vectors to Binary Hashes: LSH

// returns a dense binary vector using signals >0 as activation threshold.

func (h *NH) Hash(v []float32, b []byte) []byte {…}

Similarity Search in High Dimensions via Hashing

A. Gionis, P. Indyk, R. Motwani

http://www.vldb.org/conf/1999/P49.pdf

Powering discovery

19 of 47

From Vectors to Binary Hashes: Semantic Hashing

// returns a dense binary vector using signals >0 as activation threshold.

func (h *NH) Hash(v []float32, b []byte) []byte {…}

Semantic Hashing

R. Salakhutdinov, G. Hinton

http://www.cs.columbia.edu/~smaskey/CS6998-0412/supportmaterial/semantic_final.pdf

Powering discovery

“We show how to learn a deep graphical model of the word-count vectors obtained from a large set of documents.

The values of the latent variables in the deepest layer are easy to infer and give a much better representation of each document [...].

When the deepest layer is forced to use a small number of binary variables (e.g. 32), the graphical model performs “semantic hashing”: �Documents are mapped to memory addresses in such a way that semantically similar documents are located at nearby addresses.”

Interesting insight in the paper: “Fast retrieval methods often rely on intersecting sets of documents that contain a particular word or feature.”

E.g. Faceting, or ‘matches keyword X’, or even advanced computed features like ‘predicted to be out of stock soon’.

So we might as well train a model to learn to find the best binary features!

By training a deep model to find binary features that separates similar documents from different ones, we can learn an efficient semantic binary hashing function.

And once we have vectors which represent well the partitions in our data, we can retrieve them efficiently:

20 of 47

From Hashes to Results: Retrieval with HNSW

Efficient and robust ANN search using� Hierarchical Navigable Small World graphs

Yu. A. Malkov, D. A. Yashunin�https://arxiv.org/abs/1603.09320

Powering discovery

21 of 47

Putting it all together

Vectorization + Hashing

Nearest Neighbors (ANN) retrieval via HNSW

Powering discovery

22 of 47

Putting it all together

Indexing at 47 qps, �with p95 latency under 500ms

Powering discovery

23 of 47

Preliminary results

Powering discovery

24 of 47

Powering discovery

25 of 47

Preliminary results

Powering discovery

26 of 47

Preliminary results

Powering discovery

27 of 47

The Limitation:

�What you See is

(mostly)

What you Get

Powering discovery

28 of 47

The “What you see is what you get” problem

Powering discovery

29 of 47

The “What you see is what you get” problem

Nearest (different) neighbor

Input

Best match

Powering discovery

30 of 47

The “What you see is what you get” problem

Nearest (different) neighbor

Input

Best match

Powering discovery

31 of 47

Powering discovery

What we would like instead

Vectorization

With img2vec

🔢

Retrieval

With HNSW

🎣

Hashing

With a Learned LSH

🪓

Filtering

Using object-level metadata

🪣

Rules

boost/bury on demand

📐

Powering discovery

Add a step with

Reviewing Image Recommendation API eBook

Variant A : for business owners

Target release: as soon as our Image Recommendation API is open for business (at least Public Beta status - ETA Q4 2023)

Welcome Changes from ebook v1 (as of2023-10-16)

Page 3: from keyword to multimodal search

Good intro imo - let’s be careful of release timing, as the content states “we created a powerful image and recommendation service” while it’s not yet available to the general public as of today

Page 4: the business of images: ✅

Page 5: potential use-cases

This should be removed or severely edited: that content was giving ideas to API builders how they could leverage image vectors, while the API we’re building didn’t implement any of these scenarios. I’d keep it for the “ML engineer interested in building such systems” ebook, not for the “business reader looking to buy something from Algolia” one

Page 6: what would you build CTA? ✅

Page 7-13: Education on modern Recommendation User Expectations ✅

Page 14-23: Education on Vectors and how to build such an image recommendation API

This is valuable for people who want to build an API such as ours. For business readers who’d use our API instead of rebuilding their own, this might be too much details.�If we really want to still go with this level of detail, let’s be extra careful with phrasing, e.g. the rephrasing from original talk to this ebook form introduced this quite false statement: �”[...] a clever technique called hashing. The concept was first introduced as "semantic hashing" [...]”

Hashing has been known since the 1940s. The clever new technique is the “semantic hashing” idea by Hinton.

Page 17 is hard to grasp even as a Senior ML engineer who wrote the original talk. �I’d either keep the formatting in the original slide (highlighting its core ideas: graphical model, world-count vectors, small number of binary variables, semantically similar => nearby addresses)

32 of 47

03

—

Designing an image rec. API �for broader use-cases

Powering discovery

33 of 47

Leveraging object-level properties

Powering discovery

34 of 47

Filtering results based on object-level properties

Powering discovery

35 of 47

API-based Filtering to enable unexpected use-cases

{

threshold: 0.3,

maxRecommendations: 5,

queryParameters: { parameters },

fallbackParameters: { parameters }

}

Powering discovery

36 of 47

QueryParameters: filtering the happy path

Powering discovery

37 of 47

QueryParameters: concrete example

queryParameters={{

facetFilters: [

`roastLevel:${selectedProduct.roastLevel}`,

`inStock: ${true}`,

],

}}

Powering discovery

38 of 47

QueryParameters: concrete example

queryParameters={{

facetFilters: [

`store:San Francisco`,

],

}}

Powering discovery

39 of 47

FallbackParameters: recover from not-happy path

fallbackParameters={{

facetFilters: [

`category:${selectedProduct.category}`,

`colourFilter: ${selectedProduct.colourFilter}`,

`inStock: ${true}`,

],

}}

Powering discovery

How do we handle ‘empty state’ of no recommendations?

“Something is better than nothing”, so good UX == good fallback

---

Now sometimes your need is more specific than ‘in case of no result’.

Maybe your marketers want to expose a promotion for Samsung phones when users look at an iphone’s similar images?
Maybe you want to make sure than when someone watches a specific documentary, you suggest a specific new one on top of similar screengrabs?
Maybe you want to make sure, when someone looks at items from the categories “Documentary” and “Kids”, to Bury at the bottom a controversial video?

Vector embeddings (e.g. of the category, or of -v(controversary_video), can only bring you so far.

For all these use-cases and many others, you need a last layer of postprocessing: A FLEXIBLE RULES SYSTEM!

40 of 47

Rules: enable campaigns and merchandising

Powering discovery

41 of 47

Rules: enable campaigns and merchandising

Powering discovery

42 of 47

Rules: enable campaigns and merchandising

Powering discovery

43 of 47

From an API to a Service

Powering discovery

44 of 47

From an API to a Service

Powering discovery

Integration

GitHub code reviews�
CircleCI integration � & deployment�
GCP config connector for � review environments �
ArgoCD for review apps � & continuous deployment �

Operation

Kubernetes for deployment� and worker autoscaling �
Google Cloud managed services�
DataDog for monitoring � & observability�
Slack notifications for alerting

Powering discovery

45 of 47

To summarize

Powering discovery

46 of 47

To summarize

Hashing vectors is key to scaling

Vectorization is a powerful core tech

An expressive API unlocks many use-cases

Paul-Louis Nech | ECIR23 Industry Day | Slides: alg.li/ecir23-talk 🔗

Powering discovery

Vectorization technologies are a powerful core technology, which enables serious image retrieval capabilities.

Vector Hashing algorithms, such as the LSH & Semantic Hashingfamily, are key to scaling vectors beyond POC to reach the scale Industrial use-cases expect

A powerful & scalable engine is not enough for SaaS as you need to serve different, sometimes opposing use-cases. It takes just a little work to wrap it into a pretty expressive API and unlock those use-cases!

An API doesn’t run itself, and unless you are Google, it makes business sense to leverage the existing ecosystem of tools for reviews, deployment, monitoring, scaling, etc

I hope you found this journey interesting - we’re far from finished on this project, so I welcome your questions and your suggestions to continue the conversation around this fascinating puzzle!

47 of 47

Building a Business-Contextual Image Retrieval API

Paul-Louis Nech | ECIR23 Industry Day

�Slides: alg .li /ecir23 -talk 🔗

Powering discovery