1 of 10

Private Eye

An on-device assistant for blind people

Point, tap, and it tells you what’s there, out loud.

Private Eye

1

2 of 10

The idea

  • I research vision-language models (VLMs)
  • Could a VLM help someone who can’t see?
  • Private Eye: an assistant for blind people, on the iPhone
  • Point the phone, tap, and it tells you what’s there

WHAT IT SHOULD DO

  • Answer any question about what’s in front of you
  • Run on the phone: private, instant, offline

Private Eye

2

3 of 10

Ask about anything in front of you

  • Tap: hear what’s in front of you
  • Hold and ask any question
  • Reads labels, signs and screens
  • “How far is that?” answered by LiDAR
  • On the phone: no upload, no wait

Tap

“Canon EOS camera with a zoom lens on the desk. Laptop behind it, to the right.”

“Watch for a T-shirt that has code on it”

“Found a t-shirt that has code on it.”

Private Eye

Camera frames and replies from the app’s own log.

3

4 of 10

It remembers your people and your things

  • “This is Sarah” teaches a face
  • Then it says “Sarah, 3 feet ahead”
  • “Remember this as my water bottle”
  • “Where is my water bottle?” finds it
  • Or tells you where it was last seen
  • Memory never leaves the phone

Under the hood: YOLOE segmentation, ArcFace for faces, DINOv3 for belongings.

Tap

“Elias to your right. Charles ahead. 3 people in view. Two people walking on sidewalk, shopping bags.”

Tap

“Your water bottle. Sleep mask.”

Private Eye

Camera frames and replies from the app’s own log.

4

5 of 10

The last 10% is the hardest

  • Working MVP by the first evening
  • Camera, detector, LFM3-VL-2B and speech, all on the phone
  • In our own testing it did fairly well
  • Then we ran 500 real photos from blind users: 45% right
  • 115 of those photos can’t answer the question asked
  • On those, it made up an answer 58% of the time

Private Eye

5

6 of 10

When it can’t see, it makes things up

“What is my blood pressure reading?”

125/80 · 120/80 · 120/75 · 120/80 · 178/92

“How much sodium is in this package?”

1000 · 10mg · 0 mg ·

140 mg · 100 mg

“What kind of medication is in this bottle?”

“Antibiotics” · “antibiotic”

“eye drops”

Private Eye

Answers from 2B models, in red. None of them is in the photo.

6

7 of 10

So we had

to dig in.

Private Eye

7

8 of 10

Into the data, on Databricks

  • 31% of photos can’t fully answer the question: blurry, dark or out of frame
  • Crowd labels are noisy: we overturned 14% and refined another 37%

1 · DATASET

VizWiz

20,000 photos taken by blind people, each with a question and 10 crowd answers

→

2 · AI GATEWAY

Relabel in under an hour

Two GPT models behind one endpoint rewrite all 20,000 labels for $87. Key kept as a secret, every call logged

→

3 · UNITY CATALOG

One source of truth

Every label, answer and grade is a row: 12 tables, with 26,000 photos in a volume beside them

4 · FINE-TUNE, RUN 1

20,000 examples, 13 min

Trained on our own GPU. Invented answers fall from 58% to 9%

→

5 · AI GATEWAY + MLFLOW

Grade it, find the faults

gpt-6.1-sol grades each run through the gateway. Run 1 declines 71 photos it could answer and loses left and right

→

6 · FINE-TUNE, RUN 2

40,000 examples, 25 min

Same training, new data: 19,400 tap and where-is-it examples, 6,400 “what to do next” declines. Wrong declines halved. This is Witness-2B

Private Eye

All three runs in MLflow, both model versions in the Unity Catalog registry, dashboard and Genie on top.

8

9 of 10

Witness-2B: our fine-tune of LFM3

  • 7× fewer hallucinations than any rival
  • Most accurate, with under half the errors
  • Knows when it can’t see, and says what to do

“What is my blood pressure reading?”

“Glare obscures the reading. Tilt the phone.”

All 500 answers

Right

Partly

“Can’t tell”

Wrong

Hallucinations, 115 unanswerable photos

Share where the model invents an answer

Witness-2B

 66%

13% 

8%

Qwen3.5-2B

 50%

35% 

80%

Qwen3-VL-2B

 49%

32% 

70%

LFM3-VL-2B (stock)

 45%

33% 

58%

SmolVLM2-2.2B

 45%

31% 

57%

Gemma 4 E2B

 39%

38% 

57%

InternVL3.5-2B

 39%

39% 

74%

Private Eye

500 VizWiz photos, the same bare question to every model, graded by gpt-6.1-sol.

9

10 of 10

Try it

yourself!

  • Point the phone at anything and tap
  • Hold and ask a question
  • Turn a label away from the camera and ask what it says

Private Eye

10