AI від Apple
Як Apple розкриває потенціал AI: Секрети Apple Intelligence та ключові дослідження
Hello
there!
Maksym Kmet
Senior Data Scientist
We befriend people with technology
$9M+
З 24 лютого 2022 року задонатили на підтримку України
30M+
Понад 30 мільйонів користувачів наших продуктів
180+
Країн, в яких живуть наші користувачі
5
Кожен п’ятий Mac на планеті має хоча б один продукт MacPaw
2008
Рік заснування компанії
ML Research
ML Frameworks
Advanced
Hardware
Як Apple стає AI-first company
Браковані NVIDIA GPU у новому Macbook та судові позови
Apple AI:
Передумови
2008
2016
2017
2020
2024
Apple Intelligence локально на iPhone 15 Pro
Apple Silicon (M1)
Neural engine для прискорення ML inference
GPU від AMD у Macbook
ML research
А що крім заліза?
Frameworks for ML
ML Frameworks
build and train a model with the Create ML app
use Core ML Tools to convert models built with other libraries into the Core ML
fine-tune on-device with users data
ML framework used across Apple products for fast inference with easy integration of pre-trained models on the edge
CoreML (2017)
build and train a model with the Create ML app
use Core ML Tools to convert models built with other libraries into the Core ML
fine-tune on-device with users data
ML framework used across Apple products for fast inference with easy integration of pre-trained models on the edge
CoreML (2017)
MLX is an array framework optimized for the unified memory architecture of Apple silicon. The NumPy-like API makes it familiar to use and flexible.
MLX (2024)
MLX is an array framework optimized for the unified memory architecture of Apple silicon. The NumPy-like API makes it familiar to use and flexible.
MLX (2024)
AXLearn is a library built on top of JAX and XLA to support the development of large-scale deep learning models
AxLearn (2024)
ML
Research
On-screen Entities (ex. phone number on a restaurant webpage)
Background Entities (ex. alarm ringing or music in the background)
Conversational Entities (entities from the conversation with user (ex “Call Mom” or “Show path to the first restaurant from the list”)
Reference resolution as language modeling.
Understand what user is referring to in the conversation even if it is not explicitly stated.
Small model Flan-T5 matches GPT-4 on benchmarks (their own)
ReALM (2024)
Elementary Tasks:
Referring:
Grounding:
Advanced Tasks:
Multimodal large language model for enhanced understanding of mobile UI screens. (encoder CLIP-ViT-L + Vicuna + projection layers + �visual sampler)
Ferret-UI (2024)
Open-ELM (May 2024)
DCLM-7B (July 2024)
Large Language Models
Open-ELM and DCLM-7B
How Apple �Intelligence works
Were our guesses correct?
Apple Intelligence
Features
…sounds like ReALM or Ferret-UI
Context Awareness
…sounds like Open-ELM
Language generation
Image generation
What we actually �know
Architecture
of Apple Intelligence
Architecture
of Apple Intelligence
On-device models: foundation models (3B params) with a set of small LoRA adapters
App Intents ToolBox: predefined functions such as Open Email, Start playing music, create a new Reminder. Each LoRa adapter is for a specific type of intents
Server models: The stack of Apple models in the Private Cloud
Cloud Compute: larger foundation LLM + LoRa
Orchestration: routes user request �to on-device model/ server /ChatGPT
Semantic index: is a vector database �of personal context (emails, photos, contacts …)
Architecture
of Apple Intelligence
described a photo of my beautiful baby as “group of people in a room with white walls.”
photo “clean up” function made me look bald
misinterprets the meanings of text, inverts people’s names
lifted to priority a scam email about suspended account
from a column by The Washington Post
Apple Intelligence can be kind of dumb
Apple Intelligence
Brief history �of Apple computational powers
ML Frameworks
ML Research
Q&A