1 of 53

4 of 53

Background

How do displays work?

http://j.mp/chrome-begin-frame

5 of 53

How do displays work?

Monitors constantly refresh their output�
Graphics driver generates new pixel data into graphics memory and then sends to the monitor

http://j.mp/chrome-begin-frame

6 of 53

How do displays work?

Application should provide a new frame to graphics driver before the monitor’s next refresh time

To generate a new frames before the upcoming refresh event, application should know when refresh will happen

http://j.mp/chrome-begin-frame

7 of 53

How do displays work?

VSync (Vertical Synchronization)

During the VSync pulse, vertical retraces happens
Image data can't be updated during the vertical retraces to prevent tearing

vsync

vertical retraces

vertical blank interval

vsync

http://j.mp/chrome-begin-frame

8 of 53

http://j.mp/chrome-begin-frame

9 of 53

How do displays work?

With vsync info, application knows time budget for generating new frame before next refresh

Frame Time
Interval

http://j.mp/chrome-begin-frame

10 of 53

11 of 53

BeginFrame

What is it and why do I care?

http://j.mp/chrome-begin-frame

12 of 53

What is a BeginFrame message?

Notification to CC (Chromium Compositor)

cc starts generation of new frame at the beginning of vsync period after receiving this message
frame - pixel data (or gpu command) needed to display the content on the screen

Layout

Composite

time

BeginFrame

arrives

Paint

JS

BeginFrame

arrives

JS

vsync period

http://j.mp/chrome-begin-frame

13 of 53

Why are BeginFrames needed?

vsync

BeginFrame

VS.

http://j.mp/chrome-begin-frame

14 of 53

Why are BeginFrames needed?

Render once during the vsync period

one frame can be displayed on the output at a time
reduced power consumption

Reduces latency
Uniform timestamps

http://j.mp/chrome-begin-frame

15 of 53

When are BeginFrames needed?

When CC has something which require the screen to be updated, BeginFrame message is requested

Animations, js, user & system inputs

Sending BeginFrame to CC in every vsync just wastes the battery

BeginFrame message is toggled by CC

http://j.mp/chrome-begin-frame

16 of 53

How are BeginFrames used?

ImplFrame - "Compositor"

BeginImplFrame - message to start ImplFrame
created and lived in impl(compositor) thread

MainFrame - "Blink"

BeginMainFrame - message to start MainFrame
created and lived in main(blink) thread

http://j.mp/chrome-begin-frame

17 of 53

How are BeginFrames used?

When BeginFrame message arrives,
ImplFrame generation is started in impl thread
When this impl frame needs re-layout/paint, ImplFrame requests MainFrame to main thread

ImplFrame generation could be delayed when MainFrame is high latency mode

main thread

impl thread

ImplFrame

MainFrame

time

ImplFrame

BeginImplFrame

commit

BeginMainFrame

MainFrame

http://j.mp/chrome-begin-frame

18 of 53

What does a BeginFrame contain?

Frame time - frame start time
Deadline - expected time of MainFrame generation
Interval - the difference between current frame time and next frame time

http://j.mp/chrome-begin-frame

19 of 53

main thread

impl thread

ImplFrame

MainFrame

time

BeginImplFrame

commit

BeginMainFrame

Draw

Activate

BeginFrame.

Deadline

BeginMainFrame.

Deadline

BeginFrame

swap

Vsync

http://j.mp/chrome-begin-frame

20 of 53

Unified BeginFrame API

XXXXXBeginFrameSource

BeginFrameSource

SetNeedsBeginFrame()

DidFinishFrame()

Add/Remove observer()

Timer

BeginFrameObserver

Scheduler

BeginFrameSource

SchedulerClient

BeginFrameObserver

OnBeginFrame()

LastUsedBeginFrameArgs()

SchedulerClient

ScheduledActionSendBeginMainFrame()

ScheduledActionCommit()

ScheduledActionAnimate()

...

CC

http://j.mp/chrome-begin-frame

21 of 53

Where do BeginFrames originate?

BeginFrame from outside of CC

ExternalBeginFrameSource

Generates BeginFrame inside CC

SyntheticBeginFrameSource
BackToBackBeginFrameSource (--disable-gpu-vsync)

Switchable BeginFrame source

BeginFrameSourceMultiplexer

http://j.mp/chrome-begin-frame

22 of 53

BeginFrame

Request “Please draw me pixels”

with Common abstract API

http://j.mp/chrome-begin-frame

23 of 53

Unified BeginFrame Scheduling

What is it and why do I care?

&

VSync Aligned Input

What is it and why do I care?

http://j.mp/chrome-begin-frame

24 of 53

BeginFrame Scheduling

Browser process

Renderer process

BeginFrame message

http://j.mp/chrome-begin-frame

25 of 53

State of BeginFrame Scheduling

Most platforms�BeginFrame message is generated on renderer - Not scheduled.

normally by SyntheticBeginFrameSource�

Android Chrome uses BeginFrame scheduling

http://j.mp/chrome-begin-frame

26 of 53

Unified BeginFrame Scheduling!

Makes all the systems work the same

use common code

Enables

VSync aligned Input handling to all platforms
and much more ideas!

http://j.mp/chrome-begin-frame

27 of 53

The problem

Input event is generated and delivered from browser at input scan rate
BeginFrame message is generated on renderer
This can lead drawing and input handling to happen out of order

http://j.mp/chrome-begin-frame

28 of 53

The problem

ImplFrame

vsync

BeginFrame message

MainFrame

Input

vsync

Draw

Input handling

ImplFrame

MainFrame

Draw

BeginFrame message

Input

main thread

impl thread

vsync-unaligned input handling

http://j.mp/chrome-begin-frame

29 of 53

Android Solution

Input event is coordinated to a vsync timing

Since the JellyBean (API v16)

If application registers Choreographer.FrameCallback, input events is always followed by the callback

http://j.mp/chrome-begin-frame

30 of 53

Android Solution in Chrome

java

c++

Input

vsync

Content layer

Compositor

Browser

Renderer

1

2

External BFS

BeginFrame generator

http://j.mp/chrome-begin-frame

31 of 53

The improvement

Improved vsync aligned input handling

Input handling

ImplFrame

Input

MainFrame

Input

Input handling

ImplFrame

MainFrame

BeginFrame

main thread

impl thread

Input

Further reading about vsync scheduling in android (by @skyostil):

http://bit.ly/chrome-sami-vsync

http://j.mp/chrome-begin-frame

32 of 53

BeginImplFrame

Request BeginMainFrame

Draw

OnVSync

Handling

Gesture

Scroll

Update

TouchEnd

BeginImplFrame

Handling

TouchEnd

BeginMainFrame

Gesture

Scroll

Update

OnVSync

http://j.mp/chrome-begin-frame

33 of 53

Unified BeginFrame Scheduling

The effort to bring Android’s BeginFrame

handling to other platforms�

VSync Aligned Input

Using Unified BeginFrame Scheduling

to replicate Android's input model

http://j.mp/chrome-begin-frame

34 of 53

Unified BeginFrame Scheduling

The effort to bring Android’s BeginFrame

handling to other platforms�

VSync Aligned Input

Using Unified BeginFrame Scheduling

to replicate Android's input model

BeginFrame

http://j.mp/chrome-begin-frame

35 of 53

Implementation

How are we going to do this?

http://j.mp/chrome-begin-frame

36 of 53

Lets start with Aura!

Aura is good initial target
It covers Linux, Windows and ChromeOS

http://j.mp/chrome-begin-frame

37 of 53

BeginFrame Scheduling on Aura

We need a BeginFrame generator in the browser process because there is no vsync callback registration api on Aura platform

Make a new one? Or..

http://j.mp/chrome-begin-frame

38 of 53

BeginFrame Scheduling on Aura

Browser also has a cc for UI and Über compositing

Browser’s cc’s BeginFrameSource can act as BeginFrame message generator for child (renderer) compositor

http://j.mp/chrome-begin-frame

39 of 53

BeginFrame Scheduling on Aura

Browser cc’s SyntheticBeginFrameSource generates BeginFrame message

vsync info from gpu process
default interval (16.6ms) if there is no gpu process

http://j.mp/chrome-begin-frame

40 of 53

BeginFrame Scheduling on Aura

SchedulerStateMachine::� BeginFrameRequiredForChildren()

ui::Compositor

maintain CompositorBeginFrameObserver list
Add/RemoveBeginFrameObserver

http://j.mp/chrome-begin-frame

41 of 53

BeginFrame Scheduling on Aura

In-progress implementation

content layer

CC

ui::Compositor

BeginFrameSource

Scheduler

RWHVAura

Renderer

Browser

Renderers

http://j.mp/chrome-begin-frame

42 of 53

BeginFrame Aligned Buffered Input handling on Aura

With BeginFrame scheduling, input handling can be aligned with BeginFrame message on renderer

Predictable event timing
Reduced event processing cost
Smooth and consistent visual output
See the design doc (by @jdduke) at http://goo.gl/MdmpCf for more details.

http://j.mp/chrome-begin-frame

43 of 53

TODOs for BeginFrame Scheduling

Still trying to land on Aura

https://crrev.com/1016033006/

Busy main thread of browser process
OOPIF

Mac support

http://j.mp/chrome-begin-frame

We are still trying to land this on aura. Landing and reverting is repeated.

Browser compositor doesn’t use threaded compositing mode.

So, if browser’s main thread is busy, BeginFrame message generation can be delayed.

Tt could make renderer janky.

OOPIF.

With our current implementation, we cannot easily support OOPIF.

The reason is BeginFrame message is generated from compositor’s scheduler,

but, RenderWidgetHostViewChildFrame doesn’t have a platform window.

If it uses cc surface, we can solve it.

Tim will say a little bit about DisplaySurface in the next section.

(As you saw, Current implementation sends BeginFrame via RenderWidgetHostViewAura

because each RenderWidgetHostViewAura has platform window.

So, we can easily makes it as a ui compositor’s observer.

But, RenderWidgetHostViewChildFrame doesn’t have platform window.

Android also has same issue. So, for the workaround, iframe’s renderers rely on SyntheticBeginFrameSource.)

Mac support. We will start after aura platform finished.

Maybe, we can implement BeginFrame scheduling on Mac similar to android

because mac provides vsync callback registration by the CVDisplayLink.

44 of 53

The Future

What is next?

http://j.mp/chrome-begin-frame

45 of 53

Unified Begin Frame enables new things

http://j.mp/chrome-begin-frame

46 of 53

Unified Begin Frame enables new things

"things we have wanted for a long time but have been too hard"

http://j.mp/chrome-begin-frame

47 of 53

Filtering / Improving VSync Signal

VSync information reported from OS has noise!

(Some platforms are better than others.)

Now have one place to fix it!

http://j.mp/chrome-begin-frame

48 of 53

Output Time / "Glass Time"

It takes time to do work

Gave a talk at BlinkOn 2 in Zurich

Rendering for “Glass Time”

http://goo.gl/MHGWTc

http://j.mp/chrome-begin-frame

49 of 53

Surfaces Display Scheduler - WIP

http://bit.ly/chrome-surfaces

"Surfaces are a concept to allow graphical embedding of heterogeneous untrusting clients efficiently into one scene."�

“Please render me stuff”

http://j.mp/chrome-begin-frame

50 of 53

Surfaces Display Scheduler - WIP

Global Browser Level Scheduler

Global knowledge about focus and content

Enables more things!

Latency recovery
Partial Updates
more…...

http://j.mp/chrome-begin-frame

51 of 53

Unified�BeginFrame

VSync Aligned Input

Glass Time

Output Time

Surfaces Display Scheduler

Partial Updates

Blink Scheduler

Jank Free Animations

Jank Free Input

Bad Actor

Sin Bin

Latency Recovery

Better Performance

Less

Memory Usage

?????

Motion Vectors

?????

http://j.mp/chrome-begin-frame

52 of 53

Special thanks to

Brian C. Anderson <brianderson@chromium.org>

Jared Duke <jdduke@chromium.org>

Sami Kyöstilä <skyostil@chromium.org>

and many more!

http://j.mp/chrome-begin-frame

53 of 53

Q&A

scheduler-dev@chromium.org

http://j.mp/chrome-begin-frame