1 of 28

History of the World�of Chrome Graphics

PART I

A super biased, compositor-centric

retrospective from enne@

2 of 28

Goals of this talk

Survey of graphics features across time
Understand Chesterton’s fences
Learn important Chrome GPU facts like “ugh gosh why is everything named Impl” and “why are we stuck with 256x256 tiles??”

3 of 28

Ancient History (2008)

No GPU process: only browser and renderer processes.

No compositor. No shaders. No command buffer.

Everything software rastered on demand.

(Windows only)

4 of 28

Historical diagram outlining how ancient Chromites got content on screen

RenderWidget’s

PaintAggregator

(1) Invalidate

(2) DoDeferredUpdate

(3) Aggregate invalidations

(4) Raster aggregate areas via Blink paint

To bitmaps

(5) Send bitmaps to Browser via transport dibs

WebKit

Renderer process

Browser process

HWND

Shared memory

5 of 28

Accelerated Compositing (2009)

3D CSS is the first trigger to go into “accelerated compositing” mode.

Once any renderer content is drawn on screen with the GPU then suddenly:

All renderer content needs to be drawn with GPU
Overlapping content needs to be detected (thus RenderLayerCompositor)
OpenGL needs to be executed safely (thus GPU process)
OpenGL needs to get to GPU process (thus command buffer)
OpenGL is slow on Windows (hence ANGLE)

accelerated compositing = using the gpu for compositing

Now that we have basic graphics, what are people clamoring for? 3D CSS of course.

In order to composite web content, WebKit had GraphicsLayer, originally added by Apple folks to represent some 2d piece of composited content.

The compositor at the time (aka LayerRendererChromium) had a few kinds of layers (content, image, software video).

The ContentLayerChromium just had one big texture. Invalidations would be unioned, WebKit would raster into a bitmap. That bitmap would get texSubImage2d’d into the larger texture. Then LayerRendererChromium would stick that layer on screen. If the texture was too big, it’d go into a “cropped” texture mode and would just maintain a texture for the visible rect.

LayerRendererChromium had some logic for scrolling, and could shift and blit by copying out of the backbuffer and then only updating part of the screen.

-- raw notes --

2009-01, 2009-02: 3d css transforms are the first 3d content

2009-07-16: poster circle

https://trac.webkit.org/changeset/45980

3d css transforms turn "promoted" render layer subtrees into a GraphicsLayer

Once promoted content exists, then overlapping content gets promoted

Apple folks wrote RenderLayerCompositor

as compared to LayerRendererChromium, the compositor

RenderLayerCompositor figures out which RenderLayer

now PaintLayerCompositor, once DeprecatedPaintLayerCompositor

Chrome originally had LayerChromium, each having one texture per layer

2010-08-17: Vangelis refactored into having layer subclasses

content, image, (software) video layers

https://bugs.webkit.org/show_bug.cgi?id=44148

ContentLayerChromium just has one big texture

Paint goes into a 2d bitmap, bitmap is texSubImage2d'd

If layer too big, then maintain a cropped bitmap for the visible rect.

"gl renderer" (LayerRendererChromium):

https://trac.webkit.org/browser/branches/chromium/597/WebCore/platform/graphics/chromium/LayerRendererChromium.cpp

to draw the screen, it would bind one big viewport texture

then draw textures into it

6 of 28

GPU process mode switch

GPU process spin-up was optional and on demand.

Browser would create an HWND for the accelerated content that was a child of the non-accelerated content so that the two modes could coexist.

Entire accelerated path could turn off whenever, even mid-paint.

I think an important thing to remember here is that the GPU process was optional and was spun up on demand.

If a page had accelerated content, then the GPU process would get spun up. Graphics layers would get created. The entire page would get rerastered in the accelerated path. Everything would get drawn to a texture. Then the video stops playing or whatever, everything gets torn down. Everything then would get rerastered in the legacy software path from a few slides ago. This was less than optimal.

Hilariously, accelerated compositing could turn off even mid paint. So imagine you’re in the call stack of a paint, deep inside cc’s layer tree, doing updates, and it calls back to WebKit to paint some layer, which then figures out it doesn’t need to be composited after all due to a lazy state update. Then, WebKit realizes that the whole page doesn’t need to be in accelerated compositing mode, so it destroys all the layer trees. Except these are the layer trees that you’re in the middle of painting.

-- raw notes --

In order to do any gl, we added a command buffer

Command buffer lets renderer do gl, but proxies commands to gpu process

The gpu process was for better or for worse optional.

Accelerated content would turn it on, and then would turn off after that

It's possible that accelerated compositing could turn off mid paint

https://trac.webkit.org/browser/branches/chromium/648/Source/WebCore/platform/graphics/chromium/LayerTilerChromium.cpp?rev=81031

Gpu process would start up and start down.

Leaving it on all the time regressed startup times, oops.

OpenGL terrible on Windows, so we added ANGLE to do gl->directx.

To get stuff on screen, gpu process was handed an ImageTransportSurface.

On windows, this was a child hwnd of the software hwnd of the top level one

Other ui had its own windows (omnibox too?).

7 of 28

Other accelerated compositing reasons

WebGL (2009)

Expose OpenGL to Javascript.

Swiftshader for blacklisted cards.

o3d successor.

Video (2010)

Previous video decode all software.

Pink video bug: https://crbug.com/57741

Canvas (2010)

Implementation initially in Blink.

Eventually switched to Skia and Ganesh.

Small canvas optimizations.

3D css and poster circle weren’t the only reasons for accelerated compositing. Other common triggers were these three culprits.

WebGL was a successor to o3d. o3d, as an aside, was an approach to bring high level graphics concepts to the web, and did it all in an NPAPI extension with ActiveX etc. Think high level graphics at maybe a little lower level than Unity. (I mention Unity mostly just as an example of a game engine here.) Except imagine trying to get other browser vendors to agree to ship something like Unity. Imagine furthermore to get other browser vendors to agree on what Unity should be before it even exists. It just didn’t get much traction, but WebGL was something everybody could agree on.

The media team added hardware decoded video after that, leading to the best bug ever.

Finally, there was accelerated canvas. Previously canvas would go into a bitmap, but if we can use the gpu for canvas commands, it can (often) be faster. I was surprised to find out that there was an initial “legacy” implementation of canvas in Blink, that had its own shaders and its own loop blinn path rendering optimizations. Eventually this got switched over to use Ganesh.

All of this meant that the browser had an accelerated compositing mode, but it was highly content dependent. So early days here meant that a lot of compositor bugs were found on youtube and google maps.

-- raw notes --

(1) WebGL

2009-08: initial commit

https://bugs.webkit.org/show_bug.cgi?id=28018

Exposes opengl to javascript

Commands proxied via command buffer to gpu process

Eventually swiftshader to handle blacklisted cards.

Maybe talk about o3d here

(2) Accelerated video

2010-07: vrk initial implementation, added video layer

2011-02: pink video bug https://crbug.com/57741

(3) Accelerated canvas

2010-07 jamesr initial patch

2010-07: jamesr initial patch https://bugs.webkit.org/show_bug.cgi?id=43094

Initial canvas implementation was roughly ganesh-in-Blink

Not actually Ganesh, but opengl directly there.

Had accelerated path support (loop-blinn).

(We are always working on accelerated path rendering)

Eventually switched to using skia and Ganesh here.

Deleted the initial/legacy implementation.

Canvas pricy, so eventually heuristics to not accelerate small canvaes

2d canvas enabled by default "with skia" https://bugs.webkit.org/show_bug.cgi?id=59929 (junov 2011-05)

delete the "legacy" unused canvas 2d path https://bugs.webkit.org/show_bug.cgi?id=64214

accelerated canvas min size 256x256 https://bugs.webkit.org/show_bug.cgi?id=80451 (2012-03)

8 of 28

Scrolling is slow (part 1, tiling)

Problem: Scrolling the root layer used shift and blit. Lots of raster work, even when nothing changed but scroll offset.

Solution: tile the root layer to re-use raster work.

Why 256x256 tiles? http://bugs.webkit.org/show_bug.cgi?id=49947#c6

Scrolling the root layer used shift and blit in gl. Other layers had no accelerated path. Fixed position elements just invalidated everything they moved past.

Initial method to fix this was to tile the root layer with 256x256 tiles.

Why 256x256? I measured a massive number (3) pages and checked how long it took to raster into a backbuffer and then copy into various tile sizes (and also to raster directly into those tiles). This peer reviewed comprehensive study is obviously why we still use 256x256 tiles today, even though there are no different factors like gpu raster, io surface allocation needs, or even changes in invalidation patterns over time.

-- raw notes --

Updating layers is slow.

2010-12: tile the root layer with 256x256 tiles

http://bugs.webkit.org/show_bug.cgi?id=49947#c6 256x256 decision

Non-root and image layers got tiled shortly after.

This helped update but not really

9 of 28

Scrolling is slow (part 2, threading, 2011)

Problem: Scrolling is still slow. Problem viewed as WebKit main thread jank.

Solution: Add a more responsive thread to handle input.

Tiling the root layer clearly wasn’t enough. There was a growing concern that the WebKit main thread was kind of unreliable, and between garbage collection, random user JavaScript, and raster work, that basically the main thread would go out to lunch all the time.

In order to fix input performance and have good scrolling, we added a second thread to handle input that would be able to scroll and do some basic animations.

To do this we added a parallel Layer tree. The WebKit side of the graphics layer tree would then “commit” to the main thread atomically while a lock was held.

Since we have a parallel layer tree that does different things, we have to name it something, and hence “LayerImpl” tree.

--raw notes--

Mid-late 2011

Fix input performance issues in scrolling by adding a thread.

Thread takes input, maybe handles it, can draw/animate frames independently.

Has a parallel layer tree. WebKit side "commits" to it atomically.

Layers commit to their "LayerImpl" implementation.

10 of 28

Infamous “impl” suffix, aka blame nduca@

See: https://bugs.webkit.org/show_bug.cgi?id=55013#c5

James Robinson 2011-02-24 11:50:23 PST

(In reply to comment #4)��> Is there any reason you call the new class CCLayerImpl instead of plain CCLayer ?�> ��Nat pointed out that it makes more sense to name the interface we exposed to the rest of WebKit CCLayer and have CCLayerImpl be the compositor's internal implementation. IOW GraphicsLayerChromium will have a set of CCLayers and set properties on them, etc, and only the compositor internals will ever have knowledge of CCLayerImpl.�

In retrospect, this sort of makes sense. You have a one set of layer classes as the public implementation of the compositor, and then a separate set of “implementation” classes that do the meat of the work. At least in a theroretical abstract sense, this made some sense.

But then people added LayerTreeHostImpl as the “implementation” of LayerTreeHost. And then people started calling it the “impl” thread, because that’s where all the “impl” classes got used. And then people called compositor thread rasterization “impl-side painting”, and it was pretty much all over by that point.

Now everybody thinks that folks on the graphics team can’t actually name anything and don’t understand what Impl classes are supposed to be.

--raw notes--

Infamous "impl" suffix: https://bugs.webkit.org/show_bug.cgi?id=55013#c5

> Is there any reason you call the new class CCLayerImpl instead of

> plain CCLayer ?

Nat pointed out that it makes more sense to name the interface we

exposed to the rest of WebKit CCLayer and have CCLayerImpl be the

compositor's internal implementation. IOW GraphicsLayerChromium

will have a set of CCLayers and set properties on them, etc, and

only the compositor internals will ever have knowledge of

CCLayerImpl.

cclayerimpl added in https://bugs.webkit.org/show_bug.cgi?id=55013

11 of 28

The awkward dance of early compositor thread implementations

GPU context is on the compositor thread. Main thread has the bitmaps to raster into. So, the only bit of code that had both the context and the bitmaps was the upload. Needless to say this was kind of slow, and sent David Reveman off to figure out better texture upload paths, which led to zero copy and one copy.

Image decode, scaling, filtering, caching all happens on the main thread here. Blink maintains this cache.

This diagram: https://docs.google.com/presentation/d/1nPEC4YRz-V1m_TsGB0pK3mZMRMVvHD1JXsHGr8I3Hvc/edit#slide=id.gb5d9bbc1_020

Other commit diagrams: https://docs.google.com/presentation/d/1_pgxfPHp4hKAn-BmhWdiqh_hfwbmQ3nWpiJ_Sj1EUk0/edit#slide=id.g1e91c55b_0_652

Texture upload talk: https://docs.google.com/presentation/d/1OS4jLvB3ntLqfqwWOdM9HOlJ1LxgB-l8_BS3T3F5T10/edit

--raw notes--

Rasterization

Still paint into one unioned bitmap, uploads copied out of that.

Bitmaps now uploaded during the commit (bad if uploads slow).

Tons of random uploader classes depending on flags and modes.

Draw bitmaps on compositor thread.

Easy to scroll past the end of content, but those checkers were SUPER SMOOTH.

Images (and filter results) all cached in Blink.

Painting an image would cause it to be cached, so no prepainting.

12 of 28

Ancient scheduler code that somehow fits entirely on one slide

CCSchedulerStateMachine::Action CCSchedulerStateMachine::nextAction(bool insideVSyncTick) const

{

switch (m_commitState) {

case COMMIT_STATE_IDLE:

if (m_needsRedraw && insideVSyncTick)

return ACTION_DRAW;

if (m_needsCommit)

return ACTION_BEGIN_FRAME;

return ACTION_NONE;

case COMMIT_STATE_FRAME_IN_PROGRESS:

if (m_needsRedraw && insideVSyncTick)

return ACTION_DRAW;

return ACTION_NONE;

case COMMIT_STATE_UPDATING_RESOURCES:

if (m_needsRedraw && insideVSyncTick)

return ACTION_DRAW;

if (!m_updatedThisFrame)

return ACTION_UPDATE_MORE_RESOURCES;

return ACTION_NONE;

case COMMIT_STATE_READY_TO_COMMIT:

return ACTION_COMMIT;

}

void CCSchedulerStateMachine::updateState(Action action)

{

switch (action) {

case ACTION_NONE:

return;

case ACTION_BEGIN_FRAME:

m_commitState = COMMIT_STATE_FRAME_IN_PROGRESS;

m_needsCommit = false;

return;

case ACTION_UPDATE_MORE_RESOURCES:

m_updatedThisFrame = true;

return;

case ACTION_COMMIT:

m_commitState = COMMIT_STATE_IDLE;

m_needsRedraw = true;

return;

case ACTION_DRAW:

m_updatedThisFrame = false;

m_needsRedraw = false;

return;

}

13 of 28

Compositor thread timeline (2011-2015)

2013

2012

2011

2014

2015

Finally shipped on Mac

Try to ship on Mac

Ship on Windows

Browser compositor briefly is threaded

Make browser never threaded

Start! 🏁

Scrollbars update on thread

Universal accelerated scrolling

Start work on non-root accelerated scrolling

Timelines are not the most interesting thing, but am pulling this out because I think that this is pretty demonstrative of most of the graphics features that get shipped.

�Initial implementation is pretty easy (one quarter, maybe two). There’s some follow on work, but then shipping on different platforms takes about an entire year to ship. Lots of surprises. Then finally to clean up the last bits take another year or so.

We see this with compositor thread. This was true for forced compositing mode shipping. This was true for shipping impl-side painting. This was true for ganesh rasterization too.

I guess the conclusion here is that the complexity of Chrome and differences between platforms (both performance and correctness) make things hard to roll out globally.

-- raw notes --

Mid-late 2011 started

mid 2012, scrollbars scrolling on compositor thread

oct 2013 vollick adds accelerated scrolling

other layers can scroll, although stuff is complicated

https://codereview.chromium.org/26809004

show demo of fixed position hell case

Rollout

took a long time to turn on, slow rollout

issues with mac and special mac input (jan 2013 ccameron)

may 2013 finally turned on for mac

almost always on for windows (aug 2013? tied to fcm?)

Browser compositor

early 2013 backer adds threaded compositing to the browser compositor

december 2013 danakj says only on ChromeOS

feb 2015 weiliangc turns it off https://codereview.chromium.org/781163003

14 of 28

Quads (early 2012)

Problem: Drawing code littered through layer classes. Hard to add new drawing modes, like software raster.

Solution: Draw into an abstraction (quads and shared quad states).

Surprisingly, cc::DrawQuads were not there to support ubercompositor.

Each layer class knew how to draw itself using OpenGL. This was pretty intrusive and spread out shader code and gl knowledge everywhere. To sort of refactor this out, quads were introduced. SharedQuadState is just a way to save on shared data for quads all coming from the same layer. It’s probably premature optimization.

Once draw quads were introduced, then a software renderer was added shortly afterwards. This just sort of lucked out to be a convenient way for ubercompositor to send compositor frame data later on.

-- raw notes --

december 2011 - feb 2012 (post threaded, pre ubercompositor)

Didn't want gl logic littered through lots of layer classes.

Moved to LayerRendererChromium (future GLRenderer)

https://bugs.webkit.org/show_bug.cgi?id=73059

Mostly refactoring to make an abstraction, not a fundamental transport unit

This helps enable software compositor by making a second renderer

(instead of injecting software logic in every layer class)

june 2012: software compositor

aelias https://bugs.webkit.org/show_bug.cgi?id=87920

15 of 28

Feature grab bag interlude~ 🎶

Edge antialiasing (2011): extend quads in screen space for soft edges

Occlusion tracker (2012): only at draw time, opaqueness hard to detect

Accelerated Filters (2012): use Ganesh in cc::GLRenderer for filter chains

A few fun side features to briefly mention here.

Edge antialiasing added to add soft edges. Quad vertices are extended in screen space by one pixel (and then clamped to a 1 pixel bounding box to fix corners). This extra geometry allows fragment shader samples in the extended geometry to create a feathered edge. This method is a way to not require MSAA or super sampling or anything more complicated while still making edges appear soft. It also doesn’t look good when two layers abutt each other. It’s not “correct” but it’s fast and looks good.

Occlusion tracker for drawing was added. There was a point where there was occlusion tracking of content from Blink. However, we never did a good job figuring out which layers were opaque, and when there were a lot of layers in the renderer, it was a lot of work for nothing. nduca@ once claimed that one of the hardest problems in computer science was figuring out if a particular subrectangle of a page was opaque or not. That code eventually was ripped out of Blink and now the only occlusion code is at draw time in the browser (and eventually in the future, the display) compositor.

We also got accelerated filters. A lot of work was done to pull out FilterOperations into a serializable safe struct. These filter operations then work on composited content (like a blur on a video or something) at draw time. Since Skia already was able to handle all of these filter types, the GLRenderer leans on Ganesh to do this work (although there’s the typical GL overhead to let somebody else use your context briefly and then have to restore state afterwards).

-- raw notes--

* edge antialiasing (may 2011)

Proposal: https://docs.google.com/document/d/185WI0XVS5Swvrd3n3Fhukvi_6gxNdIZqVwqO-1DIK_A/edit

Corner cases: https://docs.google.com/document/d/1g-rUtp8X-ECkMpMqy8oSDWkk_VU8bKCmVxYCt0HPOrM/edit#

Alternate thoughts: https://docs.google.com/document/d/1iQn-VmwlZ4Up98FpL1ODThX1Ogs8T2iFJ0t6r1m2wzA/edit

* occlusion tracker for drawing (march 2012)

* accelerated filters (oct 2012)

More ganesh in chrome, by using an offscreen, and later onscreen context.

GLRenderer calls into Ganesh to do this; work to make filters safe.

senorblanco/ajuma: https://chromiumcodereview.appspot.com/11175009

16 of 28

Übercompositor (2012-2013)

Problem: Renderer draws into an gl texture. Compositing that texture with the rest of the ui is an unneeded copy. ChromeOS devices often bandwidth bound.

Solution: Send quads to a browser compositor; composite “once”.

The output of the renderer in some cases is just going into a gl texture. Then the browser compositor would draw that to screen and copy it again, essentially doubling the blending bandwidth for all renderer content.

The solution here was to let the browser compositor do the final composite. Instead of taking a compositor frame full of draw quads and shared quad states and then turning those into GL commands, instead it would transport that compositor frame from the renderer to the browser. Resources were handled via texture mailboxes, which allowed for the same gpu resource to be shared between multiple processes with synchronization. This let the renderer create these resources and then the browser compositor use them to draw.

Important concepts here are the DelegatedRendererLayer, which is a layer that embeds the contents of a renderer. It accepts a CompositorFrame and when the layer is asked to generate a frame itself, it inserts those quads and shared quad states into the parent frame.

There was also a concept of DelegatingRenderer which instead of drawing to screen like GLRenderer or SoftwareRenderer, would forward along the CompositorFrame to the browser through the (at the time) OutputSurface.

At the moment, all compositor instances (browser and renderer) use “DelegatingRenderer” which has been folded into LayerTreeHostImpl. All the “direct” renderers are called from the final display compositor.

-- raw notes --

https://docs.google.com/document/d/1ziMZtS5Hf8azogi2VjSE6XPaMwivZSyXAIIp0GgInNA/edit

September 2012(?)-October 2013

Why: remove copy, back ui/views with layers and a compositor (aura)

Aura (aura_wm / views_compositor)

aug 2011 (ben@), sky, oshima, sadrul

linux first

Backed views with ui::Layers which had-a cc::Layer

ui::Layers originally had a single ui::Texture on them

Used WebLayer: http://codereview.chromium.org/8222028

Used cc directly: https://chromiumcodereview.appspot.com/11415089

Mailboxes

TODO(enne): add some explanation here about mailboxes

mailboxes shipped first, replaced some ImageTransportSurface stuff

only shipped on Android (maybe Windows)

Ubercompositor replaced the need f

but ubercompositor replaced the need for this path

aside: jun 2013 copy output requests (previously everything read back to cpu into skbitmap) https://chromiumcodereview.appspot.com/17018002

DelegatedRendererLayer added in sep 2012

https://chromiumcodereview.appspot.com/10916307

http://crbug.com/123444

browser compositor receives frames

delegated renderer layer inserts those as a part of append quads

resources shipped with mailboxes / returned

child renderer has a delegating renderer, which sends compositor frames

parent renderer inserts them into the frame, has a direct renderer

Lots of discussion about what to send.

Originally talked about sending layers.

Layers and SkPictures deemed too complicated to send ehehehe

gpu process changes

in the past, browser would give a native child window to gpu process

jbauman: this was maybe a child of the of the sofwtare window?

views painted into a native window, maybe omnibox too

ubercomp hoisted everything up a level

browser still has a window but composites everything into that

ubercompositor shipped everywhere in oct 2013

17 of 28

Aura (2011-2013)

aka “views compositor”

One important thing to mention on the side here from the ui graphics perspective is Aura.

The ui::views system existed since the beginning of time. Used a lot of Windows-isms and used native Windows things to update widgets and ui on screen.

There was some desire to have a fancier system to support ChromeOS windows and compositing.

The idea was to back views with ui::Layers which then used cc::Layers underneath them so that the same compositing code could be used by both Blink and ui without having to be rewritten.

The initial thought I think was that we’d just move the compositor thread over to some other compositing process, but nobody was super happy at the time with the idea of shipping layer trees or recordings around, and it seemed a lot easier to validate compositor frames and quad data.

Diagram from: https://www.chromium.org/developers/design-documents/aura-desktop-window-manager

18 of 28

Project GTFO (2011-2013)

Context: remember that all of this work in the compositor is being done inside of third_party/WebKit/Source/WebCore/platform/graphics/chromium

Problems:�(1) Aura wants to back views with cc::Layers�(2) WebKit reviewer policy super stringent

Solution: move the compositor out of Blink via abstractions

That leads directly into project GTFO.

If Aura wants to use the compositor, and the compositor lives in WebKit, then we need to...link WebKit into the browser process? Sad trombone noise.

The other social problem here is that becoming a WebKit reviewer involved landing at least 80 patches at the bare minimum, and then having the support of three or four other reviewers, at least one of which was not from your company. Needless to say, there were very few reviewers and folks at different companies aren’t super excited to review your Chromium only patches.

So, the solution to all these problems was to move the compositor out of WebKit and into cc. Added an abstraction layer WebLayer to wrap cc::Layer, which obviously has been removed by now. :C

This made cc become the first directory to enable git cl formatting by default, c++ linting on by default, and also converted from WebKit style to Chromium style mostly by hand (urgh). No egrets.

-- raw notes --

Oct 2011-August 2012 (maybe March 2013)

Oct 2011 first weblayer commit

https://bugs.webkit.org/show_bug.cgi?id=69107

grew WebLayer and friends that we still haven't removed

WebLayer exists so that the browser could back views with cc::Layers.

Once WebLayer existed, then the browser just linked to Blink.

This is suboptimal, so cc moved out of blink repo.

Also, WebKit reviewer requirements super rough.

August 2012 initial cc commits

/third_party/WebKit/Source/WebCore/platform/graphics/chromium -> /cc

Late 2012 filenames changed

March 2013 style all changed

first git cl format presubmit (also one of the only c++ linters)

19 of 28

Force compositing mode (2012-2014)

Compositor still has three modes:

Legacy software (remember that diagram, many slides ago? Still there!)
Hardware compositing (using gl for final draw)
Software compositing (still have layers, but using SoftwareRenderer)

This sounds simple, but took a surprising amount of calendar time to roll out.

With all of that going on, legacy software path STILL exists.

Lots of complication in Blink and content/ about switching modes. Other GPU features blocked on it (threaded compositing, impl-side painting).

Turning this on also flushed out remaining gpu issues. Previously only certain pages triggered the gpu, but now every single page does. So, lots of bugs too.

Even worse, depending on the GPU process in startup regressed startup times. What magic fixed that regression? Apologies, mostly.

-- raw notes --

june 2012-july 2014

Compositor had three modes: legacy software, software compositing, hardware compositing

Needed to convert legacy path to software compositing as an ubercomp prereq

Force compositing just meant always accelerate compositing without triggers

Android always used it (although also had software modes)

field trial jun 2012

ccameron may 2013 mac on

gab aug 2013 windows on

gab sep 19 tied to threaded compositing everywhere

danakj april 2014 https://codereview.chromium.org/240183005 all but layout tests

enne june/july 2014 layout tests finally converted

20 of 28

Impl-side painting (2012-2015)

Problem: Scrolling is now smooth, but quickly run out of prepainted content. Scrolling checkerboards somehow not super exciting to users.

Solution: Move rasterization from main thread to the compositor thread

Additional problems: image decodes, assumptions about scales at record time

Prepainting was really hard in the threaded compositing world, because preraster work had to be done on the main thread, which only increased the critical path to get things on screen. (And how do you know what to prepaint at all?)

Android’s second release was coming up (m25 refresh from m18?), and this was especially bad there because commits were really long. So, motivated by Android, we embarked on the “impl-side painting” project.

Yes yes, impl-side painting is about rasterization and not about painting.

The first issue this had to solve that doesn’t get talked about a lot is image decodes. Previously images were all decoded at paint time and cached in Blink. However, now the main thread paints a very large prepaint region (all the content that could be rastered on the compositor thread) and doesn’t want to decode every single image that it paints. So now, recorded images are backed by “lazy decoding” pixel refs that will get decoded later at raster time.

21 of 28

Impl-side painting (2012-2015)

If rasterization only happens on the compositor thread, and we don’t want (slow) raster work to block scrolling. Therefore, impl-side painting introduced the idea of pending trees (to stage the raster work from a compositor frame) and an active tree (the tree that’s currently being scrolled and display). Once the pending tree finished its raster work and uploaded all the resources, then it activates and the pending tree becomes the active tree.

We could probably make pending trees, active trees, commit and activation all a lot cheaper, but it’s really the unit of “atomic update” in the compositor. Single threaded compositors (in the browser) that don’t have scrolling commit directly to the active tree for performance reasons.

Pending layers these days only contain the tiles that are different from the active tree. I could imagine some world in the future where we similarly only have layers in the pending tree that have updates, and then updates should be much more efficient and proportional to the amount of update work.

This shipped for Android first, and then took a lot of work to ship for other platforms, largely because of images. Android uses low quality (aka bilinear) scaling, none of which get cached. So therefore the only caching needed was a decode cache. Desktop platforms expect high quality scaling, which takes a lot of memory. So Blink caching of scaled images needed to be removed. Now that we’re decoding at raster time, now Skia needs to cache these. Wait, that’s too much memory, so now there’s a discardable memory system to decode into. Eventually image decoding moved out of Skia and into compositor control so images could be decoded and scaled before rastering.

-- raw notes --

Diagram from: https://docs.google.com/presentation/d/1nPEC4YRz-V1m_TsGB0pK3mZMRMVvHD1JXsHGr8I3Hvc/edit#slide=id.gb9272b82_22

November 2012-Nov 2013 (alternatively June 2015)

Why: scrolling on compositor thread checkerboard too much

also mostly for (first?) major Android upgrade

Move raster from main thread to compositor thread

ImageDecoding

hclam deferred image decoding oct 2012

images now decoded at raster instead of at record

https://bugs.webkit.org/show_bug.cgi?id=94240

ships on Android first (images easy, high quality still in Blink)

image caching left in skia, as no easy way to figure out where images were

but couldn't give skia a cache size the same as blink

so piped in discardable there

turned on for android aug jan 2013

always on for android nov 2013

april 2014

impl-side painting on by default on all platforms where threaded is on

https://codereview.chromium.org/242803007

then disabled on mac and windows by june 2014 :P

windows july 2014, just kidding august 2014

mac august 2014 (scrollbar issues)

ui compositor is mid 2014

unit tests oct 2014

single threaded oct 2014

layout tests late 2014

about:flags removed on january 2015

finally old code removed in june 2015

http://crbug.com/155209

stub classes in november 2012

https://chromiumcodereview.appspot.com/11265046

picturelayertilingset nov 2012 https://chromiumcodereview.appspot.com/11417111

picture pile jan 2013

22 of 28

Layer Squashing (2013-2014)

Problem: Layers are expensive. Many sites create too many layers!

Solution: Squash more PaintLayers together into a single composited layer.

1

2

3

A

Animated

1

2

3

root

R

PaintLayer tree

before squashing

A

R

PaintLayer tree

after squashing

2D diagram on the left on the page

The root layer and the animated layer have direct reasons for compositing.

Blue squares here indicate requiring a composited layer backing (aka GraphicsLayer / WebLayer / cc::Layer)

In the absence of the animated layer, 1+2+3 would all just be part of the root layer’s composited layer

The animated layer means that 1+2+3 can’t share the same backing as the root layer and previously would create a “layer explosion” where each PaintLayer gots its own composited layer backing.

Squashing lets 1+2+3 all share the same composited layer and “squash” together.

Difficulty of implementation here (top Blink engineers for several quarters) because layerization had to happen pre-painting (and take web painting algorithm into account while doing so) is one of the reasons for the slimming paint v2, which will do painting first and then layerize afterwards.

--raw notes--

shawnsingh late 2013

abarth,chrishtr,vollick early 2014

layer squashing turned on april 2014 bumpy landing

23 of 28

Surfaces (2014)

Browser compositor

Renderer compositor

OOPIF compositor

Display compositor

Renderer compositor

OOPIF compositor

Browser compositor

Problem: delegated rendering layer pipeline too deep. Renderers create a CompositorFrame of quads and send to DelegatingRenderer of their parent. Unfortunately, this is a lot of work to shuttle the same data up into every compositor. There’s also more latency for updates. It also means that every single renderer update means the browser compositor also needs to update even if nothing has changed there.

�Solution: create surfaces. Surfaces are a system to refer to the Compositor Frame output of another compositor. Surface ids is a reference to embedded contents from another compositor, but the id persists across multiple compositor frames and change when size or device scale factor changes. So the browser could commit one frame with id=3, and then the renderer can now submit any number of frames. Surface aggregator stitches all these compositor frames together and the display compositor does the final drawing of the compositor frame to screen.

Still lots of work in aggregating surfaces every frame, but maybe somebody smart can make that more efficient.

-- raw notes--

surface aggregator jan/feb 2014 (jamesr)

overlay support mar 2014

alexst https://codereview.chromium.org/197223003

cc::Display may 2014

jbauman https://codereview.chromium.org/302903003

jbauman lands surfaces https://codereview.chromium.org/756453003

2014-12-09: six tries later it lands <_<

24 of 28

Ganesh Rasterization (2013-2017?)

Ganesh has been used for a long time in Chrome for various things:

accelerated canvas
composited filters
main thread ganesh rasterization, pre compositor thread (!!!)

Lots of desire to use ganesh instead of software for rasterization, especially given that it’s faster and more power efficient. Also the case that many Android devices have giant screens and slow cpus and reasonable gpus, and so ganesh is a much better fit.

However, fitting into the compositor architecture has always been a little bit tricky.

In the past we did experiment with it briefly prior to threaded compositing times with a special texture uploader for ganesh. I forget why this never went anywhere, but I think our 3d context tech was not quite up to par to support doing this from the main thread. I think also our major issues at the time were wanting to make scrolling more smooth.

-- raw notes --

Used for canvas since ancient times.

Used for ganesh filters in the compositor as well.

Ganesh rasterization is do opengl using ganesh in compositor instead of software rasterization to improve raster performance.

Experimented with direct-to-backbuffer path in december 2013

Quads make SkPictures, handed to compositor.

https://codereview.chromium.org/98463008

Never really used and then removed.

25 of 28

Synchronous Ganesh Raster (v1)

Impl Thread

Commit

PrepareTiles

Raster

NotifyReadyToActivate

Activate

BeginImplFrameDeadline

DrawAndSwap

Renderer Thread

BeginImplFrame

BeginMainFrame

Commit

diagram by vmiura@

26 of 28

Threaded Ganesh Raster (v2)

Worker Thread

Impl Thread

Commit

PrepareTiles

NotifyReadyToActivate

Activate

BeginImplFrameDeadline

DrawAndSwap

BeginImplFrame

Draw

TaskGraphRunner

RasterTask

...

diagram by vmiura@

Synchronous approach didn’t really cut it at the time, and so we went to a threaded gpu for ganesh. Ganesh gets its own raster worker. Image decodes happen in parallel on other threads. Unfortunately, the original version of image decode was just to “warm up” the cache and hope that by the time the raster work got around to happening that it was still decoded and the cache wasn’t thrashed. Image decodes are a lot smarter now.

Everything is awesome and with a ridiculous amount of diligent testing from ericrk, is slowly rolling out to every platform this year. /o\

-- raw notes --

Diagram shamelessly stolen from vmiura’s https://docs.google.com/document/d/1aRms3eSMHNYU1VQXKun6iiwifSTw0gLmHyDLNlTU85U/edit#

Threaded Ganesh Raster (vmiura)

Eventually we gave up on the synchronous approach.

Put ganesh on its own raster worker.

Added better image decode service to decode images in parallel.

Long time to roll out

on for android in ?

on for windows in 2016 Q4

on for mac in 2017 Q1

on for chromeos in maybe 2017 Q2

on for linux in ???

Skia had Ganesh in 2012, so it took us until 2017 to ship everywhere. OOPS.

1024x1024 tiles to start https://codereview.chromium.org/112263002 (dec 2013)

oops large tiles failed https://codereview.chromium.org/141633006 (jan 2014)

viewport wide tiles https://codereview.chromium.org/247973005 (apr 2014)

per layer gpu rasterization veto https://codereview.chromium.org/222903005 (apr 2014)

per page gpu veto https://codereview.chromium.org/270823003 (may 2014)

(originally each tile would mark it or not, now we just reset the tile manager)

synchronous prepare to draw raster https://codereview.chromium.org/820743002 (jan 2015)

threaded gpu raster https://codereview.chromium.org/916723002 (feb 2015)

27 of 28

THE END

(for now)

Other things currently in flight:

MUS
slimming paint v2, v3, v4
color correctness
custom display lists
image checkering
WebGL next
cc::SkiaRenderer
Vulkan
VR
&c &c &c

28 of 28

Conclusions

We are always already working on accelerated path rendering.
Pick your numerical constants well the first time.
Major graphics features take a hella long time to ship everywhere.
Chrome graphics makes terrible cases ok and great cases...also just ok.
Everything starts out simple but is often complicated for a reason. :C

Path rendering: initial accelerated canvas approach had loop blinn. Skia and Ganesh had its own path rendering algorithm. senorblanco@ has been working on various path rendering optimizations for some time. We’ve discussed trying to ship nvidia path rendering for pretty much forever. This seems like good job security to be working on path rendering. I’m sure in like 2020 there’s going to be some new Vulkan path rendering hotness.

Numerical constants are hard. We’ve been stuck with 256x256 tiles for forever. It’s easy to pick a number in the first place. Like, HYPOTHETICALLY[0], you say that you write some code to tell the compositor how far away to paint, and you say “314 is a nice round number” (ba-dum-chh). Then you move that code around and accidentally change it to 312. And then you, like a diligent engineer, come back and say “hmm, maybe I should change that back” and some thoughtful reviewer goes “well, can you defend that change to me?” [1] And so the diligent engineer goes and does something else productive instead. The story of Chrome constants.

[0] https://codereview.chromium.org/271733003

[1] https://codereview.chromium.org/910733004/diff/20001/cc/resources/picture_layer_tiling.cc

Features in the graphics stack take a long time to ship. Chrome is complicated. Platforms are unique. Lots of benchmarks yell when you change anything. Even if the initial feature takes a balmy two quarters to write, it still takes another year or two to ship on all platforms and suss out all the bugs and quirks you never knew existed. This seems really sad to me, but maybe it’s just the nature of large project development?

Most of the compositor work over the past 8 years has been to mitigate the effect of slow Blink, slow javascript, slow decodes, slow raster and still try to be responsive. So, pages can be really bad in terms of content and we do a pretty good job at them. On the other hand, it means that the common case that should go super speedy also has to go through this same infrastructure, and is a lot pricier than it should be. We need to be a lot better here

Pretty much everything here described started off being really simple and then grew to be way more complicated than it had any right to be. I don’t think we’ve added anything without good reason, which means it’s hard to get away from a lot of the complexity that’s being carried.

See https://trac.webkit.org/browser/branches/chromium/597/WebCore/platform/graphics/chromium/LayerRendererChromium.cpp for a version of the first compositor. Pretty much all of cc is there in mini.

Finally, you should always agree to give historical talks because diagrams come pre-made.