LIFE OF A
Steve Kobes
skobes@chromium.org
Nov 2020
slides: bit.ly/lifeofapixel
with special thanks to past presenters
dknox@, chrishtr@, pdr@
"The unexamined pixel is not worth rendering."
— Socrates
LIFE OF A
88.0.4307.0
(Nov 2020)
LIFE OF A
content
pixels
rendering
content
NOT "content"
content::WebContents
sandboxed renderer process
Blink
cc
content
(Hyper-Text Markup Language)
<p> hello </p>
(Cascading Style Sheets)
p { color: red }
(JavaScript)
p.innerHTML = "goodbye";
(other kinds of rendered content: <video>, <canvas>, WebAssembly, WebGL, WebVR, PDF, ...)
<img src="kitten.gif">
content
https://www.nytimes.com
Operating System
pixels
Device Driver
#include <GL/gl.h>
( )
( )
( ?)
goals
initial render
update
update
update
time
stages
<html>...</html>
content
stage
stage
stage
?
?
pixels
lifecycle
intermediate data structures
parsing
HTMLDocumentParser
builds
Document
HTML
BODY
DIV
P
P
Text
Text
(these tags are optional)
(always the root)
HTMLTreeBuilder
<div>
<p> hello </p>
<p> world </p>
</div>
DOM
Document
HTML
BODY
DIV
P
P
Text
Text
This is the
Document
Object
Model.
The DOM
is a tree!
V8
DOM
Document
HTML
BODY
DIV
P
P
Text
Text
The DOM is both
var div = document.body.firstChild;
var p2 = div.childNodes[1];
p2.appendChild(document.createElement("span"));
p2.foo = {arbitrary: "state"};
[bindings]
[ JS ]
DOM
<my-widget>
<b>
<wrapper>
<slot>
A custom element has a shadow tree.
<my-widget>
<b>inside</b>
</my-widget>
customElements.define('my-widget',
class extends HTMLElement { constructor() { super();
this.attachShadow({mode: 'open'}).innerHTML =
"<wrapper><slot /></wrapper>"; }});
ShadowRoot()
ShadowRoot
AssignedNodes()
host tree
shadow tree
DOM
<my-widget>
<b>
<wrapper>
<slot>
ShadowRoot()
ShadowRoot
AssignedNodes()
host tree
shadow tree
A flat tree traversal presents a composed view.
FlatTreeTraversal
<my-widget>
<wrapper>
<b>
style
/* every <p> has red text */
p { color: red }
Document
HTML
BODY
DIV
P
P
Text
Text
selects
hello
world
style
hello
font-weight: bold;
margin-left: 2cm;
hello
outline: dashed blue;
hello
transform: rotate(20deg);
hello
background: url(kitten.jpg);
hello
style
/* every other <p> in any <div> without class="foo" */
div:not(.foo) > p:nth-of-type(2n) {
color: red !important;
}
p {
color: blue;
}
style
div:not(.foo) > p {
color: red !important;
}
p {
margin: 0;
border-left-color: blue;
...
}
...
CSSParser
StyleSheetContents
builds
StyleRule
CSSPropertyValueSet
CSSPropertyValue
Property()
Value()
BorderLeftColor
CSSColorValue
0x0000FFFF
CSSSelectorList
css_properties.json5
make_css_property_subclasses.py
border_left_color.cc
CSSSelector
style
Document
HTML
BODY
DIV
P
P
Text
Text
Document::UpdateStyle
ComputedStyle
ComputedStyle
ComputedStyle
ComputedStyle
ComputedStyle
ComputedStyle
ComputedStyle
ComputedStyle
ComputedStyle {
fontWeight = ...
marginLeft = ...
outline = ...
transform = ...
background = ...
...�}
Style recalc: for every node in the DOM tree, compute the value of every style property.
StyleResolver::StyleForElement
StyleSheetContents
StyleSheetContents
StyleSheetContents
style
getComputedStyle(element)["padding-top"]
[ JS ]
(But: getComputedStyle returns some layout data also.)
layout
DIV
LayoutRect
x = 183
y = 148
width = 402
height = 116
layout
block flow
Simple "block" layout objects are placed one after another, flowing down the page.
<span class="bigger">
layout
inline flow
Text and "inline" elements flow left-to-right, and are broken into lines.
(Some languages flow right-to-left.)
Lorem ipsum dolor sit amet,
consectetur adipiscing elit.
#text
line break
العربية
עִבְרִית
layout
Layout measures runs of text in fonts.
"fire"
HarfBuzzShaper
Font
ShapeResult
HarfBuzz
fire
Shaping selects the glyphs and
computes their placement.
layout
CSS
IS
AWESOME
border box rect
layout overflow rect
The contents of a layout object can overflow its border box.
Overflow can be visible, hidden, or scrollable.
layout
Other kinds of layout are even more complex.
LocalFrameView::UpdateLayout
layout
Document
HTML
BODY
DIV
P
P
Text
LayoutView
LayoutBlockFlow
LayoutBlockFlow
LayoutBlockFlow
LayoutBlockFlow
LayoutBlockFlow
LayoutText
DOM tree
layout tree
StyleEngine::RebuildLayoutTree
Each layout pass walks the layout tree, computing the visual geometry of each LayoutObject.
layout
DOM nodes are not 1:1 with layout objects.
<div>
ComputedStyle {
display: none
}
no LayoutObject!
LayoutInline SPAN
<div> block </div>
<span> inline </span>
LayoutBlock DIV
LayoutBlock (anonymous)
no DOM node!
<my-widget>
<div>
ShadowRoot
LayoutBlock MY-WIDGET
LayoutBlock DIV
The layout tree also crosses shadow roots.
LayoutObject
layout
LayoutNGMixin
LayoutObject
LayoutObject
LayoutNGMixin
The layout tree has legacy and NG layout objects.
“next generation” layout engine!
LayoutObject
layout
LayoutNGMixin
LayoutObject
LayoutObject
LayoutNGMixin
UpdateLayout()
🔍
🔍
🔍
🔍
A legacy layout object holds inputs, outputs, and algorithms of layout.
It can see the state of the whole tree.
immutable!
NGLayoutInputNode
NGLayoutInputNode
LayoutObject
layout
LayoutNGMixin
LayoutObject
LayoutObject
LayoutNGMixin
NGLayoutAlgorithm
NGConstraintSpace
✓
✗
UpdateLayout()
NGLayoutResult
LayoutNG separates the inputs and outputs of layout.
LayoutObject
layout
LayoutNGMixin
LayoutObject
LayoutObject
LayoutNGMixin
UpdateLayout()
NGLayoutResult
A layout result has a tree of fragments.
NGPhysicalFragment
NGPhysicalFragment
NGPhysicalFragment
layout — example
<div style="max-width: 100px">
<div style="float: left; padding: 1ex">F</div>
<br>The <b>quick brown</b> fox
<div style="margin: -60px 0 0 80px">jumps</div>
</div>
layout — example
<div style="max-width: 100px">
<div style="float: left; padding: 1ex">F</div>
<br>The <b>quick brown</b> fox
<div style="margin: -60px 0 0 80px">jumps</div>
</div>
layout — example
<div style="max-width: 100px">
<div style="float: left; padding: 1ex">F</div>
<br>The <b>quick brown</b> fox
<div style="margin: -60px 0 0 80px">jumps</div>
</div>
DOM tree
BR
"The "
B
"quick brown"
" fox"
DIV
"F"
DIV
DIV
"jumps"
layout — example
layout tree
LayoutBR
LayoutText "The "
LayoutInline {B}
LayoutText "quick brown"
LayoutText " fox"
LayoutNGBlockFlow {DIV}
LayoutText "F"
LayoutNGBlockFlow (floating)
LayoutNGBlockFlow {DIV}
LayoutText "jumps"
LayoutNGBlockFlow (anonymous)
<div style="max-width: 100px">
<div style="float: left; padding: 1ex">F</div>
<br>The <b>quick brown</b> fox
<div style="margin: -60px 0 0 80px">jumps</div>
</div>
layout — example
Box (block-flow) at 0,0 100x12
Box (block-flow children-inline) at 0,0 100x54
LineBox at 24.9,0 0x18
Box (floating block-flow children-inline) at 0,0 24.9x34
LineBox at 8,8 8.9x18
Text 'F' at 8,8 8.9x17
Text '\n' at 24.9,0 0x17
LineBox at 24.9,18 67.1x18
Text 'The ' at 24.9,18 28.9x17
Text 'quick' at 53.8,18 38.25x17
LineBox at 0,36 69.5x18
Text 'brown' at 0,36 44.2x17
Text ' fox' at 44.2,36 25.3x17
Box (block-flow children-inline) at 80,-6 20x18
LineBox at 0,0 39.125x18
Text 'jumps' at 0,0 39.125x17
fragment tree
layout — example
Box (block-flow) at 0,0 100x12
Box (block-flow children-inline) at 0,0 100x54
LineBox at 24.9,0 0x18
Box (floating block-flow children-inline) at 0,0 24.9x34
LineBox at 8,8 8.9x18
Text 'F' at 8,8 8.9x17
Text '\n' at 24.9,0 0x17
LineBox at 24.9,18 67.1x18
Text 'The ' at 24.9,18 28.9x17
Text 'quick' at 53.8,18 38.25x17
LineBox at 0,36 69.5x18
Text 'brown' at 0,36 44.2x17
Text ' fox' at 44.2,36 25.3x17
Box (block-flow children-inline) at 80,-6 20x18
LineBox at 0,0 39.125x18
Text 'jumps' at 0,0 39.125x17
fragment tree
Box (block-flow) at 0,0 100x12
Box (block-flow children-inline) at 0,0 100x54
LineBox at 24.9,0 0x18
Box (floating block-flow children-inline) at 0,0 24.9x34
LineBox at 8,8 8.9x18
Text 'F' at 8,8 8.9x17
Text '\n' at 24.9,0 0x17
LineBox at 24.9,18 67.1x18
Text 'The ' at 24.9,18 28.9x17
Text 'quick' at 53.8,18 38.25x17
LineBox at 0,36 69.5x18
Text 'brown' at 0,36 44.2x17
Text ' fox' at 44.2,36 25.3x17
Box (block-flow children-inline) at 80,-6 20x18
LineBox at 0,0 39.125x18
Text 'jumps' at 0,0 39.125x17
layout — example
fragment tree
PaintArtifact
paint
LocalFrameView::PaintTree
PaintCanvas::drawRect
DIV
LayoutObject
DrawRectOp
- rect {x, y, w, h}
- PaintFlags {color, …}
PO
PO
PO
…
paint op buffer
DisplayItem
display
item
list
Paint builds a list of paint ops.
Paint()
paint
<div class="yellow"></div>
<div class="green"></div>
<style>
.yellow { z-index: 2; ... }
.green { z-index: 1; ... }
</style>
DIV .yellow
DIV .green
BODY
DOM tree
yellow paints last
Paint uses stacking order,
not DOM order.
paint
<div id="green">
green's text
</div>
<div id="blue"></div>
blue after green, but
foregrounds after backgrounds
Each paint phase is a separate traversal of a stacking context.
backgrounds
floats
foregrounds
outlines
paint phases
(simplified)
paint — example
<style> #p {
position: absolute; padding: 2px;
width: 50px; height: 20px;
left: 25px; top: 25px;
border: 4px solid purple;
background-color: lightgrey;
} </style>
<div id=p> pixels </div>
paint — example
<style> #p {
position: absolute; padding: 2px;
width: 50px; height: 20px;
left: 25px; top: 25px;
border: 4px solid purple;
background-color: lightgrey;
} </style>
<div id=p> pixels </div>
paint — example
<style> #p {
position: absolute; padding: 2px;
width: 50px; height: 20px;
left: 25px; top: 25px;
border: 4px solid purple;
background-color: lightgrey;
} </style>
<div id=p> pixels </div>
DrawingDocumentBackground
DrawingBoxDecorationBackground
DrawingPaintPhaseForeground
DrawRectOp
{0, 0, 610, 316} kFill_Style
DrawRectOp
{25, 25, 87, 57} kFill_Style
DrawRectOp
{27, 27, 85, 55} kStroke_Style (w=4)
DrawTextBlobOp
{31, 45} "pixels"
display items
DrawingPaintPhaseForeground
paint — example
<style> #p {
position: absolute; padding: 2px;
width: 50px; height: 20px;
left: 25px; top: 25px;
border: 4px solid purple;
background-color: lightgrey;
} </style>
<div id=p> pixels </div>
DrawTextBlobOp
{31, 45}
SkTextBlob
53 4C 5B 48 4F 56
0.0 8.0 12.4 20.4 27.5 32.0
raster
PaintOp
…
cc::DisplayItemList
bitmap
Rasterization turns (part of) a display item list into a bitmap of color values.
FFFF
FFFF
FFFF
FFFF
FFFF
FFFF
0000
00FF
0000
00FF
0000
00FF
FFFF
00FF
FFFF
00FF
FFFF
00FF
(red, green,
blue, alpha)
cc::ResourcePool::InUsePoolResource
raster
raster
<img src="kitten.jpeg">
DrawImageOp
paint
PaintImage
raster
DecodedDrawImage
ImageDecodeCache
Raster includes image decoding.
InUsePoolResource
raster
GpuRasterBacking
GPU memory
GL texture ID
Raster can be accelerated by the GPU.
PaintOp
…
cc::DisplayItemList
raster
DrawRectOp
{rect, flags}
Raster()
SkCanvas::drawRect
GrRenderTargetContext::addDrawOp
GrOp
FillRectOp
…
SkSurface::flush
GrOp::execute
GrGLOpsRenderPass::onDraw
glDrawArrays(...)
Raster uses the Skia graphics library to issue GL calls.
another op list!
DrawRectOp::RasterWithFlags
GPU process
gpu
renderer process
Raster runs in the GPU process.
RasterInterface
RasterTaskImpl
RasterDecoderImpl
PaintOp::Raster
PaintOp
…
PaintOp
GPU process
gpu
renderer process
GpuChannelHost
The request is sent through a command buffer.
RasterInterface
GpuChannel
RasterCHROMIUM
PaintOp
…
PaintOp
RasterDecoderImpl
GPU command
buffer
GPU process
gpu
In the GPU process, the GL functions
link dynamically to native OpenGL.
On Windows, we translate to DirectX.
GLApi::glDrawArraysFn
dlopen("libGLESv2.so")
dlsym("glDrawArrays")
LoadLibrary("libglesv2.dll")
GetProcAddress("glDrawArrays")
ANGLE
change
DOM
style
layout
paint
raster
gpu
We now have a complete pipeline.
(in memory)
“Be the change you wish
to see in the pixels.”
—not Gandhi
But what if state can change?
frames
The renderer produces animation frames.
time
VSync signal
= time spent to render the frame
Below 60 frames per second,
scrolling and animations look "janky".
invalidation
Each pipeline stage tracks granular asynchronous invalidations.
Node::SetNeedsStyleRecalc()
style
LayoutObject::SetNeedsLayout()
PaintInvalidator::InvalidatePaint()
RasterInvalidator::Generate()
layout
paint
raster
Outputs are reused from previous frames when possible.
repaint
Paint + raster remain expensive if a large region is transformed…
scroll
all the pixels changed!
jank
… and anything on the main thread competes with JavaScript.
<script>
mineSomeBitcoins(); // why not?
</script>
time
main
enter: compositing
layers in Developer Tools
main
impl*
build layers
draw layers
commit
* ("impl" = compositor thread) ¯\_(ツ)_/¯
which raster independently.�
layers
<div>
AAA
<p class="wobble"> BBB </p>
</div>
CCC
BODY
DIV
#text
P
#text
#text
A simple layer captures
a subtree of content.
layer
layers
<div class="wobble">
AAA
<p> BBB </p>
</div>
CCC
BODY
DIV
#text
P
#text
#text
layer
A simple layer captures
a subtree of content.
compositing
Animation:
a layer moves
Pinch Zoom:
a layer scales
Scrolling:
a layer moves; another clips
threaded input
The compositor thread
can handle input events.
main
impl
scroll
scroll
scroll
busy
input from
browser process
renderer process
LayerTreeHost
layer promotion
Layers are created by "promoting" elements with compositing triggers.
LayoutView
LayoutBlockFlow
LayoutBlockFlow
LayoutText
LayoutBlockFlow
transform: ...
layout tree
PaintLayer
PaintLayer
PaintLayer is a
compositing
"candidate".
layer tree list
cc::Layer
cc::Layer
🤦♂️
scrolling layers
Scroll containers
create a set of special layers.
LayoutBlockFlow
overflow: scroll
PaintLayer
main layer
CompositedLayerMapping
scrolling contents layer
horz. scrollbar layer
vert. scrollbar layer
scroll corner layer
scrolling layers
Compositing a transparent scroller disables subpixel antialiasing.
LayoutBlockFlow
overflow: scroll
CompositedLayerMapping
Android? ChromeOS?
high DPI display (DSF >= 1.5)?
solid background?
user disabled LCD font smoothing?
non-composited scroller
✓
no
cc::Layer
GraphicsLayer
compositing assignments
DOM
style
layout
paint
main
compositing assignments
PaintLayerCompositor::UpdateAssignmentsIfNeeded
cc::Layer
GraphicsLayer
cc::Layer
GraphicsLayer
cc::Layer
builds
DI
GraphicsLayer::Paint
DI
DI
Each layer is painted separately.
property trees
The compositor can apply some paint properties to a layer.
transform
tree
clip
tree
effect
tree
scroll
tree
property trees
...
layout
comp. assignments
paint
main
prepaint
PrePaintTreeWalk::Walk
The prepaint stage builds the property trees.
PaintPropertyTreeBuilder
composite after paint (CAP)
In the future, layers will be created after paint.
PaintArtifact
PaintOp
…
paint
main
composite
prepaint
...
compositing
assignments
cc::Layer
PaintArtifactCompositor
commit
main
impl
commit
paint
(blocked)
ready to
commit
commit
complete
…
…
copy layers and properties
cc::Layer
LayerImpl
tiling
impl
prepare tiles
viewport
layer
visible
tiles
Layers are broken into tiles for raster.
raster
raster
raster task
CategorizedWorkerPool
rastered
tile
GPU memory
CompositorFrame
DrawQuad
DrawQuaDrawQuad
d
draw layers
Tiles are "drawn" as quads.
impl
draw
Tile
TileDrawInfo
raster output
PictureLayerImpl::AppendQuads
(raster complete)
DrawQuad
rect = …
GPU
process
CompositorFrame is the "output" of the renderer process.
resource_
activation
impl
commit
raster
activation
draw
draw
pending
tree
active
tree
Drawing can continue while
a new commit is rastered.
LayerImpl
LayerImpl
LayerImpl
GPU process
display (viz)
browser UI
The display compositor
combines frames for multiple surfaces.
renderer process
CompositorFrame
browser process
CompositorFrame
renderer process
CompositorFrame
ui::Compositor
SurfaceAggregator
display
compositor
viz
GPU process
display (viz)
Display::DrawAndSwap
GLRenderer::DrawTileQuad
CompositorFrame
DrawQuad
GLES2Implementation
Viz draws quads
to the back buffer.
GLES2DecoderImpl
DrawQuad
viz
gpu
command
buffer
GPU process
display (viz)
Display::DrawAndSwap
SkiaRenderer::DrawTileDrawQuad
SkiaOutputSurfaceImplOnGpu
viz
gpu
SkDeferredDisplayList
SkSurface::draw
CompositorFrame
DrawQuad
DrawQuad
GPU process
display (viz)
Display::DrawAndSwap
Finally, we swap the buffers.
GLSurface::SwapBuffers
viz
gpu
GLRenderer::SwapBuffers
eglSwapBuffers(...)
The user can see the pixels.
GPU process
review
DOM
style
layout
comp. assign
paint
commit
raster
activate
main
impl
draw
display
Blink
tiling
prepaint
renderer process
<html>...</html>
content
end
LIFE OF A