1 of 13

Out of Process Rasterization

enne@

2 of 13

What is OOP-R?

Previous GPU raster pipeline:

paint (SkPicture) → raster task → raster (Skia) → gles2 command buffer → gl

Current OOP raster pipeline:

paint (paint ops) → raster task → raster command buffer → raster (Skia) → gl

3 of 13

Why OOP-R?

Top Level Goal: use Vulkan instead of OpenGL (for performance / stability)

Problem: command buffer has OpenGL baked in

Solution: create a new command buffer interface and serialization format, run raster (Skia) in the gpu process

Bonus goals: OpenGL is chatty; serializing paint data structures is faster

This will be short, as I think everybody understands the motivations here, but I think it’s important to go through it all the same. When I say Vulkan anywhere in this presentation, also substitute in Metal or DX12 or things like that.

Vulkan has less CPU overhead than GL. Metal is definitely more stable than OpenGL drivers. There’s some hope that Vulkan drivers will be easier to write and thus more stable??

Chrome has long used the command buffer as its OpenGL abstraction, which serializes OpenGL commands (and a lot of Chromium custom commands) from one untrusted process to another. It is not possible to make a Vulkan command buffer, and thus we need a different communication mechanism for graphics needs.

Solution is to write a new kind of command buffer interface, serialize data in something other than opengl, run raster and display compositing in the gpu process so that we can use a different back end.

Bonus goals here is that ideally this project in itself is also a speed up. We saw 10-15% wins on Android for total raster time.

The other complexity advantage here is that exposing gl extensions does not requiring plumbing them all the way through the command buffer to get them to Skia in the renderer.

4 of 13

Data Structures

Why?

Differing security needs
Image replacement
Mutability / updatability
Project differences

Mappings from analogous Skia classes:

SkCanvas -> PaintCanvas
SkPaint -> PaintFlags
SkPicture -> PaintRecord (== PaintOpBuffer)
SkShader -> PaintShader
SkImage -> PaintImage

PaintCanvas subclasses:

SkiaPaintCanvas is PaintCanvas -> SkCanvas

RecordPaintCanvas is PaintCanvas -> PaintRecord

The first step in serializing was to pick a format to serialize. We previously painted into Skia data structures, which already have their own serialization code. However, Skia’s serialization is there for primarily writing to files. It is less intended to be a general reader of possibly malicious data, let alone reading data from shared memory which may be modified by an attacker. So, the overall approach was to make a bunch of similar classes to Skia, and write our own serialization with fuzzing.

cc::Paint also has a number of other needs, such as replacing images while rastering. Skia recording data structures are not introspectable or mutable, and this limits future optimizations of incremental updates. There are also project differences, and it’s convenient to be able to add arbitrary APIs to Chrome’s version of SkCanvas and SkPicture in ways that Skia would not want to add themselves.

Also, nobody likes PaintFlags but you can’t really call it PaintPaint. Sorry.

There is no way to go from SkCanvas -> PaintCanvas or SkPicture -> PaintRecord. This is intentional. This is because if you have opaque Skia objects, there’s no way to make the equivalent Paint objects from them.

5 of 13

PaintOpBuffer / PaintRecord dissection

class CC_PAINT_EXPORT PaintOp {

public:

uint32_t type : 8;

uint32_t skip : 24; // size of this serialized op

};

PaintOps = heterogenous types serialized in a flat buffer, e.g.:

enum class PaintOpType : uint8_t {

Annotate,

ClipPath,

ClipRect,

ClipRRect,

Concat,

CustomData,

DrawColor,

DrawDRRect,

DrawImage,

DrawImageRect,

DrawIRect,

DrawLine,

DrawOval,

DrawPath,

DrawRecord,

DrawRect,

DrawRRect,

DrawSkottie,

DrawTextBlob,

Noop,

Restore,

Rotate,

Save,

SaveLayer,

SaveLayerAlpha,

Scale,

SetMatrix,

Translate,

};

SaveOp

DrawRectOp

DrawColorOp

RestoreOp

&c &c &c

PaintOpBuffer = PaintRecord as a typedef. I’ll say both and they mean the same thing.

�Original goal was to have an introspectable data structure that was serializable, and to use the same data structure for both recording into, serializing, deserializing, and rastering from. Trying to move away from things that have pointers so that ops can just be memcopied.

PaintOpBuffer is an array of PaintOps that are placement new’d into a flat buffer.

�PaintOps themselves have a one level class hierarchy that are named after their type, e.g. DrawRectOp, DrawImageOp, SaveOp.

Base PaintOp stores a type and a skip, i.e. offset to the next op in the buffer. Type lets you know what to cast the PaintOp* to and offset tells you where the next one starts. This is pretty similar to contiguous container, except instead of a side vector of void* that tells you where the ops are, the offsets are stored inline. Serialized ops also use this same type/skip header and in some cases are direct copies of the PaintOps.

PaintOps know how to raster to a canvas, can serialize to a buffer and be deserialized from a buffer.

Once OOP-Raster ships everywhere, it might be possible to pre-serialize more data at paint time such that the PaintOpBuffer itself more closely resembles the data that is serialized. Currently it is still very pointer-y.

6 of 13

Serialization Overview

command buffer

transfer buffer

mapped memory

GPURasterBufferProvider

RI::BeginRasterCHROMIUM(mailbox, etc)

RI::RasterCHROMIUM(record, settings)

Grab all transfer buffer space

POBSerializer::Serialize(paint record)

SerializePreamble()

for each op in record:

serialize op into transfer buffer

if it doesn’t fit:

SendSerializedData()

Allocate/wait until tb space available

Shrink transfer buffer to space used

SendSerializedData()

RI::EndRasterCHROMIUM()

rough call stack / psuedocode

BeginRC

RC

Create�TCEntry

RC

EndRC

(serialized ops)

image

data

glyph

data

GPURasterBufferProvider is the caller in cc of gpu and oop rastering. It makes calls into RasterInterface which provides oop raster. The details are in RasterImplementation.

From a command buffer point of view, BeginRasterCHROMIUM and EndRasterCHROMIUM bracket any number of RasterCHROMIUM commands, and can also include transfer cache commands as well. See AllowedBetweenBeginEndRaster in raster_decoder.cc. One way this can happen is because of an at raster image which is not decoded when we get to an image that needs it. The IssueCreateTransferCacheEntry will happen before the RasterCHROMIUM command that depends on it.

One design decision that has caused some headaches and maybe could be revisted is that the serialization code in cc doesn’t know how big serialized ops are ahead of time, so instead of measuring once before allocating, it just tries to allocate into what space it has and fails if it ever runs over. This unfortunately also means that oop-r hogs the transfer buffer, and anything that would have used the transfer buffer needs to go into mapped memory. The transfer buffer is shrunk at the end down to only the size used.

7 of 13

Command Buffer Changes

GL Command Buffer:

GLES2Interface + GLES2Implementation

Renderer side
All GL functions plus Chromium extensions

GLES2Decoder(Impl)

GPU side of gl command buffer

Raster Command Buffer:

RasterInterface + RasterImplementation

RasterCHROMIUM functions, sync tokens, also some texture functions

RasterInterface + RasterImplementationGLES

Allows using RasterInterface backed by GLES2 (for the non-OOPR functions)

RasterDecoder(Impl)

SharedContextState maintains shared GrContext + ServiceTransferCache + shader cache among all RasterDecoders

8 of 13

Transfer Cache Overview

Coordinates caching of data between the renderer and gpu.

Entries don’t depend on other entries.

Types: images, paint shaders, raw memory

Depends on discardable gpu memory system via gpu::(Client|Service)DiscardableManager. (Note: not the same as base::DiscardableMemoryAllocator!)

Create/Lock/Unlock commands in command buffer, issued by client but processed on service side.

Lock on client side and unlock on service side access shared memory, so no need for ipc to check availability.

GPU side can purge unlocked items freely so long as the client locks before using.

9 of 13

Image Serialization Details

Transfer cache used for all image types:�pre-decoded, at raster, lazy, non-lazy

ServiceImageTransferCacheEntry handles uploading, software vs hardware images, color space conversion

Serializing an image is just serializing its transfer cache id

Decoding still driven by TileManager and GpuImageDecodeCache in the renderer. Decodes still run in the renderer. Image lifetimes are the same between oop raster and gpu raster.

When using oop raster, the transfer cache is used for all image types: lazy, non-lazy, at raster, pre-decoded.

As a part of putting the image in the transfer cache, the decoded data is copied into the transfer cache from the GPUImageDecodeCache and serialized to the gpu process. In the gpu process, we upload the texture data and tag it with a colorspace. If the image is too large, we create a bitmap-backed SkImage, and do the color space conversion right then.

When serializing, we ask for the decoded draw image from the GPUImageDecodeCache (which may do that decoding and transfer cache entry creation as a part of that request) and then serialize the the transfer cache id

This leads to a little bit of logic duplication between GPUImageDecodeCache and ImageTransferCacheEntry that can get cleaned up once oop raster ships everywhere.

10 of 13

(Client|Server)PaintCache

Questions:

Yikes, why another type of caching?
Gosh, why cache small objects at all?

Implementation Details:

Used for paths and blobs, currently
Client side manages memory and budget
Paint cache insertion is part of serializing
Deserializer creates service side entries
Separate command buffer commands for deletion of entries
Cleared during raster context idle cleanup, so only a working set cache

Why another type of caching? Transfer cache locking is slow. It’s great for large things like images that change less frequently and are locked across multiple raster requests, but less good for things that can appear hundreds of times per raster task, like paths and blobs.

Why cache at all if small? Internal skia caches need a consistent object (e.g. paths have a generation id), and also prevent repeated serialization of the same object

Unlike transfer cache, client side manages memory and has a budget. The client manages the creation/deletion of all entries and thus knows the state of the service side cache.

When serializing paint ops and encountering one of these types, it writes a status of (1) empty (2) already cached + just the id, or (3) adds a new entry in the cache, serializes the object, and sends the id as well.

Separate command buffer commands to delete.

Note: possibly paint shaders are small enough to be here but haven’t run into a case where this happens enough to have motivated changing them.

11 of 13

Typeface and Glyph Serialization

Renderer Side

While serializing, draw all PaintOps into a SkTextBlobCacheDiffCanvas to find blobs.

SkStrikeServer + ClientFontManager find new typefaces and strikes, lock any strikes needed using discardable handles.

Prior to a RasterCHROMIUM command, serialize new typefaces and strikes into mapped memory.

GPU Side

When receiving a RasterCHROMIUM command, add all new typefaces and strikes to the SkStrikeClient + ServiceFontManager.

When rastering to a Ganesh-backed SkCanvas, look up any glyphs needed from the SkStrikeClient.

What is a strike? From Khushal: A strike is a logical grouping of glyphs which are likely to be used together during text rendering. A common example would be glyphs from a particular typeface and a particular size. The reason for maintaining this logical grouping is to help in caching of these glyphs. Firstly, we use this to allocate glyphs in a group on an arena, to avoid multiple small allocations and freeing of memory. Secondly, this makes a strike in a single unit of what we cache (instead of per glyph). So that when you purge the cache, you can delete the complete strike. And that would involve feeing a single block of memory, on which all glyphs were allocated.

Font serialization uses discardable handles just like gpu discardable or transfer cache entries. So, in this case, the gpu is responsible for purging when the budget goes over.

RasterCHROMIUM here itself has special parameters for the mapped memory and serialized font data.

Also yes, SkStrikeServer and SkStrikeClient are backwards from ClientFontManager and ServiceFontManager.

On the gpu side, we mostly hope that the same dance that we did applying all ops to the SkTextBlobCacheDiffCanvas creates the same state to look up glyphs when rastering to the real SkCanvas. Sometimes this doesn’t work out and glyphs disappear, but we’re tracking this with a DumpWithoutCrashing. See http://crbug.com/879110

12 of 13

Unexpected

Hurdles

Shader caching
Command buffer workarounds
gles3 vs gles2
Skia path renderers
Mac driver bugs

Previously, the gl that was exposed to Skia was the command buffer. OOP-R exposes “real” gl to Skia, which has a different set of capabilities and no longer has the protection of the command buffer.

The command buffer handles shader caching for us, so this needed to be reimplemented in Skia APIs to avoid regressing startup.

The command buffer has years and years of bugfixes and workarounds accumulated in it, which needed to be ported to Skia in order to not regress mysterious driver bug workarounds.

On Android, switching to “real” gl also meant switching from gles2 to gles3 which let Skia use more APIs. This caused bugs, so we forced it back to gles2. This isn’t an option on desktop platforms, but helped us ship on Android. This should probably be revisited.

Skia has SO MANY path renderers, and a number of perf tests regressed because we switched from using one path renderer to another which had different perf characteristics.

We also ran into a number of driver bugs, particularly on Mac, where ccameron has had to add flushes around quite a number of commands to not get crashes inside of Mac gl drivers.

13 of 13

Follow-up Work

Ship OOP-R everywhere
Clean up temp Skia objects
Canvas OOP-R
Parallelize GPU main thread
Software raster

Ship OOP-R everywhere:

currently on Android m72
webview m73
windows/mac/chromeos are canary finch trial and mostly ready to roll out
ship gpu raster and oop raster for ui
no plans to ship for Linux, will just make it the default without turning on gpu raster for Linux first ¯\_(ツ)_/¯

Once OOP-R is shipped everywhere, we can remove GPU discardable and some paths in the renderer. We can also remove creation of temporary Skia objects during paint that we have to pre-create because we don’t know at paint time whether or not we will be using oop-r, and can’t create them lazily in a raster worker because of thread safety issues.

Canvas OOP-R: once canvas is switched over to using shared image, the next step is to have it use oop-r, which would allow hardware accelerated canvas to use vulkan as well

Parallelize GPU main thread: previously running Skia in a raster work thread allowed for some parallelization between renderer and gpu and oop-r took this away, a next step would be to build ddls incrementally on other threads, this would enable more throughput and also decrease latency in the cases that the gpu main thread was overloaded

Software Raster: once we have raster workers in the gpu process, we can consider what to do with software raster. We could do software raster in the gpu proces but my intuition is that this will not be a performance win. We could consider swiftshader instead of software raster and really remove other paths. If we were able to remove this path, we could get rid of a lot of code in the renderer around different raster buffer providers and probably clean up raster interface some too.