DrawPile architectural design

Introduction

Architecture

State

Brush

Network protocol

Command stream

Resize canvas

Layer create

Layer attribute change

Layer order change

Layer delete

Putimage

Tool change

Pen move/up

Annotations

Undo stroke

Meta stream

User joins

User leaves

User state change

Session title change

Chat

Layer ACL change

Snapshot mode

Stream position

Disconnect

Delay

Login stream

Server side state

Locking users

Barrier lock

Hard lock

Layer lock

Command processing while locked

Synchronizing new users

Undo

Redo

Implementation example for a tile based drawing engine

Access controls and operator commands

Session recording

Raw recording

Indexed recording

Introduction

DrawPile is a collaborative networked drawing program that enables multiple people to draw on the same canvas simultaneously. This document describes the redesigned architecture and network protocol introduced in version 0.7.0

Architecture

On a high level, the drawing program consists of the following components:

• The Graphical User Interface includes the drawing tools and the view of the drawing canvas
• The network client handles communication with the server. All drawing actions pass through here
• The loopback server is a dummy server that echoes everything back to the client. It is used when the user is drawing alone.
• The server (accessed via network) handles input from all clients and ensures a consistent command stream.
• The state tracker converts the command stream into primitive drawing commands for the paint engine
• The paint engine is responsible for the drawing canvas and applying modifications to it.
• The GUI widget presents the paint engine to the user interface

State

To ensure consistent user experience, the state of all program instances must be synchronized. The state consists of:

• The canvas contents

• Layer attributes
• Layer order
• Layer contents

• Annotations
• Drawing context state

The state is kept synchronized by ensuring a consistent ordering of all state mutating commands. This is accomplished by having all such commands travel through a central server which determines the official order.

The canvas is a stack of layers. Each layer has a set of attributes, such as its name, blending mode and opacity. The layer content is a bitmap image.

Annotations are floating text layers. They always float above the layer stack and have no defined ordering.

Each user has a drawing context. The drawing context records the state needed to apply the drawing commands. It contains information such as the coordinates of the last stroke and currently selected tool settings.

Brush

Presently, the only supported drawing tool is the brush. The brush requires only one primitive from the paint engine: the dab. Dabbing the brush means drawing a single picture of the brush onto a layer. A stroke consists of a series of line segments. Dabbing can be performed directly or indirectly. In direct mode, the dabs are drawn straight onto the canvas. In indirect mode, the dabs are drawn onto a temporary layer which, at the end of the stroke, is composited onto the canvas in one go. Indirect dabbing makes it possible to paint large areas of uniform color.

A brush has the following attributes:

• Composition mode
• Hard-edge mode (disable subpixel positioning)
• Dab spacing
• Color (transparency included)
• Size
• Hardness

In order to support pressure sensitive tablets, the attributes color, size and hardness come in pairs: one indicates the value at high pressure, the other at low. The actual values for a given pressure are linearly interpolated. Support for pressure curves may be added in the future. The pressure value at any given point on the line segment being drawn is linearly interpolated from the start and end pressures.

At least the following composition modes are supported:

• Erase (set alpha channel to zero)
• Normal (normal alpha blend)
• Multiply
• Divide
• Burn
• Dodge
• Darken
• Lighten
• Subtract
• Add

Network protocol

The DrawPile protocol is message based.

Every message begins with a three byte header:

The server may disconnect a misbehaving client. If possible, a chat message informing the reason for disconnection is sent.

Command stream

The command stream is the subset of the client/server protocol that mutates canvas content. The command stream can be saved to disk to replay the entire drawing process.

The following messages are part of the command stream:

• Resize canvas
• Layer create
• Layer attribute change
• Layer order change
• Layer delete
• Putimage
• Tool change
• Pen move
• Pen up
• Annotation create
• Annotation edit
• Annotation delete
• Undo stroke

Resize canvas

Set the size of the drawing canvas. This command must be sent before the first layer create.

Presently, this command is sent once only in the very beginning of the session or snapshot point.

Layer create

Initially, the session starts with an empty canvas containing zero layers. A layer create command is therefore the second drawing command that must be transmitted (right after canvas resize which sets the initial size). The layer is created with default attributes and filled with the given fill color.

When this command is sent by the client, layer ID 0 should be used. To server assigns an unused ID for the layer. An exception to this is when using layer create in snapshot mode. In this case the user (who is a session operator) must use the true layer ID which was previously assigned by the server. (Note. During session initialization, the client must create new layers in snapshot mode. In this case the client can just make the IDs up.)

Layer attribute change

The layer attribute change is used to update a layer’s attributes. The layer is identified by its ID.

Layer order change

New layers are always added to the top of the stack. This command includes a list of layer IDs that define the new stacking order. A order change command sent by the server must list all layers. A command sent by the client may be missing (newly created) layers, in which case the missing layers will be included at the top of the stack in their existing relative order.

For example: if the current stack is [1,2,3,4,5] and the client sends a reordering command [3,2,1], the server must add the missing layers: [3,2,1,4,5].

Layer delete

Cause the deletion of the layer identified by an ID. Only session operators are allowed to use this command.

Putimage

This command is used to draw arbitrary bitmap data onto a layer. This is most often used when constructing a snapshot point or importing a starting image, but it can also be used for baking text layers and pasting pictures from the clipboard, etc. The put image is not blended, but simply overwrites existing layer content.

The user ID does not affect the way this command is executed.

The image data is transmitted in DEFLATEd ARGB pixel format. Due to the 64Kb message length limitation, a large image may not fit in a single message. In this case, the image should be divided into multiple putimage commands.

Tool change

A tool change must be sent before a pen move command if the settings have been changed. The initial tool selection is undefined!

Note that there is typically a 1:1 mapping between a drawing context and a user, but the state tracker/paint engine does not know or care about users. Drawing contexts may be recycled if a new user joins after another has left.

Pen move/up

The pen move command includes the coordinates for the new pen position and pressure. If the pen status was up, the command starts a new stroke. The stroke ends with a pen up command.

If the currently active layer set by the previous tool change command does not exist (e.g. has been deleted), the pen move command is ignored.

A pen move command may include up to 10922 () X,Y,P triplets. Note that packed moves are not just an optimization but are semantically different: since the path is packed into a single command, it is applied atomically, meaning no other user’s path can be interleaved with it.

Annotations

An annotation is a floating text layer. Annotations always float over all other layers. An annotation can be converted to a bitmap and merged with a layer (baked). The user baking the annotation is responsible for generating the bitmap and transmitting it to other users, as due to font and possible rendering engine differences, the same text may not be rendered identically on all instances.

An annotation is created with the annotation create command.

An annotation can be moved and resized with a annotation reshape command. Its attributes are:

To change the content of an annotation, an annotation edit message is sent:

Annotation edit commands referencing a missing annotation are silently ignored. (typical cause: annotation deleted by another user mid-edit.)

How the annotation is rendered is very client specific, as differences in installed fonts and library versions may affect the way text is drawn. For this reason, annotation baking (convert to pixel data and merge with a layer) does not have its own command, but should be implemented using PutImage.

An annotation can be deleted with the annotation delete command.

Undo stroke

Undo the previous action(s) by this user. See the “Undo” section for more information.

Meta stream

The meta stream is the part of the protocol that deals with commands that do not directly affect drawing.

The following commands are part of the meta stream:

• User joins (server → client only)
• User leaves (server → client only)
• User state change (server → client only)
• Session title change
• Chat
• Layer ACL change
• Snapshot mode
• Stream position (server → client only)
• Disconnect
• Delay

The meta stream can be filtered out from the session recording with no effect on the drawing, however it can also be left in to preserve user names and chat messages.

User joins

A new user has joined the session. Includes the user name and drawing context ID. The drawing context ID may be recycled, but the server may avoid doing so to make session recordings neater.

Duplicate usernames are allowed, but a server may also optionally enforce uniqueness.

This has no other effect on the program state, except associating a user name with a drawing context.

User leaves

The user with the given context ID has left the session. It is acceptable for a client to invalidate the corresponding drawing context.

User state change

The user state change has no effect on drawing (that is, locks are not enforced client side), but it can be used to update the user interface.

Session title change

Change the session title. By default, no title is set. This command is only accepted if it is sent by a session operator.

Chat

A chat message sent by the user with the given context ID. Some session operator commands are sent using the chat interface. (IRC style commands.)

Layer ACL change

ACL change is used to change layer access controls. Default ACLs are: unlocked and free for all.

Note. Layer ACL change is considered part of the meta stream rather than the command stream since it affects how the server filters the command stream and not how the commands are applied. ACL change commands are sent to clients so they can update their user interface hints, but they must have no effect on the interpretation of drawing commands!

Snapshot mode

The snapshot mode command relates to snapshot point creation. When a server creates a snapshot point, it needs a client to build the actual snapshot content.

The server starts by sending a REQUEST type snapshot mode command to the client. The client can send the existing session history, if available, as a snapshot in response. The REQUEST_NEW means the client should construct a new snapshot and send that.

Upon receiving a snapshot request, the client generates the snapshot and responds with an ACK mode command to indicate the snapshot is ready.

The client will then start sending the commands that make up the snapshot, with each command preceded by a SNAPSHOT type message. This allows the snapshot uploading to be interleaved with normal traffic. Note: In snapshot mode, the client is allowed to set the drawing context ID to something other than their own! This makes it possible to transmit a complete session recording as the initial state.

To signal the end of the snapshot, an END type message is sent.

See the “Synchronizing new users” section for details.

Stream position

The stream position is occasionally sent by the server to inform that the client’s stream pointer is much behind the stream head. This allows the client to draw a progress bar or some other hint to indicate it is not fully up to date.

Disconnect

The disconnect message is a friendly notification of disconnection. The connection is closed immediately after sending this message. Possible reasons are:

• Quit session (no error)
• Kicked out (kicked by session operator. Error code is the user ID of the operator)
• Protocol violation (error code indicates the type of violation)
• Server error (an internal server error occurred)

Delay

This command is used to preserve stroke timings for session recordings. It may be transmitted over the network, but should be ignored when not in replay mode.

Login stream

The third and final part of the client/server protocol is the login protocol.

The login process is as follows:

1. Server sends: DRAWPILE (version) [PASS]
2. Client disconnects if it doesn’t support that version
3. If PASS was included, the client responds with the password
4. If password was wrong, server sends BADPASS and disconnects.
5. Otherwise server responds with OK.
6. Client sends: HOST <minor version> <userid> <username> or JOIN <username>
7. If the name is taken or otherwise unacceptable: BADNAME and disconnect
8. If attempted to join a non-existent session: NOSESSION and disconnect
9. If the session is not accepting new users (full or manually closed): CLOSED and disconnect
10. Login success: OK <user id>

The client connects to the server with the intention of either hosting a session or joining an existing session. The first user to connect to the server must host the session. The server will automatically request the initial snapshot from the hosting user.

The hosting user, being the first user on the server, gets to decide their own user ID. (Note. ID=0 is reserved.) This is to preserve the context ID they used when creating the initial state for the session.

The server version number is in the format <major>[.<minor>], where the major number is the protocol revision number and minor number is the client version. If the client’s major version number differs from that of the server, they should disconnect. The server may kick out any client sending unrecognized commands. Differences in minor versions may cause inconsistent rendering on the client side. The server may lock clients with different minor versions by default.

Note. The as the minor version number is a property of the client and not the server, it is set by the hosting user.

Server side state

The server is a “dumb server” in that it does not understand the semantics of the drawing commands beyond what is needed to distribute them to users in a consistent order.

The server side state can be divide into two parts: the command stream record and the current state.

The command stream record is the history of the canvas. The canvas state can be reconstructed by playing back the command stream, starting from a snapshot point. The start of the record is an implicit snapshot point. Explicit snapshot points can be created by establishing a known state and saving it.

The server does not track the state of the canvas (except for a list of used layer and annotation IDs), but it does keep track of the users.

Each user context has three important attributes:

• Command stream pointer, which indicates how much of the command stream has been sent to the user. The client is up to date when the pointer points to the head of the command stream
• Substream pointer, which is used when the current command contains another command stream (snapshot point)
• Input queue, which is the command stream received from the client. Commands are taken from the queue, validated and added to the command stream.

The server tracks part of the user’s state:

• Lock status (server side state only)
• Pen up (needed for barrier locking)
• Layer (needed for layer locking)
• Active and used user IDs

Some of the canvas state is also tracked:

• Layer order
• Layer lock status & ACL (enforced on server side only)
• Active and used layer IDs
• Active and used annotation IDs

To conserve memory, the server may discard the command stream. In this case, a new snapshot point must be constructed when a new client logs in. A server may also choose to construct a new snapshot point if it estimates that less data will have to be transferred with a fresh snapshot. The session must terminate when the last user logs out and no snapshot point exists in remembered history.

Locking users

A locked user is prevented from interacting with the canvas. Locks come in two types: barrier locks and hard locks. A barrier lock is a short-lived lock used to establish a consistent state across all users when creating a snapshot point. A hard-lock is typically set by the session owner to prevent users from drawing.

Note that a user can be both hard locked and barrier locked at the same time, and lifting one lock does not affect the other!

Also, locks are enforced on the server only! Clients should apply all drawing commands received from the server, even if they originate from users thought to be locked.

Barrier lock

A barrier lock is issued by the server when it needs to synchronize user states to create a snapshot point. A barrier lock always affects all clients participating in the session.

A barrier lock has two states: pending and locked. If the user is in the middle of a stroke, the state will be set to pending. The state will change to locked automatically on pen-up. If the user is already in pen-up state, the user will be locked immediately. A timeout can be used to combat misbehaving clients. If the user has not sent a pen-up within the timeout, a pen-up will be synthesized by the server and the user locked.

The barrier lock will be complete once all users have been locked.

During a barrier lock, the client’s input queue is put on hold. When the lock is lifted, the input queue is added to the command stream as normal.

Hard lock

A hard lock is used to prevent individual users from drawing. The lock is typically set by the session owner. If the user who is being locked is currently drawing, a pen-up event is synthesized by the server to forcefully end the stroke. All drawing commands received during a hard lock are silently dropped.

Layer lock

This lock type is layer specific rather than user specific. A layer is locked (from the users perspective) if its general lock bit is set or if it has exclusive access. Drawing commands directed at a locked layer are quietly dropped, but commands already in the input queue are still added. A pen-up command is synthesized to stop drawing on the locked layer if necessary.

Command processing while locked

In normal unlocked operation, a received command is first added to the user’s input queue. When not on hold, the input queue contents is added to the canonical command stream.

Synchronizing new users

Synchronization can begin from a snapshot point. A snapshot point is a point containing all the required information to start a session. The session begins with a snapshot point that sets up the canvas. New snapshot points can be created using the following process:

1. Barrier lock all clients
2. Select a client and send request for a snapshot
3. Wait until the client has confirmed snapshot
4. Construct an incomplete snapshot point
5. Lift barrier lock
6. Receive snapshot data (asynchronous)

After login, the new client is synchronized by first sending a “User joined” command for each user (self excluded) and setting command stream pointer set to the snapshot point. If no snapshot point exists at the time, the client will be set to “WAITING FOR SNAPSHOT” state. As soon as a snapshot point exists, the client’s stream pointer will be set to it and the state changed to “IN SESSION”.

A snapshot point is different from other stream commands in the following ways:

• Snapshot point is a pseudo command, meaning it is never actually transmitted or saved in a session recording. A recording must start from a snapshot point (usually the implicit one at the start of the session), but what is saved is the snapshot’s substream.
• It contains a substream: When pointed to, commands from the substream are enqueued until the end of the substream reached.
• The substream may be incomplete: to support asynchronous snapshot upload, the snapshot point may be marked as incomplete. In this case, the main stream pointer is not incremented even if end-of-substream is reached, as more will soon follow.

The initial content for the new snapshot point is the introduction of users logged in at the moment the snapshot was generated. The content describing the drawing is constructed by the selected user. Typically the oldest session operator is selected. Since the snapshot can be fairly large, snapshot related commands are transmitted asynchronously. To distinguish snapshot building commands, each command is preceded by a “snapshot mode” message.

A new snapshot of an existing session contains the following:

• Canvas size
• Session title
• Layer initialization (in layer stack order)

• Layer creation command
• Layer attributes
• PutImages to fill the layer

• Annotation creation commands
• User tool changes

A session initialization snapshot may also be the command stream of a previously recorded session.

Note that unlike in normal layer/annotation creation, the client includes the real IDs since they have already been assigned by the server.

Undo

Note. This chapter is still tentative.

The protocol supports multi-user aware undoing of strokes.

Limitations:

• Undo cannot cross a snapshot point
• Undo is only accepted if the user is in pen-up state.
• Undo depth may be arbitrarily limited to conserve resources

The server records the undo commands like any other state mutating commands. To keep things simple, the undone strokes are not retroactively expunged from the command stream. However, they can be edited out from saved streams.

When an undo command is received, the last not-undone stroke by the user is marked as undone.

Both the server and client mark the stroke as undone. (Note. The undo flag is for local use only and is not included in the network protocol.) The server uses this information to validate undo commands. Invalid undo commands are silently dropped by the server.

The client must also undo the change to the canvas. This is done as follows:

1. Revert the canvas to some point before the stroke that is to be undone
2. Replay command stream from that point, excluding undone strokes

Redo

Tentative. Redoing can be implemented in two ways:

• Purely client side: retransmit undone stroke as new stroke. Does not preserve original ordering with other users’ strokes.
• Redo command: performs the same procedure as undo, except unmarks the oldest undone stroke in the sequence of undone strokes. Preserves original ordering, but needs server side support.

Implementation example for a tile based drawing engine

To enable canvas rollback without starting from the previous snapshot point, savepoints are made. The main difference between a snapshot and a savepoint is that while a snapshot contains all the data needed to reconstruct the state for a new client, a savepoint is specific to the client who created it and can thus utilize certain optimizations.

As with snapshots, the beginning of the command stream is an implicit savepoint. A simple implementation might not ever save any snapshots, but this would mean the entire session history would have to be replayed to perform an undo.

In addition to the canvas and state tracker content, a snapshot also contains a pointer to the command stream head at the time the snapshot was taken.

A savepoint is typically made on a pen-up event when all other users have also stopped drawing. Alternatively, a savepoint may be made after a set number of strokes have been drawn, or the total length of strokes drawn since the last savepoint exceeds some limit.

Savepoints between complete strokes

The savepoint contains:

• Layer tile pointers
• State tracker state
• Command stream pointer

Creating a savepoint:

1. Record command stream pointer
2. Save state tracker state
3. Save layer tile pointers. For every tile marked as dirty, replace tiles in current canvas with fresh copies.

To restore a savepoint:

1. Replace canvas tiles with savepoint tiles
2. Set state tracker state to saved state
3. Delete all newer savepoints
4. Replay commands from command stream, undone commands excluded

Access controls and operator commands

A user with the “operator” attribute has special privileges. The first user to join a session is automatically granted OP status. If the only operator in the session leaves, the next oldest user will be granted OP. An operator can also OP other users.

Operator actions are issued via the chat interface using IRC style commands:

Some drawing actions also require operator privileges:

• Layer creation
• Layer deletion
• Layer ACL change
• Canvas resize
• Session title change

Session recording

A session recording is essentially the command stream written to a file. There are two variants of the file format: raw recording, which is the command stream recorded as it is received and indexed recording, which contains additional metadata to simplify playback.

Both file types begin with a header:

The type is either R for raw recordings or I for indexed recordings. The major version is the protocol revision number. The minor version indicates changes in the interpretation of drawing commands. Reading a file with the same major but different minor version than the program will work, but may result in different output. Difference in the major version indicate compatibility breaking protocol changes. The client may support a backwards compatible mode for reading old recordings.

Raw recording

The raw recording format is a stream of commands in the same form as they are sent over the network. Like a network session, the recording must start with a snapshot point.

To reconstruct the canvas state at any given time, it is sufficient to record the command stream only. However, to preserve metadata such as user names, the following meta stream commands may be included:

• User login (associate context ID with user name)
• User logout
• Chat messages
• Title change
• Delays

Indexed recording

An indexed recording is otherwise like a raw recording, but includes an extra header in the beginning of the file. An indexed recording may also have been edited to remove undone strokes, combine stroke messages, etc.

An indexed recording may also contain savepoints to allow quick jumping back and forth. (tentative)

The header contains:

• Context id → username mappings (assumes non-recycled IDs)
• Stroke index
• Savepoint index (tentative)