1 of 55

W3C

WebRTC/MediaCapture

WG Meeting

January 11, 2018

8 AM PDT

1

Chairs: Stefan Hakansson

Bernard Aboba

Harald Alvestrand

2 of 55

W3C WG IPR Policy

2

3 of 55

Welcome!

  • Welcome to the interim meeting of the W3C WebRTC WG!
  • During this meeting, we hope to:
    • Make progress on open issues in webrtc-pc, media capture and webrtc-stats
    • Introduce the webrtc-quic and webrtc-ice documents
    • Discuss updates to the WG Charter
  • Editor’s Draft updates to follow meeting

3

4 of 55

Current Status of WebRTC-PC

  • 96 open issues:
    • 23 relating to Identity
    • 19 editorial
    • 13 arising from test suite development
    • 11 PR exists
    • 4 questions
    • 4 pending IETF actions
    • 2 icebox

4

5 of 55

About this Virtual Meeting

Information on the meeting:

​

February Virtual Interim: Focus on Screen Capture

5

6 of 55

Links to the latest drafts

​

New documents:

6

7 of 55

Agenda for Today

  • 16:00 - 16:45 UTC: Issues
    • WebRTC-PC
    • Media-Capture
    • Statistics
  • 16:45 - 17:15 UTC: Introduction to new documents
    • webrtc-quic
    • webrtc-ice
  • 17:15 - 17:30 UTC: Charter Review

7

8 of 55

For Discussion Today

  • WebRTC-PC Issues
    • Issue 1662: addTransceiver woes? (Stefan)
    • Issue 1689: Why is RTCRtpSynchronizationSource.voiceActivityFlag required-but-nullable? (Jan-Ivar)
    • Issue 1497: Not possible to tell how old `RTCRtpContributingSource.timestamp` is (Jan-Ivar)
    • Issue 1690: RTCRtpContributingSource.timestamp needs a clearer definition (Taylor)
    • Issue 1695: Effect of mute/disable on on-the-wire framerate is not described (AdamBe)

8

9 of 55

For Discussion Today (cont’d)

​

  • Media Capture Issues
    • Issue 472: How to implement web-compatible camera downscaling? (Jan-Ivar)
  • Statistics Issues
    • Issue 235: Is keeping stats around a memory problem?

(Jan-Ivar)

  • webrtc-quic and webrtc-ice (Peter Thatcher)
  • WebRTC WG re-charter (Bernard)

9

10 of 55

WebRTC-PC Issues

  • Issue 1662: addTransceiver woes? (Stefan)
  • Issue 1689: Why is RTCRtpSynchronizationSource.voiceActivityFlag required-but-nullable? (Jan-Ivar)
  • Issue 1497: Not possible to tell how old `RTCRtpContributingSource.timestamp` is (Jan-Ivar)
  • Issue 1690: RTCRtpContributingSource.timestamp needs a clearer definition (Taylor)
  • Issue 1695: Effect of mute/disable on on-the-wire framerate is not described (AdamBe)

10

11 of 55

Issue 1662: addTransceiver woes? (Stefan)

​

​

  • #1662 raises two questions for addTransceiver(kind):
    • 1. What is the direction (in SDP offer, and transceiver.direction)
    • 2. What is the mid ?
  • Mid question resolved (‘pending’ mid at creation, may be overridden by SDP answer)
  • Direction: What is the default (Note: can be overridden by app)
    • Spec now says: default is always “sendrecv”
    • Has been argued that if transceiver is created with “kind” as first argument the default direction should be “recvonly”
    • My recommendation: keep “sendrecv”, simple and no strong reason for changing
  • Follow up question: Should replaceTrack work if no track was ever attached?
    • Conclusion: yes (in fact, the ‘warm-up’ example (Example 13) in the spec requires it)
  • Further conclusions:
    • Need to clarify “send”
    • Need to clarify that the direction should not be considered when determining if negotiation is needed

11

12 of 55

Issue 1689: Why is RTCRtpSynchronizationSource.voiceActivityFlag required-but-nullable? (Jan-Ivar)

​

​

  • WebIDL team advice: prefer optional over null. (both only for feature detection).
  • Let’s allow implementations to not implement voiceActivityFlag yet (like Firefox):
    • dictionary RTCRtpSynchronizationSource : RTCRtpContributingSource {� required boolean? voiceActivityFlag; // unimplemented, null=no-header-ext, false=v, true=v�};
  • Where implementation is required, use optional over null:
    • dictionary RTCRtpContributingSource { ...� required byte? audioLevel; // undefined=no-header-ext, 0-255=level�};
  • Internally inconsistent, but consistent with best practice.

​

​

12

13 of 55

Issue 1497: Not possible to tell how old `RTCRtpContributingSource.timestamp` is (Jan-Ivar)

Issue 1690: RTCRtpContributingSource.timestamp needs a clearer definition (Taylor)

​

​

  • Define timestamp in reference to context's global monotonic clock.
  • Prefer to have something that gives "something like the wall clock" for logs, and for somewhat better backwards compatibility.
  • Comparable syncOrContribSource.timestamp - stats.timestamp
  • Sounds like we want a timestamp that's comparable to either:
    1. performance.timing.navigationStart + performance.now() // page-load reset
    2. performance.timeOrigin + performance.now() // monotonic to browser-startNEW
  • Ignore footgun of “don’t compare it to Date.now()!” (fiddle)

​

13

14 of 55

Issue 1695: Effect of mute/disable on on-the-wire framerate is not described (AdamBe)

​

​

  • From the original mediacapture-main issue (#441)
    • At what framerate should the “blackness” from a muted or disabled MediaStreamTrack be transmitted?
    • Firefox and Chrome seem to be doing different things here
    • The effect may be observable with MediaRecorder and stats
  • Discussion at TPAC
    • According to minutes [1], we decided to specify 1 fps as an advice.
  • Currently in the spec:
    • “If track is ended, or if track.muted is set to true, the RTCRtpSender sends silence (audio) or a black frame (video).”

[1] https://www.w3.org/2017/11/06-webrtc-minutes.html#item19

​

14

15 of 55

For Discussion Today (cont’d)

​

  • Media Capture Issues
    • Issue 472: How to implement web-compatible camera downscaling? (Jan-Ivar)
  • Statistics Issues
    • Issue 235: Is keeping stats around a memory problem?

(Jan-Ivar)

  • webrtc-quic and webrtc-ice (Peter Thatcher)
  • WebRTC WG re-charter (Bernard)

15

16 of 55

Media Capture

​

  • Issue 472/PR 502: How to implement web-compatible camera downscaling? (Jan-Ivar)

16

17 of 55

Issue 472/PR 502: web-compatible camera downscaling (jib)

Peter's {resizeMode: “crop-and-scale”} constraint from TPAC, modulo "box-and-scale" *

​

​

��� enum VideoResizeModeEnum {� "none",� "crop-and-scale"� };

​

“The UA SHOULD use the one with the smallest fitness distance, as calculated in step 3, but MAY prefer ones with resizeMode set to "none" over "crop-and-scale".”��*) "box-and-scale" adds pixels and is incompatible with the constraints algorithm (these fitness distances would compete with "crop-and-scale" and "none" modes, with unpredictable and undesirable results).

​

17

resizeMode

ConstrainDOMString

This string (or each string, when a list) should be a member of VideoResizeModeEnum. The members describe the means by which the resolution can be derived by the UA. In other words, whether the UA is allowed to use cropping and downscaling on the camera output.

​

The UA may disguise concurrent use of the camera, by cropping and/or downscaling to mimic native resolutions when "none" is used, but only when the camera is in use in another browsing context.

none

This resolution is offered by the camera, its driver, or the OS.�

Note: The UA may report this value to disguise concurrent use, but only when the camera is in use in another browsing context.

crop-and-scale

This resolution is downscaled and/or cropped from a higher camera resolution by the user agent.

18 of 55

Statistics

​

  • Issue 235: Is keeping stats around a memory (really speed) problem? (Jan-Ivar)

18

19 of 55

Issue 235: Is keeping stats around a speed problem? (jib)

  • getStats() gathers a non-trivial amount of live data, and is often called at a high frequency.
  • Two uses of getStats: a) Live operational feedback, b) Accounting. Mirrors different impls:
    1. Firefox reports snapshots of live objects that exist at that time. Removed tracks disappear
    2. Chrome stores everything that ever happened.�
  • Uncapped accumulation of stats slows down getStats() over time + complicates result parsing:��[...(await sender.getStats()).values()].find(s => s.type == “track”) // latest track stats?�
  • Use-cases: “always-on” meeting rooms, security feeds, flipping between cameras (front/back):
    • Repeated removeTrack/addTrack accumulates RTCRtpTransceivers + related stats.
    • Repeated sender.replaceTrack accumulates “track” stats, even flipping same tracks.
    • Frequent ICE restarts accumulate old ice candidates.�
  • Spec since 2012: “The basic statistics model is that the browser maintains a set of statistics referenced by a selector. The selector may, for example, be a MediaStreamTrack. For a track to be a valid selector, it MUST be a MediaStreamTrack that is sent or received by the RTCPeerConnection object on which the stats request was issued.”

19

20 of 55

Issue 235: Is keeping stats around a speed problem? (jib)

Proposal to fix this:

Omit objectDeleted stats from getStats, and add new methods to get them instead:

partial interface RTCRtpSender { partial interface RTCRtpReceiver {� RTCStatsReport getCompletedStats(); RTCStatsReport getCompletedStats();�}; };��partial interface RTCPeerConnection {� RTCStatsReport getCompletedStats(optional MediaStreamTrack? selector = null);�};

These return deleted stats (tracks + “stopped” senders/receivers) synchronously from a cache.

Benefits: Simpler parsing (no filtering on booleans). Isolates overhead (out of live-update hot path).

Trivial to combine results in JS:�[...(await sender.getStats()).values(), ...sender.getCompletedStats().values()] // All tracks

​

20

21 of 55

Issue 235: Is keeping stats around a speed problem? (jib)

If previous proposal is well received, a question: Do we need “track” stats in getStats()?

Users can compute live “track” stats from “sender” stats using JS:

let senderStat = [...(await sender.getStats()).values()].find(s => s.type == “sender”);�let oldTracks = [...sender.getCompletedStats().values()].filter(s => s.type == “track”);

let trackFramesSent = oldTracks.reduce((n, old) => n - old.framesSent, senderStat.framesSent);

​

21

22 of 55

WebRTC-QUIC Status

At TPAC, we decided to start an extension spec�(to decide if we like it).

Now, here it is:

�https://w3c.github.io/webrtc-quic/

https://github.com/w3c/webrtc-quic/issues

​

​

22

23 of 55

WebRTC-QUIC Approach

  • QuicTransport constructed from IceTransport (and optional sequence of certificates)
  • Outgoing QuicStream from .createStream()
  • Incoming QuicStream from .onstream
  • Read with readInto(buffer)
  • Write with write(data)
  • Controls for buffering/back-pressure, finish (clean end), reset (abrupt end)
  • Access to state

23

24 of 55

WebRTC-QUIC Stream states (summarized)

As "receiver"

.onstream: -> open

FIN/RST received -> closing

finish or reset() -> closed

��Write when open/closing

Read when open

As "sender"

createStream: -> opening

stream frame acked -> open

finish() or reset() -> closing

FIN/RST received -> closed�

Write when opening/open

Read when open/closing

24

25 of 55

WebRTC-QUIC Issues Raised

  • "QUIC" in the charter
  • QUIC vs SCTP
  • WHATWG streams
  • "unadorned" QUIC
  • ALPN
  • Unidirectional/bidrectional streams
  • 0-RTT

25

26 of 55

"QUIC" in the charter

Instead of mentioning "QUIC" in the charter, should we explain what we want to accomplish?

​

We should discuss in the charter discussion. But if it's not QUIC, what is it?

26

27 of 55

QUIC vs SCTP

A desire for "good reasons" for QUIC vs SCTP.�

Some of my reasons:

  • Fewer round trips to setup
  • Ease of deployment/termination
  • Robustness, maturity, variety of impls
  • Direction of future innovation/protocols

27

28 of 55

WHATWG streams

Basically choose between these two things.�Proposal: Keep the left w/o dependencies

readInto(Uint8Array); write(Uint8Array);�waitForReadable(amount); waitForWritable(amount);

​

(And implement WHATWG streams on top, if you want them)

​

attribute ReadableStream readableStream;

attribute WritableStream writableStream;

​

(With a big dependency on WHATWG ReadableStream and WritableStream)

​

28

29 of 55

"unadorned" QUIC

To do multiple protocols on top of QUIC streams over the same 5-tuple at the same time, need either a new mechanism for segregating the QUIC streams, or multiple QUIC connections. This is an issue with QUIC in general.

�Proposal: If a solution comes up for QUIC in general, use it. Maybe propose one in QUIC WG.

​

​

29

30 of 55

ALPN

You're supposed to put a value in the ALPN field in the client hello. It's supposed to identify the "application protocol". Who decides what that is?

​

Proposal: let the JS decide.

30

31 of 55

Unidirectional/bidrectional streams

The API is bidirectional because unidirectional streams didn't exist in QUIC until very recently. But it does make sense to support both.��Proposal: Update createStream/onstream to support both.

​

31

32 of 55

0-RTT

We don't have anything in the API for 0-RTT, which QUIC is capable of. Should we add support for it?

​

Proposal: Finish everything else we want and then come back to this later.

32

33 of 55

WebRTC-ICE Status

At TPAC, we decided to start an extension spec�(to decide if we like it).

Now, here it is:

�https://w3c.github.io/webrtc-ice/

https://github.com/w3c/webrtc-ice/issues

​

​

33

34 of 55

WebRTC-ICE Approach

  • Extends existing IceTransport object with:
    • Constructor (no need for PeerConnection)
    • gather(...) to start gathering candidates
    • start(...) to start pairing/checking/selecting
    • onlocalcandidate to get local candidates
    • addRemoteCandidate to add remote candidates
  • Does not have separate IceGatherer object
  • Does not support parallel forking

34

35 of 55

Tricky Parts: What if you don't call gather()?

If you don't call gather(), you won't get local candidates.

So you have to call both gather() and start().

Is that too big of a foot gun?

​

The alternative is to make start() semi-magic. But then ICE restarts become less clear.

​

Proposal: Require calling both. You need to, anyway, for good use of trickle ICE on the caller side. So it encourages good use.

​

​

35

36 of 55

Tricky Parts: ICE Restarts

To initiate a full ICE restart, two things must happen:

  1. New local parameters (ufrag/pwd) and candidates
  2. New remote parameters (ufrag/pwd) and candidates

​

Proposal:�- New gathering done via call to gather(), which changes local parameters�- New remote parameters via restart(), which changes remote parameters

Like when you first start, you must call both. Again, on the restarter side, this is good trickle practice.�

36

37 of 55

Tricky Parts: Changing gathering policy

If you want to add TURN servers without an ICE restart, like with PeerConnection.setConfiguration, what do you do?

​

Proposal: Have a variant of gather() which doesn't change the local ufrag/pwd, but just gathers new candidates.

�

37

38 of 55

Tricky Parts: Gathering done

Like with PeerConnection, it might be useful to signal "end of candidates" easily out of the box.

​

Proposal: Make it just like WebRTC: candidate.candidate == "" means "end of candidates". It's ugly but it works and we don't have to specify anything new.

38

39 of 55

Open Question: stats

Option A: iceTransport.getStats()

​

Option B: statCollector.getStats(iceTransport)

​

​

Which one?

​

39

40 of 55

An alternative to IceTransport: SliceTransport

SLICE: Simple, Low-level ICE

​

Basically, Cullen's idea from TPAC of a low-level ICE where the JS gets to control (almost) everything.

​

40

41 of 55

Today

41

Application

JS

IceTransport in Browser

IP Network Enumeration

STUN

TURN

ICE checks

ICE agent (gathering, pairing, timing, ...)

UDP ports

TCP ports

42 of 55

Cullen's idea

42

IP Network Enumeration

STUN

TURN

ICE checks

ICE agent (gathering, pairing, timing, ...)

UDP ports

TCP ports

Application

JS

Low-level components in Browser

43 of 55

New APIs

43

NetworkManager

StunClient

TurnClient

SliceTransport

UdpPort

TcpPort

Application

JS

Low-level components in Browser

44 of 55

How to use them

44

  • Use NetworkManager to enumerate network interfaces and IPs
  • Use NetworkManager to open UdpPorts and TcpPorts.
  • Use UdpPort/TcpPort to connect to STUN/TURN servers
  • Use StunClient to obtain server reflexive addresses
  • Use TurnClient to allocation TurnAllocations
  • Signal addresses to remote side and obtain remote addresses
  • Pair addresses together
  • Use UdpPort/TcpPort/TurnAllocation + remote address to get PacketTransports
  • Add PacketTransports to SliceTransport as IceNetworkRoutes
  • Send checks over IceNetworkRoutes
  • Select an IceNetworkRoute

45 of 55

SliceTransport: checking and selection

45

interface IceTransport {

// Can receive any network route and send on selected one

IceNetworkRoute addNetworkRoute(PacketTransport transport);

void selectNetworkRoute(IceNetworkRoute networkRoute);

void removeNetworkRoute(IceNetworkRoute networkRoute);

}

interface IceNetworkRoute {

// Sends an ICE check. Resolves when the response is received

Promise sendCheck(...);

// Checks responses automatically sent.

eventhandler oncheckreceived;

}

​

​

46 of 55

UDP PacketTransports

46

interface UdpPort {

readonly attribute IpPort localAddress;

UdpTransport connect(IpPort remoteAddress);

void close();

// Fired once for each new remote address

eventhandler onnewremoteaddress;

}

// Only usable within SliceTransport, not by itself.

interface UdpTransport : PacketTransport {

readonly attribute IpPort localAddress;

readonly attribute IpPort remoteAddress;

void close();

}

​

​

​

​

​

47 of 55

TCP PacketTransports

47

interface ClientTcpPort {

readonly attribute IpPort localAddress;

Promise<TcpTransport> connect(IpPort remoteAddress,

TlsMode tlsMode);

void close();

}

// Only usable within SliceTransport, not by itself.

interface TcpTransport : PacketTransport {

readonly attribute IpPort localAddress;

readonly attribute IpPort remoteAddress;

void close();

}

​

​

​

​

48 of 55

How to get IP, UDP, TCP (with permission)

48

interface NetworkManager {

Promise<sequence<NetworkInfo>> EnumerateNetworks();

Promise<UdpPort> OpenUdpPort(IPAddress localIp);

Promise<ClientTcpPort> OpenClientTcpPort(

IPAddress localIp, bool tls);

attribute eventhandler onnetworkschanged;

}

​

interface NetworkInfo {

readonly attribute sequence<IPAddress> ips;

readonly attribute NetworkAdapterType type;

}

​

​

​

​

49 of 55

Getting STUN addresses

49

interface StunClient {

// Resolves when response is received

Promise<IpPort> sendBindingRequest(

PacketTransport transport, ...);

}

​

​

​

​

50 of 55

Getting TURN addresses

50

interface TurnClient {

Promise<TurnAllocation> sendAllocationRequest(PacketTransport transport, ...);

}

interface TurnAllocation {

// TODO: Are refreshes automatic or controlled by JS?

readonly attribute IpPort serverAddress;

Promise sendCreatePermissionRequest(IpPort remoteAddress);

TurnTransport connect(IpPort remoteAddress);

void close();

eventhandler onnewremoteaddress; // IpPort

}

interface TurnTransport : PacketTransport {

// TODO: Are channels automatic or controlled by JS?

readonly attribute IpPort serverAddress;

readonly attribute IpPort remoteAddress;

void close();

}

​

​

​

​

​

51 of 55

Advantages

51

  • Apps can optimize for their use case:
    • TURN-first
    • Control over wifi/cell usage
    • Add non-relay candidates without an ICE restart
    • Continual gathering or ICE half-restarts
    • Long-lived candidates
    • Backup candidates pairs
    • Variable check rate
  • New capabilities automatically/naturally:
    • Connect to TLS host candidates
    • Parallel forking
    • TURN within TURN

...

​

​

​

​

52 of 55

WebRTC WG Charter (Bernard)

​

  • Charter proposal:
  • Changes:

52

53 of 55

WebRTC WG Charter Discussion (cont’d)

  • Issues raised on the list:
    • General stream API (for SCTP as well QUIC?)
    • Use cases
    • Protocol issues (for IETF)
      • Multiplexing (e.g. single QuicTransport limitation)
      • Pluggable congestion control for data channels
      • Definition of QUIC data channel protocol
      • ALPN usage with QUIC
    • Collaboration with IETF beyond RTCWEB WG:
      • QUIC WG (reliable and unreliable QUIC transport)
      • Congestion control WGs (RMCAT, TSVWG) and RGs (ICCRG)
  • Please file Issues (and PRs)!

53

54 of 55

For extra credit

54

Name that bird!

55 of 55

Thank you

Special thanks to:

W3C/MIT for WebEx

​

WG Participants, Editors & Chairs

The bird

55