1 of 58

TG4 Source Maps

June 2024

Jonathan Kuperman and Agata Belkius

2 of 58

This update

  • Our constituencies and process
  • Specification fixes
  • Open specification questions
  • New feature proposals
  • Our source map validator
  • Our source map tests
  • Feedback!

3 of 58

Quick updates

  • Our test suite has merged into FireFox DevTools https://github.com/mozilla/source-map/pull/505
  • We’ll soon be merged into Chrome DevTools as well (needs us to add a license)
  • Added the ecma BSD license
  • Our hackathon is June 24-25 in Munich (sign up here)

4 of 58

Goals of this update

  • Formed this TG a little less than a year ago
  • Hoping to come to plenary at the end of the year, looking for rubber stamp
  • Trying to find a communication balance with TG1
  • Want to give a more in-depth update, get feedback and show our roadmap

5 of 58

Constituencies

  • Generators. Bundlers, transpilers, compilers. Tools that have access to the source code and emit (among other things) source maps.
  • Debuggers. Browsers and stand alone debuggers. These are tools that have access to the source map and the generated code.
  • Error monitoring tools. Tools like Sentry and New Relic. These tools have access to an error stack trace and a source map.

6 of 58

Our Process

https://github.com/tc39/source-map/blob/main/PROCESS.md

7 of 58

Stage 1

The problem is defined in an explainer document. There might already be a sketch of a solution, but it's not required. There does not need to be agreement on the direction of the solution yet.

8 of 58

Stage 2

  • There's a concrete set of details written out for the precise format and what it does. The details are not final and further iteration is expected.
  • When a proposal reaches Stage 2, experimental implementation in all areas is encouraged. There is a very high risk of incompatible changes, so they should not be exposed to production users.

​

List of implementations: https://github.com/jkup/source-map-users

​

9 of 58

Stage 3

  • There is a complete written description of the solution.
  • There's at least 1 implementation (in generating tools, ideally also at other levels)
  • A test suite has been started (but it might not be complete)
  • When a proposal reaches Stage 3, no further improvements are possible without implementing the solution and testing it in real-world scenarios. All the other types of research have already been done in the previous stages.

10 of 58

Stage 4 (complete)

  • 2 source map generators ready
  • 2 interactive debuggers
  • 2 stack trace decoder implementations
  • Test suite is complete
  • When a proposal reaches Stage 4, it is ready to land in the editor draft. There might still be editorial changes, but all the semantics are final.

11 of 58

Fixes to the spec

Version is always the number “3” as an integer

12 of 58

Fixes to the spec

Clients fetching source maps over HTTP should check for and remove “)]}”

13 of 58

Fixes to the spec

Remove sections.url from index maps

14 of 58

Fixes to the spec

sourcemap HTTP header takes precedence over sourceMappingURL comment

15 of 58

Fixes to the spec

“Columns” in WebAssembly refer to byte offsets from the beginning

16 of 58

Fixes to the spec

“Columns” in WebAssembly refer to byte offsets from the beginning

17 of 58

Fixes to the spec

Deprecate x_google_ignoreList in favor of ignoreList

18 of 58

Fixes to the spec

No longer support x_ extensions

19 of 58

Fixes to the spec

Define WASM module custom section and the names binary format

20 of 58

Fixes to the spec

Describe fetching source maps in terms of the fetch API

21 of 58

Open question in spec

  • Source map URL algorithm
    • All generators add the sourceMappingURL comment, mostly to the last line of the file
    • Finding JavaScript comments with RegExp is hard
    • Parsing large generated files is slow
    • We’d like to publish an either / or approach

22 of 58

New Proposals

  1. Scopes
  2. Range mappings
  3. Debug ID

23 of 58

Anatomy of a source map

24 of 58

Mappings

  • The mappings data is broken down as follows:
    • each group representing a line in the generated file is separated by a semicolon (;)
    • each segment is separated by a comma (,)
    • each segment is made up of 1, 4, or 5 variable length fields.

25 of 58

Explainer of VLQ

  1. Generated file, starting column
  2. (optional) index into the sources array
  3. (optional) source file, starting line
  4. (optional) source file, starting column
  5. (optional) index into the names array

​

(lines in the generated file are represented by semicolons)

26 of 58

27 of 58

Scopes

​

28 of 58

29 of 58

30 of 58

It’s not just names

31 of 58

Scopes Proposal goals

  • Inline Functions: Reconstruct and step through inlined functions.
  • Variable Mapping: Map renamed/erased variables back to original names.
  • Scope Reconstruction: Rebuild original and hidden scopes.
  • New Fields:
    • originalScopes: Describes original code scopes.
    • generatedRanges: Describes generated code scopes and bindings.

32 of 58

Scopes proposal

33 of 58

Scopes proposal

34 of 58

Scopes proposal

35 of 58

Range Mappings

https://github.com/tc39/source-map/blob/main/proposals/range-mappings.md

36 of 58

Debug IDs

  • Multiple source maps
  • Outdated source maps
  • Stack traces

37 of 58

​

Source Maps

Spec Testing

​

38 of 58

39 of 58

​

Testing Generators

​

40 of 58

​

Testing Generators

​

same source code

same result

41 of 58

​

Testing Generators

​

same source code

different source map

42 of 58

​

Testing Generators

​

43 of 58

​

Testing Generators

​

44 of 58

​

Testing Generators

​

Source map validator can validate:

​

  • format - ensures the source map is in the correct format

​

  • source files -checks that all referenced source files exist and are accessible

​

  • mapping - verifies that mappings accurately point to the correct locations in the source files

​

45 of 58

​

Testing Generators

​

Source map validator:

https://github.com/jkup/source-map-validator

​

npx [todoToolName] --sourceMap sm.js.map --generatedFile g.js --originalFolder src/

46 of 58

​

Testing Consumers

47 of 58

​

Testing Consumers

Source map tests:

​

​

  • test checklist for desired cases

​

  • test data and test specification

​

  • test harness for: Chrome, Firefox, WebKit, source-map library (Firefox)

​

  • tests implementations in Chrome, Firefox, Webkit

​

48 of 58

​

Testing Consumers

Test checklist is very extensive. Available in source-map-tests repo

​

​

49 of 58

​

Testing Consumers

Test specification: JSON format�

{

"tests": [

{

"name": "testCaseNameForHarnessUse",

"description": "Descriptive text for reader",

"baseFile": "generated-source-file.js",

"sourceMapFile": "the-source-map.js.map",

"sourceMapIsValid": true

"testActions": [ … ]

}

]

}

​

​

50 of 58

​

Testing Consumers

Test specification: test actions check more detail cases e.g. mappings

​

"testActions": [

{

"actionType": "checkMapping",

"generatedLine": 10,

"generatedColumn": 13,

"originalSource": "orig.js",

"originalLine": 0,

"originalColumn": 1,

"mappedName": "foobar"

}

]

​

​

​

51 of 58

​

Testing Consumers

There are several types of actions:

  • check mappings,
  • check transitive mappings
  • check ignore lists

​

​

And more will be added!

​

​

​

52 of 58

​

Testing Consumers

What does valid mean?

​

Problem:

  • test cases strictly follow the spec
  • in practice implementations don't reject many invalid source maps (and that's ok)

​

Solution:

  • specify different kinds of validity strictness in test
  • or make the spec more precise about what cases must error, and where it can be loosely checked (to be closer to implementations)

​

Example: most browsers don't check version field, or check it loosely (string "3" is accepted instead of 3).

​

​

​

​

​

53 of 58

​

Testing Consumers

Tests revealed that more precise spec is needed.

​

1. VLQs: how should the 32-bit limit be interpreted? https://github.com/tc39/source-map/issues/80

​

  • is source map exceeding the limit considered invalid?
  • should the value be clamped to 32 bits?
  • what about the sign bit, should it be included in the limit?

​

​

​

​

​

54 of 58

​

Testing Consumers

Tests revealed that more precise spec is needed

​

2. Index maps: spell out how implementation knows if it’s an index map

https://github.com/tc39/source-map/issues/81

​

​

  • Precise spec of how sections should be ordered & not overlap�

​

​

  • Explicitly spell out which fields are mandatory (e.g., names field)
  • Allowed to overlap with regular maps or error in case of "mappings" key?

​

​

​

​

​

​

​

​

​

​

55 of 58

​

Testing Consumers

Tests revealed that more precise spec is needed.

​

3. Many more:

​

  • Specify types of fields (e.g., file, sourceRoot must be string, etc)
  • Zero-field segment validity (implementations allow ",," mapping)
  • Sources field semantics are unclear, especially for null

​

​

​

​

​

​

​

56 of 58

​

Testing Consumers

For future proposals:

​

  • tests for proposals could be in separate JSON files with more cases
  • proposals likely need new test actions (e.g., "checkScope")

​

​

​

​

​

57 of 58

​

Testing Consumers

Consumers source map spec tests:

https://github.com/tc39/source-map-tests

58 of 58

​

Further Testing

Next steps:�C - consumers

G - generators

​

  • (C) add harnesses into consumers codebases (in progress)

​

  • (C) integrate the source map tests repo as a submodule for consumers (in progress)

​

  • (C) add more tests implementations in consumers

​

  • (G) better validation coverage for generators

​

  • (C + G) more tests encompassing the entire Source Map lifecycle (end to end tests)

​

​

​

​

​