1 of 42

Teaching your editor a new programming language

NICOLÒ RIBAUDO

@nicolo-ribaudo

@NicoloRibaudo

https://nicr.dev

@NicoloRibaudo

2 of 42

Me

2

This is how I look online

These are things I do you might be using

This is where I work

@NicoloRibaudo

3 of 42

Why?

3

@NicoloRibaudo

4 of 42

Why?

You are creating your own programming language

Somebody else created a programming language, and your editor does not support it

Your editor poorly supports your usage of a given programming language

4

@NicoloRibaudo

5 of 42

Why?

​

Somebody else is creating a new programming language, you think it's cool, and you want to help more people use it

5

MessageFormat 2.0

@NicoloRibaudo

6 of 42

MessageFormat 2.0

​

MessageFormat 2.0 is a Unicode standard for localizable dynamic message strings, designed to make it simple to create natural sounding localized messages.

— https://messageformat.dev/

6

@NicoloRibaudo

7 of 42

MessageFormat 2.0

const message = new Intl.MessageFormat("en-US", `

​

`);

message.format({ name: "Luca" });

7

Hello, {$name}!

@NicoloRibaudo

8 of 42

MessageFormat 2.0

const message = new Intl.MessageFormat("en-US", `

​

`);

const message = new Intl.MessageFormat("fr-FR", `

​

`);

message.format({ name: "Luca" });

8

Hello, {$name}!

Salut, {$name}!

@NicoloRibaudo

9 of 42

MessageFormat 2.0

9

Today is {$today :date}.

Aujourd'hui c'est le {$today :date}.

"en-US"

"fr-FR"

"today": "2024-11-15"

Today is Nov 15, 2024.

Aujourd'hui c'est le 15 nov. 2024.

@NicoloRibaudo

10 of 42

MessageFormat 2.0

10

.input {$today :date}

{{Today is {$today}.}}

"en-US"

"fr-FR"

.input {$today :date}

{{Aujourd'hui c'est le {$today}.}}

@NicoloRibaudo

11 of 42

MessageFormat 2.0

11

.match $genderMarker

M {{He is {$name}}}

F {{She is {$name}}}

* {{They are {$name}}}

"en-US"

.match $genderMarker

M {{Il est {$name}}}

F {{Elle est {$name}}}

* {{Iel est {$name}}}

"fr-FR"

@NicoloRibaudo

12 of 42

MessageFormat 2.0

12

.match $genderMarker

M {{He is {$name}}}

F {{She is {$name}}}

* {{They are {$name}}}

"en-US"

"name": "Luca",

"genderMarker": "M"

He is Luca

They are Luca

"name": "Luca",

"genderMarker": "X"

@NicoloRibaudo

13 of 42

MessageFormat 2.0

Check https://messageformat.dev/�to learn more about it!

13

@NicoloRibaudo

14 of 42

Let's get started!

14

@NicoloRibaudo

15 of 42

15

@NicoloRibaudo

16 of 42

Syntax Highlighting

16

@NicoloRibaudo

17 of 42

17

mf2.tmLanguage.json

  • TextMate
  • VS Code
  • Visual Studio
  • SublimeText
  • WebStorm / IntelliJ
  • Neovim (nvim-textmate)
  • …

@NicoloRibaudo

18 of 42

TextMate grammars

  • Based on regular expressions
  • Run line-by-line on the source code
  • Mark parts of the code as specific syntax (e.g. variables)
  • The editor applies its theming to the syntax parts

18

@NicoloRibaudo

19 of 42

TextMate grammars

19

{

"name": "MessageFormat 2",

"patterns": [{

"name": "variable.other.mf2",

"match": "\\$[a-zA-Z_][a-zA-Z0-9_]*"

}]

}

Anything that matches the regular expression /\$[a-zA-Z_][a-zA-Z0-9_]*/ is in the category "variable".

@NicoloRibaudo

20 of 42

TextMate grammars

20

"patterns": [{

"name": "placeholder",

"begin": "\\{",

"end": "\\}",

"patterns": [{

"name": "variable.other.mf2",

"match": "\\$[a-zA-Z_][a-zA-Z0-9_]*"

}]

}]

… anything that matches the regular expression /\$[a-zA-Z_][a-zA-Z0-9_]*/ is in the category "variable".

Inside a section that starts with /\{/ and ends with /\}/…

Today $notVar is {$var}.

@NicoloRibaudo

21 of 42

TM grammars

​

We can have a variable both inside a placeholder, and inside a .input declaration.

Grammars can get … verbose.

21

{

"name": "MessageFormat 2",

"patterns": [{

"name": "placeholder",

"begin": "\\{",

"end": "\\}",

"patterns": [{ "include": "#variable" }]

}],

"repository": {

"variable": {

"name": "variable.other.mf2",

"match": "\\$[a-zA-Z_]…

}

}

}

Reference

@NicoloRibaudo

22 of 42

TextMate grammars

​

Tip! Grammars can get verbose: generate them!

​

lucacasonato/mf2-tools - tools/vscode/grammar.ts

lucacasonato/mf2-tools - vscode/syntaxes/mf2.tmLanguage.json

​

​

22

@NicoloRibaudo

23 of 42

23

@NicoloRibaudo

24 of 42

Marking the editor understand our code

24

@NicoloRibaudo

25 of 42

25

@NicoloRibaudo

26 of 42

What can editors do?

  • Auto completions
  • Variable renaming
  • Linting
  • Code fixes
  • Formatting
  • …

26

All of this is very interactive, we cannot just define it with a declarative JSON file!

@NicoloRibaudo

27 of 42

27

We can create an extension for !

VS Code

Sublime Text

WebStorm

Vim

Eclipse

@NicoloRibaudo

28 of 42

The�Language Server Protocol

28

@NicoloRibaudo

29 of 42

The Language Server Protocol

It's kind of like the web you have a single server, and multiple clients can talk with it using a shared language (HTTP).

29

@NicoloRibaudo

30 of 42

The Language Server Protocol

30

Image from https://code.visualstudio.com/api/language-extensions/language-server-extension-guide

@NicoloRibaudo

31 of 42

The Language Server Protocol

​

31

The MessageFormat language server

Hey, the user opened this file.

The user is typing a variable, do you have any completions?

Yes: $name and $lastName

Hey, the user typed …

Hey, the user typed …

Hey, I noticed some errors

@NicoloRibaudo

32 of 42

The Language Server Protocol

​

32

The MessageFormat language server

  • VS Code
  • Visual Studio
  • SublimeText
  • Neovim
  • Eclipse
  • Webstorm
  • Emacs
  • …

@NicoloRibaudo

33 of 42

The Language Server Protocol

What features does it support?

microsoft.github.io/language-server-protocol

33

@NicoloRibaudo

34 of 42

The Language Server Protocol

Like for HTTP, you don't actually have to implement the protocol by yourself. There are libraries that abstract it away and let you focus on the interesting parts.

  • JavaScript: vscode-languageserver
  • Rust: lsp-server

34

@NicoloRibaudo

35 of 42

The interesting parts

(or "The ingredients")

35

@NicoloRibaudo

36 of 42

Parser

  • Manipulating and analyzing a string containing source code is hard
  • A parser converts a string in an "Abstract Syntax Tree" representing the program

36

Salut, {$name :str}!

Message

Text: "Salut, "

Placeholder

Text: "!"

Variable: $name

Annotation: :str

@NicoloRibaudo

37 of 42

Parser

  • You are probably going to find an existing parser for the language you are working with
  • That parser is probably not going to work

37

We need to parse incomplete code while the user is typing!

Salut, {$na│

Hello {$na│, welcome!

@NicoloRibaudo

38 of 42

Semantic Analyzer

Whenever the parser generates an AST, it analyses it to find:

38

Auto completions!

  • Which variables are visible at�a given location

Go to definition!

  • Where variables are defined

Rename!

  • Where variables are used

Inlay hints!

  • What type variables have

@NicoloRibaudo

39 of 42

Pretty-printer

Given an AST generated by the parser, it converts it back into a good looking string of code

39

.match $count 0

{{No notifications}}

1{{One notification}}

* {{{$count} notifications}}

.match $count

0 {{No notifications}}

1 {{One notification}}

* {{{ $count } notifications}}

@NicoloRibaudo

40 of 42

… and more

40

@NicoloRibaudo

41 of 42

MessageFormat 2.0 LSP

41

https://github.com/lucacasonato/mf2-tools

@NicoloRibaudo

42 of 42

Go build your own!

42

@NicoloRibaudo