Generating Kotlin Code for Better Refactorings, Tests, and IDE Support

Eugene Petrenko

@jonnyzzz

JetBrains

About me

PhD in Computer Science

Using Kotlin from beginning, Java for more than 13 years, C/C++, Go, C#

At JetBrains (ReSharper, TeamCity, Upsource, Toolbox App and many more)

Open source & DSLs

Blogging at http://jonnyzzz.com

C++, .NET, GO

Github

Problem

There is a language

unsupported language

We need an IDE support for it

An IDE is so much more than an editor.

It helps developers

Need the support!

Agenda

Adding language support for an IDE

No IDE code required / The DSL Way

Build a language support

Create a Package

A Live Demo

The Unsupported Language

cannot change the language

otherwise change to any supported language or DSL

existing code is still there

maintaining the code

Time: 3.00

not general purpose language

What is an Unsupported Language?

Domain specific

Business rules / Workflows

Configuration management

Container based (e.g. XML or JSON)

Your example?

Logger settings
Continuous Integration settings
Cloud / Devops tools (Ansible, Teraform, Packer, e.g.
Moonscript for Lua
Metrics reporting setup

container based
JSON, XML, YML

Writing an IDE Plugin

Have someone wrote a plugin for IDE? Which IDE?

Creating an IDE support as a Plugin

Learn / use plugin API

Implement fast & smart parser / reuse higher level IDE APIs

Semantics Checks / Navigation / Refactoring / Code Completion / ...

Deploy plugin / Make it installed / Make people use it / Updates

Cannot extract easily

Incorrect input

Test plugin

An IDE Plugin

Can’t make text more readable

Tricky IDE API to use / Learning Curve

Hard dependency on IDE & GUI

Eternally maintain the plugin for future IDE versions

API not deprecated

The Alternative

Time: 7.30

The DSL Way

Alternative: The DSL Way

Do we have a well-supported language in our IDE?

Java, Scala, Kotlin

Time 8.30

What IDE do you use?

Alternative: The DSL Way

Use already supported by an IDE language

Need back-and-forth transformation between languages

No IDE-related code is required here at all

Use tooling for well-supported language

Multi IDEs

Alternative: The DSL Way

Make generated code better looking in IDE?

Use refactorings / meta-extensions / existing tools

No IDE-related code is required here at all

Use tooling for well-supported language

prepare code for IDE

use IDE with supported lang

- convert code back

Definitions

Original Language
Language to be supported using in an IDE

Target Language
A Language that is supported by an IDE

Generate / Generator
Transformation from Original to Target Language

Emit / Execute
Transformation, compile, run Target to Original Language

Workflow

Original Language

generate code in Target Language

refactor, compile, test

emit Original Language

Common Code

Extract common code to a LIB
call it a DSL-API or DSL

Generate only input-dependent code

Allow extensibility

Simplify implementation

The DSL Way

Time: ~14.30

The code is compiled

OriginalLanguage =?= OriginalLanguage

What Target Language to use

Any language will work

If supported by the IDE good enough

Readability / Tooling are nice to have / Fluent & DSLs

Static typed language

Time 15.00

The DSL Way vs IDE Plugin

Time 16.40

IDE vs DSL: Parsing

Parser is required in both cases

IDE
fast & a smart parser for IDE with error recovery or IDE APIs
support invalid input

DSL
just a library for reading a valid code
can be slower
valid code always

IDE vs DSL: Dependencies

Dependencies

DSL
fewer dependencies on components, can fix
easy to reuse code

IDE
check compatibility updates
GUI dependencies

Time 22.00

IDE vs DSL: Readability

DSL
changes the representation forced
can replace it

IDE
cannot change a language

IDE vs DSL: Costs

Different

tricky parts

support

The DSL Way. Example

Time: ~20.00

Logging Configuration

Let’s talk about Apache Log4j

.properties

Any other logging framework and it’s configuration files too

Same for XML

Goal

step-by-step create a DSL-API

IDE Support

Readability / Refactorings / code reuse

Time 24.00

We use Kotlin & IntelliJ IDEA

Open source static typed language by JetBrains

Easy to learn & use

Great IDE support

Easy to write static typed DSLs

More on kotlinlang.org

IDE understands types in DSLs

Log4j Configurations

In most cases cannot change the format

But nice to be able to refactor or read it easily

Time: 19.30

Configuration in .properties

log4j.rootLogger=ERROR,stdout
log4j.logger.corp.mega=INFO
# meaningful comment goes here
log4j.logger.corp.mega.itl.web.metrics=INFO
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%p\t%d{ISO8601}\t%r\t%c\t[%t]\t%m%n

Step 0. Straightforward DSL

Line by line transformation

.properties file container replacement

Define ‘log4j’, ‘param’, ‘comment’ DSL API methods

Step 0. Straightforward generator

Design DSL API library

Implement generator

Implement emitter

= > Let’s focus on DSL API

DSL Generation Stages

The code is compiled to emit

Step 0. Basic DSL in Kotlin

log4j {
param("log4j.rootLogger", "ERROR,stdout")
param(
"log4j.logger.corp.mega", "INFO")
comment(
"meaningful comment goes here")
param("log4j.logger.corp.mega.itl.web.metrics", "INFO")
param(
"log4j.appender.stdout", "org.apache.log4j.ConsoleAppender")
param(
"log4j.appender.stdout.layout", "org.apache.log4j.PatternLayout")
param(
"log4j.appender.stdout.layout.ConversionPattern", "%p\t%d{ISO8601}\t%r\t%c\t[%t]\t%m%n")
}

Step 0. DSL Library Code

interface Log4JBase {
fun comment(text : String)
fun param(name: String, value : String)
}
interface Log4J : Log4JBase

Step 0. DSL Entry Point

interface Log4J : …


fun log4j(builder : Log4J.() -> Unit)

Introduction to Kotlin DSLs

Time: ~25.00

Creating Small DSLs with Idiomatic Kotlin [CON1242]

Hadi Hariri, Developer, JetBrains

Wednesday, Sep 21, 10:00 a.m. - 11:00 a.m. | Hilton - Plaza Room A

Kotlin Functions

fun param(name: String, value : String) : Int

fun comment(text : String)

Time 24.00

Kotlin Properties

Properties are syntax for java getters and setters.
val readOnly : String
var simple : String
var custom : String
get() { ... }
set(t : String) { ... }

Kotlin Extension Functions

fun String.parenthesize() : String {
return "(" + this + ")" // or "($this)"
}


//usage
"some text"
.parenthesize()

DSLs in Kotlin: Extension Properties

Declare extension properties this way

val String.bytesSize : Int
get() = toByteArray(Charsets.UTF_8).size

//usage

"A string object".bytesSize

DSLs in Kotlin: Higher order Functions

Function type

(String, Long) -> Int

//example

fun f(a: String, b: Long) : Int

DSLs in Kotlin: Higher order Functions

Extension Function type

Log4j.() -> Unit

//example

fun Log4J.x() // or :Unit

Reciever

Anonymous Functions

Last lambda parameter allowed outside braces

x ( { println("string") } )

fun x(b: () -> Unit) {
//just call a function
b()
}


Anonymous Functions

Last extension function lambda parameter allowed outside braces

x { this.parenthesize() }

fun x(b: String.() -> Unit) {
//just call an extension-function
"string".b()
}


Log4j Example. Continued

Time ~27.35

Goal

step-by-step create a DSL-API

IDE Support

Readability / Refactorings / code reuse

Step 0. ‘log4j’ Function

A builder is executed having Log4J as ‘this’
Last lambda parameter allowed outside braces

fun log4j(builder : Log4J.() -> Unit)

log4j ( {
this.param("log4j.rootLogger", "...")
} )

Step 0. DSL Library Code

interface Log4JBase {
fun comment(text : String)
fun param(name: String, value : String)
}
interface Log4J : Log4JBase

Step 0. Basic DSL in Kotlin

log4j {
param("log4j.rootLogger", "ERROR,stdout")
param(
"log4j.logger.corp.mega", "INFO")
comment(
"meaningful comment goes here")
param(
"log4j.logger.corp.mega.itl.web.metrics", "INFO")
param(
"log4j.appender.stdout", "org.apache.log4j.ConsoleAppender")
param(
"log4j.appender.stdout.layout", "org.apache.log4j.PatternLayout")
param(
"log4j.appender.stdout.layout.ConversionPattern", "%p\t%d{ISO8601}\t%r\t%c\t[%t]\t%m%n")
}

Step 0. Benefits from DSL & IDE support

Full .properties escaping support

Easy code templating / conditions
Extract function / Extract common templates


Not providing
semantics
object model

Time 31.00

=> Let’s improve the DSL to meet the goal

Step 1. Add initial helpers

Replace well known properties with extension functions

"log4j.rootLogger"

var Log4J.rootLogger : String
set(l : String) = param("log4j.rootLogger", l)
get() = throw Error()

Time 30.30

Step 1. RootLogger simplified

log4j {
//use
rootLogger = "stdout"

//instead of
param("log4j.rootLogger", "stdout")
}

Step 2. Builder for Appenders

Here goes so many connected properties and shared constants

Let’s have builder to simplify that

param("log4j.appender.stdout", "org.apache.log4j.ConsoleAppender")
param(
"log4j.appender.stdout.layout", "org.apache.log4j.PatternLayout")
param(
"log4j.appender.stdout.layout.ConversionPattern", "%p\t%d{ISO8601}\t%r\t%c\t[%t]\t%m%n")

Step 2. Builder for Appenders

fun Log4J.appender(name : String, type : String,
builder : Log4JAppender.() -> Unit)

interface Log4JAppender : Log4JBase {
fun layout(type: String,
builder : Log4JLayout.() -> Unit)
}

Step 2. Outcome (old)

//instead of
param("log4j.appender.stdout", "org.apache.log4j.ConsoleAppender")
param(
"log4j.appender.stdout.layout", "org.apache.log4j.PatternLayout")
param(
"log4j.appender.stdout.layout.ConversionPattern",

Step 2. Usage of Appender Builder

//use this
appender("stdout", "org.apache.log4j.ConsoleAppender") {
layout("org.apache.log4j.PatternLayout") {
param("ConversionPattern", "%p\t%d{ISO8601}\t%r\t%c\t[%t]\t%m%n")
}
}

Step 3. Builder for Logger

Let’s simplify those lines too

log4j {
rootLogger = "ERROR,stdout"
param("log4j.logger.corp.mega", "INFO")
param(
"log4j.additivity.corp.mega", "false")
}

Step 3. Logger Builder APIs

Define a builder for logger settings

interface Log4JLogger : Log4JBase {
var additivity : Boolean?
var level : Log4JLevel?
var appenders : List<String>
}

Step 3. Logger Extension Functions

fun Log4J.logger(category: String,
builder : Log4JLogger.() -> Unit)

fun Log4J.rootLogger(
builder : Log4JLogger.() -> Unit)

Step 3. Improved DSL

rootLogger {
level = ERROR
appenders
+= "stdout"
}
logger("corp.mega") {
additivity = false
level = INFO
}

Step 1 - 3. Outcome

IDE Support is implemented

Better readability / no skills needed

Harder to make a syntax error

Semantic & Object model implemented

Time: 33.00

IDE support!

Step 4. Tracking Appender Usages

Implementing IDE features

Find usages

Rename

Time 35.00

Step 4. Tracking Appender Usages

A feature projection

Let’s find similar Kotlin IDE feature

=> Kotlin variable declaration looks good

Step 4. Tracking Appender Usages

introduce Log4JAppenderRef interface

appender() function returns Log4JAppenderRef
make generator generate variables for appenders

appenders type List<Log4JAppenderRef>
Make generator use variables, not strings

Step 4. Appender variable variable

val stdout = appender("stdout", ...

appenders += stdout

Type of stdout is Log4JAppenderRef.

A type-checked call to add item into collection (+=)

Step 4. Benefits from DSL & IDE Support

IDE knows about Appender usages
Find Usages
Rename

Semantics
Only declared Appender can be used
Duplicate appenders/loggers are rejected

Steps. The DSL for Log4j

log4j {
val stdout = appender<ConsoleAppender>("stdout") {…}
rootLogger {
appenders += stdout
level = ERROR
}
logger("corp.mega.itl.web.metrics") { … }
}

Steps. Outcome

Created a Log4J configuration DSL

Helps to read / refactor / change / reuse Log4J .properties files

Support in IDE with code-completion and type inference

Semantic checks and Errors prevention included

Time 37.00

The DSL Way: Benefits

Time 39:00

Side Effect 1. Automatic Tests

DSL-API is a domain model

Let’s use domain model in tests

Small modifications are required to access build model

Time 39.00

Side Effect 1. Automatic Tests. API

interface Log4JModel {
val appenders : List<Log4JAppender>
val loggers : List<Log4JLogger>
val rootAppender : Log4JAppender
}

Side Effect 1. JUnit Integration

Kotlin / IntelliJ / Gradle supports JUnit

All we need is an API to load model from tests

class Log4JConfigTest {
@Test
fun categories_should_have_appender() { … }
}

Tests can use generated code directly too

Side Effect 2. Continuous Integration

No IDE dependency

Can simply run tests as a part of CI build

Combine generated sources with test sources

Build: generate DSL code, code & tests, run tests

DSL, generators, APIs

Side Effect 3. Supporting XML configurations

We may implement similar generators for XML configurations too

DSL-API can be reused as well as code-generation part

=> We have full-featured .properties to XML converter

Side Effect 4. Power of DSL

A semantic aware templating engine

flexible & faster development with loops, conditions, etc
generator may generate

better validation & semantic checks

extensibility with shared libraries

Time 41.00

Unlike string templates

Let’s take a look some details

Side Effect 4. Power of DSL: Logger configuration

log4j {
logger("category2warn") {
+ WARN
}
}

Side Effect 4. Power of DSL: Use forEach

log4j {
listOf("A", "B", "C").forEach {
logger("category.$it") {
+ WARN
}
}
}

Side Effect 4. Power of DSL: Use forEach 2

log4j {
listAllRootPackages().forEach {
logger("category.$it") {
+ WARN
}
}
}

Side Effect 4. Power of DSL: Library Extensibility

Extract common part

log4j {
listAllRootPackages().forEach {
warnCategory(it)
}
}

Side Effect 4. Power of DSL. Shared Library

//library
fun Log4jBuilder.warnCategory(n: String) {
logger(
"category.$n") {
+WARN
}
}

Log4j Example. Outcome

A simple, but feature-complete example

Benefits / IntelliJ IDEA support / Find Usages and Renames

Source code: https://github.com/jonnyzzz/Log4j2DSL

Designing DSLs

Time 44.00

Inventing a DSL

Know language features. Deeply

Model domain as a set of function calls & builders

Improvement Loop

Time 44.00

Inventing DSL: Kotlin

Creating Small DSLs with Idiomatic Kotlin [CON1242]

Hadi Hariri, Developer, JetBrains

Wednesday, Sep 21, 10:00 a.m. - 11:00 a.m. | Hilton - Plaza Room A

Packing things

Packing Things

Easy-to use and apply package is required

Updates should be easy

IDE support should just work with zero config

=> How one can ship it?

Time 46.00

Packing via a Build System

Requirements
Dependencies Resolution
Compiler setup
Kotlin setup
Tasks execution

One-click IDE support

Simple installation & upgrade

Gradle

Gradle plays well with
IntelliJ IDEA
Kotlin

Allows to implement an zero-configuration package

=> Let’s discuss it in example

Time 46.00

A story about Ant

The Gradle package

Setup compiler and dependencies

Tasks ‘toDSL’ and ‘fromDSL’

Bring IDE support

Initial Gradle Build

buildscript {
repositories {
mavenCentral()
}
dependencies {
classpath
"...:$kotlin_version"
}
}
apply
plugin: 'java'
apply plugin: 'kotlin'

repositories {
/// DSL apis, Generator apis, kotlin
}

dependencies {
compile
"...$DSL_VERSION"
}

task toDSL { /*...javaExec…*/ }
task fromDSL { /*...javaExec…*/ }

Time 51.00

Use Gradle plugin

Gradle plugin syntax
only 3 lines in build.gradle

Gradle Plugin repository

Move all details into plugin code

Example: Enable a Plugin in Gradle

//in build.gradle

plugins {
id
"ID" version "VER"
}

Open in IDE

Our Gradle plugin uses standard extensibility

It works in IntelliJ IDEA as is

Code is greed, refactorings are available

Tasks are detected

Example. Package

This was implemented in TeamCity2DSL project
A refactoring tool for TeamCity 8.0 - 10.0 XML configurations

A real-working The DSL Way implementation

Sources: https://github.com/jonnyzzz/TeamCity2DSL

Starting from TeamCity 10.0 it has embedded support of DSL build configurations

Time: 49.00

TeamCity in Action

JetBrains TeamCity

Powerful continuous integration server by JetBrains

Configuration objects:
Project
Build Configuration
Build Configuration Template
Meta-Runner
VCS Root (Version Control)

Project settings are in XML, available through Version Control

Why The DSL Way for Continuous Integration?

A DSL allows to use richer (Kotlin) language instead of XML

TeamCity concepts are represented explicitly
Project, VCS Root, Build Configuration, etc

Basic semantic checks of XML data is implemented
Smarter that XSD/DTD
Reports domain-related errors

Powerful TeamCity Web UI

Toolset in Gradle: Done

//root build.gradle file

plugins {
id
"org.jonnyzzz.teamcity.dsl" version "0.1.15"
}

Side Effect 5: Automatic Tests in Gradle

Once Kotlin and Gradle are used

Tests are available out-of-the box

IDE support is also included automatically

add dependency on junit & code

Side Effect 6: Extensibility support

TeamCity2DSL supports extensions via .jars
e.g. custom DSL for a build runner

We need the way to support extensibility in Gradle Plugin

=> Use Gradle’s DSL extension mechanism.

Side Effect 6: Extending DSL & Generator

plugins {
id
"org.jonnyzzz.teamcity.dsl" version "0.1.15"
}

teamcity2dsl {
extension
'...:commandline-runner:0.1.15'
}

Create an extension for Gradle DSL

Live Demo

Let’s have a look in IntelliJ on TeamCity project configuration example

Start from scratch

Adding an extension

Adding unit tests

Time: 53.30

Show the build runner

Live Demo

Package. Outline

It’s possible to build self-contained toolset as easy as

plugins {
id
"PLUGIN" version "VERSION"
}

With IDE supported, easy to use, easy to upgrade, extensible

Implementation details are fully abstracted in plugin implementation

Time 59.00

Outcome

The DSL Way toolset with IDE support is doable
with IntelliJ IDEA as IDE
with JetBrains Kotlin for DSLs
with three-lines in Gradle

Build your own toolset!


JetBrains TeamCity 10.0 will have it’s own DSL support
JetBrains MPS is a DSL languages workbench

The DSL Way

Summary

Need an IDE support for a language?

The DSL Way is also possible!

No IDE dependency required

Questions?

Creating Small DSLs with Idiomatic Kotlin

[CON1242]

Hadi Hariri, Developer, JetBrains

Wednesday, Sep 21, 10:00 a.m. - 11:00 a.m. | Hilton - Plaza Room A

TOO MANY DETAILS!

Gradle plugin case

Easy to use / upgrade

No IDE configuration required

Test runners from java plugin


=> A self-contained toolset that can be used to develop DSL code

Gradle plugin code

Ok to use Kotlin (not Groovy / Java)

Given a Gradle Project instance

Have similar code as a plugin

class TeamCity2DSLPlugin : Plugin<Project> {
override fun apply(base : Project) {

Trick: Implicit Kotlin Setup

From the plugin we know Kotlin version/dependencies that are required

=> declare dependency from Kotlin in plugin’s pom
apply plugins from plugin

project.apply { config ->
config.plugin("java")
config.plugin(
"kotlin")
}

Initial Gradle Build for Toolset

buildscript {
repositories {
mavenCentral()
}
dependencies {
classpath
"org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
}
}
apply
plugin: 'java'
apply plugin: 'kotlin'

repositories {
/// same again
}

dependencies {
compile
"...$DSL_VERSION"
}

task xml2dsl { /*...javaExec…*/ }
task dsl2xml { /*...javaExec…*/ }

Trick: Downloading Dependencies

Publish to a maven repo

Generate constants from the plugin build

Same dependency names & versions (not hardcoded)

Same components & plugins

In plugin code:

Include generated list of dependencies & repositories

Initial Gradle Build for Toolset

buildscript {
repositories {
mavenCentral()
}
dependencies {
classpath
"org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
}
}
apply
plugin: 'java'
apply plugin: 'kotlin'

repositories {
/// same again
}

dependencies {
compile
"...$DSL_VERSION"
}

task xml2dsl { /*...javaExec…*/ }
task dsl2xml { /*...javaExec…*/ }

TeamCity2DSL

My goal was to learn Kotlin & practice in DSL building

I was looking for proper tooling

Sources:
https://github.com/jonnyzzz/TeamCity2DSL
Note: TeamCity 10 has DSL support out of the box

Step 2. Outcome

Remove common / repeating string

Improve readability

Avoid some sort of typos

Few semantic checks

Light object model

Step 3. Forward declaration for constants trick

level = ERROR

The ERROR should be redeclared to use without Log4JLevel prefix

val Log4J.ERROR : Log4JLevel
get() = Log4JLevel.ERROR

Step 3. Alternative for wide screens

Via named parameters in Kotlin with default values

rootLogger(level = ERROR, appeners = listOf(stdout))

logger("corp.mega", additivity = false, level = INFO)

Declaration

fun Log4J.logger(category: String, additivity : Boolean = true,
level : Log4JLevel? =
null,
appeners : List<Log4jAppenderRef>? =
null)

Step 3. Alternative. Hacker Style

Via overloaded operators

logger("corp.mega.d2", level = INFO) >= stdout

DSL design is a matter of taste

Step 5. Use declared classes

References to Log4J classes are string constants

Let’s replace it with Class objects

log4j {
appender("stdout", "org.apache.log4j.ConsoleAppender") {
layout("org.apache.log4j.PatternLayout") {
param("ConversionPattern", "...")
}
}

Step 5. Use declared classes

fun <T : Appender> Log4J.appender(
name : String,
type :
KClass<T>,
builder : Log4JAppender.() -> Unit) ...

Step 5. Use declared classes

log4j {
appender("stdout", ConsoleAppender::class) {
layout(PatternLayout::class) {
param("ConversionPattern", "%p\t%d{ISO8601}\t%...")
}
}
}

Step 6*. Use Log4J Objects Directly

layout<PatternLayout> {
// this is a call to PatternLayout#setConversionPattern
conversionPattern = "%p\t%d{ISO8601}\t..."
}

Via reified generics and inline functions in Kotlin

inline fun <reified T : Layout> layout(l : T.() -> Unit) {
val type = T::class // allowed for reified generics

Target Language Examples

A language that is supported by an IDE
Kotlin, Scala, Groovy …

Create a DSL API library, generator, emitter

Get IDE support & refactorings

Create unit tests

Have semantics checks

Package: Use one build tool

No mix of tools
e.g. Ant + Maven

No duplication of settings
e.g. IntelliJ project + Ant

No tricky files or templates

No complex documentation

Trick: Update configuration from Extension

Update Project instance once DSL extension method is called

Pass Project into a DSL extension class

Alternative

update Project in afterEvaluate. May not work

Trick: Running TeamCity2DSL

Obvious plan:
Declare everything onto buildscript classpath
Run classes as-is

Problem: classpath is shared. Can have conflicts
It did not work with jdom which conflicted with Gradle internals

Solution:
Load classes explicitly
Avoid any buildscript classpath level dependencies

Testing the plugin code itself

All tests are integration tests
Running on a gradle script sample

Local maven deployment is used for such tests
A dedicated step is created to publish all dependencies
Local maven added into repositories

Base project is used to setup tests

There are a number of tests for the tool itself, without Gradle

Initial Solution

Implement tasks in Ant (required locally)

Download dependencies (Kotlin), call compiler, and everything is done with Ant
Partly duplicated in IntelliJ settings

Hard to update to newer version

No IDE project included (so a template project was introduced)

Complicated startup

No extensibility supported

TOO COMPLEX!

JavaOne 2016. Slides. TeamCity2DSL - Google Slides