Generating Kotlin Code for Better Refactorings, Tests, and IDE Support
Eugene Petrenko
@jonnyzzz
JetBrains
About me
PhD in Computer Science
Using Kotlin from beginning, Java for more than 13 years, C/C++, Go, C#
At JetBrains (ReSharper, TeamCity, Upsource, Toolbox App and many more)
Open source & DSLs
Blogging at http://jonnyzzz.com
Problem
There is a language
unsupported language
We need an IDE support for it
Agenda
Adding language support for an IDE
No IDE code required / The DSL Way
Build a language support
Create a Package
A Live Demo
The Unsupported Language
cannot change the language
otherwise change to any supported language or DSL
existing code is still there
maintaining the code
What is an Unsupported Language?
Domain specific
Business rules / Workflows
Configuration management
Container based (e.g. XML or JSON)
Your example?
Writing an IDE Plugin
Creating an IDE support as a Plugin
Learn / use plugin API
Implement fast & smart parser / reuse higher level IDE APIs
Semantics Checks / Navigation / Refactoring / Code Completion / ...
Deploy plugin / Make it installed / Make people use it / Updates
An IDE Plugin
Can’t make text more readable
Tricky IDE API to use / Learning Curve
Hard dependency on IDE & GUI
Eternally maintain the plugin for future IDE versions
The Alternative
The DSL Way
Alternative: The DSL Way
Do we have a well-supported language in our IDE?
Java, Scala, Kotlin
Alternative: The DSL Way
Use already supported by an IDE language
Need back-and-forth transformation between languages
No IDE-related code is required here at all
Alternative: The DSL Way
Make generated code better looking in IDE?
Use refactorings / meta-extensions / existing tools
No IDE-related code is required here at all
prepare code for IDE
use IDE with supported lang
- convert code back
Definitions
Original Language� Language to be supported using in an IDE
Target Language� A Language that is supported by an IDE
Generate / Generator� Transformation from Original to Target Language
Emit / Execute� Transformation, compile, run Target to Original Language
Workflow
Original Language
generate code in Target Language
refactor, compile, test
emit Original Language
Common Code
Extract common code to a LIB� call it a DSL-API or DSL
Generate only input-dependent code
Allow extensibility
The DSL Way
What Target Language to use
Any language will work
If supported by the IDE good enough
Readability / Tooling are nice to have / Fluent & DSLs
Static typed language
The DSL Way vs IDE Plugin
IDE vs DSL: Parsing
Parser is required in both cases
IDE� fast & a smart parser for IDE with error recovery or IDE APIs� support invalid input
DSL � just a library for reading a valid code� can be slower� valid code always
IDE vs DSL: Dependencies
Dependencies
DSL� fewer dependencies on components, can fix� easy to reuse code
IDE� check compatibility updates� GUI dependencies
IDE vs DSL: Readability
DSL � changes the representation forced� can replace it
IDE� cannot change a language
IDE vs DSL: Costs
Different
tricky parts
support
The DSL Way. Example
Logging Configuration
Let’s talk about Apache Log4j
.properties
Any other logging framework and it’s configuration files too
Goal
step-by-step create a DSL-API
IDE Support
Readability / Refactorings / code reuse
We use Kotlin & IntelliJ IDEA
Open source static typed language by JetBrains
Easy to learn & use
Great IDE support
Easy to write static typed DSLs
More on kotlinlang.org
Log4j Configurations
In most cases cannot change the format
But nice to be able to refactor or read it easily
Configuration in .properties
log4j.rootLogger=ERROR,stdout�log4j.logger.corp.mega=INFO�# meaningful comment goes here�log4j.logger.corp.mega.itl.web.metrics=INFO�log4j.appender.stdout=org.apache.log4j.ConsoleAppender�log4j.appender.stdout.layout=org.apache.log4j.PatternLayout�log4j.appender.stdout.layout.ConversionPattern=%p\t%d{ISO8601}\t%r\t%c\t[%t]\t%m%n
Step 0. Straightforward DSL
Line by line transformation
.properties file container replacement
Define ‘log4j’, ‘param’, ‘comment’ DSL API methods
Step 0. Straightforward generator
Design DSL API library
Implement generator
Implement emitter
= > Let’s focus on DSL API
DSL Generation Stages
Step 0. Basic DSL in Kotlin
log4j {� param("log4j.rootLogger", "ERROR,stdout")� param("log4j.logger.corp.mega", "INFO")� comment("meaningful comment goes here")� param("log4j.logger.corp.mega.itl.web.metrics", "INFO")� param("log4j.appender.stdout", "org.apache.log4j.ConsoleAppender")� param("log4j.appender.stdout.layout", "org.apache.log4j.PatternLayout")� param("log4j.appender.stdout.layout.ConversionPattern", "%p\t%d{ISO8601}\t%r\t%c\t[%t]\t%m%n")�}
Step 0. DSL Library Code
interface Log4JBase {� fun comment(text : String)� fun param(name: String, value : String)�}�interface Log4J : Log4JBase
Step 0. DSL Entry Point
interface Log4J : …
�fun log4j(builder : Log4J.() -> Unit)
Introduction to Kotlin DSLs
Kotlin Functions
fun param(name: String, value : String) : Int
fun comment(text : String)
Kotlin Properties
Properties are syntax for java getters and setters.� val readOnly : String� var simple : String� var custom : String� get() { ... }� set(t : String) { ... }�
Kotlin Extension Functions
fun String.parenthesize() : String { � return "(" + this + ")" // or "($this)"�}
�//usage�"some text".parenthesize()
DSLs in Kotlin: Extension Properties
Declare extension properties this way
val String.bytesSize : Int� get() = toByteArray(Charsets.UTF_8).size
//usage
"A string object".bytesSize
DSLs in Kotlin: Higher order Functions
Function type
(String, Long) -> Int
//example
fun f(a: String, b: Long) : Int
DSLs in Kotlin: Higher order Functions
Extension Function type
Log4j.() -> Unit
//example
fun Log4J.x() // or :Unit
Anonymous Functions
Last lambda parameter allowed outside braces
x ( { println("string") } )
fun x(b: () -> Unit) {� //just call a function� b()�}
�
Anonymous Functions
Last extension function lambda parameter allowed outside braces
x { this.parenthesize() }
fun x(b: String.() -> Unit) {� //just call an extension-function� "string".b()�}
�
Log4j Example. Continued
Goal
step-by-step create a DSL-API
IDE Support
Readability / Refactorings / code reuse
Step 0. ‘log4j’ Function
A builder is executed having Log4J as ‘this’�Last lambda parameter allowed outside braces
fun log4j(builder : Log4J.() -> Unit)
log4j ( {� this.param("log4j.rootLogger", "...")�} ) �
Step 0. DSL Library Code
interface Log4JBase {� fun comment(text : String)� fun param(name: String, value : String)�}�interface Log4J : Log4JBase�
Step 0. Basic DSL in Kotlin
log4j {� param("log4j.rootLogger", "ERROR,stdout")� param("log4j.logger.corp.mega", "INFO")� comment("meaningful comment goes here")� param("log4j.logger.corp.mega.itl.web.metrics", "INFO")� param("log4j.appender.stdout", "org.apache.log4j.ConsoleAppender")� param("log4j.appender.stdout.layout", "org.apache.log4j.PatternLayout")� param("log4j.appender.stdout.layout.ConversionPattern", "%p\t%d{ISO8601}\t%r\t%c\t[%t]\t%m%n")�}
Step 0. Benefits from DSL & IDE support
Full .properties escaping support
Easy code templating / conditions� Extract function / Extract common templates
�Not providing� semantics� object model
Step 1. Add initial helpers
Replace well known properties with extension functions
"log4j.rootLogger"
var Log4J.rootLogger : String� set(l : String) = param("log4j.rootLogger", l)� get() = throw Error()
Step 1. RootLogger simplified
log4j {� //use� rootLogger = "stdout"�� //instead of� param("log4j.rootLogger", "stdout")�}
Step 2. Builder for Appenders
Here goes so many connected properties and shared constants
Let’s have builder to simplify that
param("log4j.appender.stdout", "org.apache.log4j.ConsoleAppender")� param("log4j.appender.stdout.layout", "org.apache.log4j.PatternLayout")� param("log4j.appender.stdout.layout.ConversionPattern", "%p\t%d{ISO8601}\t%r\t%c\t[%t]\t%m%n")�
Step 2. Builder for Appenders
fun Log4J.appender(name : String, type : String,� builder : Log4JAppender.() -> Unit)
interface Log4JAppender : Log4JBase {� fun layout(type: String,� builder : Log4JLayout.() -> Unit)�}�
Step 2. Outcome (old)
//instead of�param("log4j.appender.stdout", "org.apache.log4j.ConsoleAppender")�param("log4j.appender.stdout.layout", "org.apache.log4j.PatternLayout")�param("log4j.appender.stdout.layout.ConversionPattern", �
Step 2. Usage of Appender Builder
//use this�appender("stdout", "org.apache.log4j.ConsoleAppender") {� layout("org.apache.log4j.PatternLayout") {� param("ConversionPattern", "%p\t%d{ISO8601}\t%r\t%c\t[%t]\t%m%n")� }�}
Step 3. Builder for Logger
Let’s simplify those lines too
log4j {� rootLogger = "ERROR,stdout"� param("log4j.logger.corp.mega", "INFO")� param("log4j.additivity.corp.mega", "false")�}�
Step 3. Logger Builder APIs
Define a builder for logger settings
interface Log4JLogger : Log4JBase {� var additivity : Boolean?� var level : Log4JLevel?� var appenders : List<String>�}
Step 3. Logger Extension Functions
fun Log4J.logger(category: String, � builder : Log4JLogger.() -> Unit)
fun Log4J.rootLogger(� builder : Log4JLogger.() -> Unit)
Step 3. Improved DSL
rootLogger {� level = ERROR� appenders += "stdout"�}�logger("corp.mega") {� additivity = false� level = INFO�}
Step 1 - 3. Outcome
IDE Support is implemented
Better readability / no skills needed
Harder to make a syntax error
Semantic & Object model implemented
Step 4. Tracking Appender Usages
Implementing IDE features
Find usages
Rename
Step 4. Tracking Appender Usages
A feature projection
Let’s find similar Kotlin IDE feature
=> Kotlin variable declaration looks good
Step 4. Tracking Appender Usages
introduce Log4JAppenderRef interface
appender() function returns Log4JAppenderRef� make generator generate variables for appenders
appenders type List<Log4JAppenderRef>� Make generator use variables, not strings
Step 4. Appender variable variable
val stdout = appender("stdout", ...
appenders += stdout
Type of stdout is Log4JAppenderRef.
A type-checked call to add item into collection (+=)
Step 4. Benefits from DSL & IDE Support
IDE knows about Appender usages� Find Usages� Rename
Semantics� Only declared Appender can be used� Duplicate appenders/loggers are rejected
Steps. The DSL for Log4j
log4j {� val stdout = appender<ConsoleAppender>("stdout") {…}� rootLogger {� appenders += stdout� level = ERROR� }� logger("corp.mega.itl.web.metrics") { … }�}
Steps. Outcome
Created a Log4J configuration DSL
Helps to read / refactor / change / reuse Log4J .properties files
Support in IDE with code-completion and type inference
Semantic checks and Errors prevention included
The DSL Way: Benefits
Side Effect 1. Automatic Tests
DSL-API is a domain model
Let’s use domain model in tests
Small modifications are required to access build model
Side Effect 1. Automatic Tests. API
interface Log4JModel {� val appenders : List<Log4JAppender>� val loggers : List<Log4JLogger>� val rootAppender : Log4JAppender�}
Side Effect 1. JUnit Integration
Kotlin / IntelliJ / Gradle supports JUnit
All we need is an API to load model from tests
class Log4JConfigTest {� @Test� fun categories_should_have_appender() { … }�}
Side Effect 2. Continuous Integration
No IDE dependency
Can simply run tests as a part of CI build
Combine generated sources with test sources
Build: generate DSL code, code & tests, run tests
Side Effect 3. Supporting XML configurations
We may implement similar generators for XML configurations too
DSL-API can be reused as well as code-generation part
=> We have full-featured .properties to XML converter
Side Effect 4. Power of DSL
A semantic aware templating engine
flexible & faster development with loops, conditions, etc� generator may generate
better validation & semantic checks
extensibility with shared libraries
Side Effect 4. Power of DSL: Logger configuration
log4j {� logger("category2warn") {� + WARN� }�}
Side Effect 4. Power of DSL: Use forEach
log4j {� listOf("A", "B", "C").forEach {� logger("category.$it") {� + WARN� }� }�}
Side Effect 4. Power of DSL: Use forEach 2
log4j {� listAllRootPackages().forEach {� logger("category.$it") {� + WARN� }� }�}
Side Effect 4. Power of DSL: Library Extensibility
Extract common part
log4j {� listAllRootPackages().forEach { � warnCategory(it) � }�}
Side Effect 4. Power of DSL. Shared Library
//library�fun Log4jBuilder.warnCategory(n: String) {� logger("category.$n") {� +WARN� }�}
Log4j Example. Outcome
A simple, but feature-complete example
Benefits / IntelliJ IDEA support / Find Usages and Renames
Source code: https://github.com/jonnyzzz/Log4j2DSL
Designing DSLs
Inventing a DSL
Know language features. Deeply
Model domain as a set of function calls & builders
Improvement Loop
Inventing DSL: Kotlin
Type-Safe builder in Kotlin� https://kotlinlang.org/docs/reference/type-safe-builders.html
Delegated properties� https://kotlinlang.org/docs/reference/delegated-properties.html
Operator overloading� https://kotlinlang.org/docs/reference/operator-overloading.html
Packing things
Packing Things
Easy-to use and apply package is required
Updates should be easy
IDE support should just work with zero config
=> How one can ship it?
Packing via a Build System
Requirements� Dependencies Resolution� Compiler setup� Kotlin setup� Tasks execution
One-click IDE support
Simple installation & upgrade
Gradle
Gradle plays well with � IntelliJ IDEA� Kotlin
Allows to implement an zero-configuration package
=> Let’s discuss it in example
The Gradle package
Setup compiler and dependencies
Tasks ‘toDSL’ and ‘fromDSL’
Bring IDE support
Initial Gradle Build
buildscript {� repositories {� mavenCentral()� }� dependencies {� classpath "...:$kotlin_version"� }�}�apply plugin: 'java'�apply plugin: 'kotlin'�
repositories {� /// DSL apis, Generator apis, kotlin�}��dependencies {� compile "...$DSL_VERSION"�}��task toDSL { /*...javaExec…*/ }�task fromDSL { /*...javaExec…*/ }
Use Gradle plugin
Gradle plugin syntax � only 3 lines in build.gradle�
Gradle Plugin repository�
Move all details into plugin code
Example: Enable a Plugin in Gradle
//in build.gradle
plugins {� id "ID" version "VER"�}
Open in IDE
Our Gradle plugin uses standard extensibility
It works in IntelliJ IDEA as is
Code is greed, refactorings are available
Tasks are detected
Example. Package
This was implemented in TeamCity2DSL project� A refactoring tool for TeamCity 8.0 - 10.0 XML configurations
A real-working The DSL Way implementation
Sources: https://github.com/jonnyzzz/TeamCity2DSL
Starting from TeamCity 10.0 it has embedded support of DSL build configurations
TeamCity in Action
JetBrains TeamCity
Powerful continuous integration server by JetBrains
Configuration objects:� Project� Build Configuration� Build Configuration Template� Meta-Runner� VCS Root (Version Control)
Project settings are in XML, available through Version Control
Why The DSL Way for Continuous Integration?
A DSL allows to use richer (Kotlin) language instead of XML
TeamCity concepts are represented explicitly� Project, VCS Root, Build Configuration, etc
Basic semantic checks of XML data is implemented� Smarter that XSD/DTD� Reports domain-related errors
Toolset in Gradle: Done
//root build.gradle file
plugins {� id "org.jonnyzzz.teamcity.dsl" version "0.1.15"�}
Side Effect 5: Automatic Tests in Gradle
Once Kotlin and Gradle are used
Tests are available out-of-the box
IDE support is also included automatically
add dependency on junit & code
Side Effect 6: Extensibility support
TeamCity2DSL supports extensions via .jars� e.g. custom DSL for a build runner
We need the way to support extensibility in Gradle Plugin
=> Use Gradle’s DSL extension mechanism.
Side Effect 6: Extending DSL & Generator
plugins {� id "org.jonnyzzz.teamcity.dsl" version "0.1.15"�}��teamcity2dsl {� extension '...:commandline-runner:0.1.15'�}
Live Demo
Let’s have a look in IntelliJ on TeamCity project configuration example
Start from scratch
Adding an extension
Adding unit tests
Live Demo
Package. Outline
It’s possible to build self-contained toolset as easy as
plugins {� id "PLUGIN" version "VERSION"�}
With IDE supported, easy to use, easy to upgrade, extensible
Implementation details are fully abstracted in plugin implementation
Outcome
The DSL Way toolset with IDE support is doable � with IntelliJ IDEA as IDE� with JetBrains Kotlin for DSLs� with three-lines in Gradle� �
Build your own toolset!
�JetBrains TeamCity 10.0 will have it’s own DSL support �JetBrains MPS is a DSL languages workbench
The DSL Way
Summary
Need an IDE support for a language?
The DSL Way is also possible!
No IDE dependency required
Questions?
Thank you!
�Follow me @jonnyzzz
�
Examples https://github.com/jonnyzzz/TeamCity2DSL�And https://github.com/jonnyzzz/Log4j2DSL�Libs https://github.com/jonnyzzz/kotlin.xml.bind
Creating Small DSLs with Idiomatic Kotlin
[CON1242]
Hadi Hariri, Developer, JetBrains
Wednesday, Sep 21, 10:00 a.m. - 11:00 a.m. | Hilton - Plaza Room A
TOO MANY DETAILS!
Gradle plugin case
Easy to use / upgrade
No IDE configuration required
Test runners from java plugin
�=> A self-contained toolset that can be used to develop DSL code
Gradle plugin code
Ok to use Kotlin (not Groovy / Java)
Given a Gradle Project instance
Have similar code as a plugin
class TeamCity2DSLPlugin : Plugin<Project> {� override fun apply(base : Project) {
Trick: Implicit Kotlin Setup
From the plugin we know Kotlin version/dependencies that are required
=> declare dependency from Kotlin in plugin’s pom � apply plugins from plugin
project.apply { config ->� config.plugin("java")� config.plugin("kotlin")�}
Initial Gradle Build for Toolset
buildscript {� repositories {� mavenCentral()� }� dependencies {� classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"� }�}�apply plugin: 'java'�apply plugin: 'kotlin'�
repositories {� /// same again�}��dependencies {� compile "...$DSL_VERSION"�}��task xml2dsl { /*...javaExec…*/ }�task dsl2xml { /*...javaExec…*/ }
Trick: Downloading Dependencies
Publish to a maven repo
Generate constants from the plugin build
Same dependency names & versions (not hardcoded)
Same components & plugins
In plugin code:
Include generated list of dependencies & repositories
Initial Gradle Build for Toolset
buildscript {� repositories {� mavenCentral()� }� dependencies {� classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"� }�}�apply plugin: 'java'�apply plugin: 'kotlin'�
repositories {� /// same again�}��dependencies {� compile "...$DSL_VERSION"�}��task xml2dsl { /*...javaExec…*/ }�task dsl2xml { /*...javaExec…*/ }
TeamCity2DSL
My goal was to learn Kotlin & practice in DSL building
I was looking for proper tooling
Sources:�https://github.com/jonnyzzz/TeamCity2DSL�Note: TeamCity 10 has DSL support out of the box
Step 2. Outcome
Remove common / repeating string
Improve readability
Avoid some sort of typos
Few semantic checks
Light object model
Step 3. Forward declaration for constants trick
level = ERROR
The ERROR should be redeclared to use without Log4JLevel prefix
val Log4J.ERROR : Log4JLevel� get() = Log4JLevel.ERROR
Step 3. Alternative for wide screens
Via named parameters in Kotlin with default values
rootLogger(level = ERROR, appeners = listOf(stdout))
logger("corp.mega", additivity = false, level = INFO)
Declaration
fun Log4J.logger(category: String, additivity : Boolean = true, � level : Log4JLevel? = null,� appeners : List<Log4jAppenderRef>? = null)
Step 3. Alternative. Hacker Style
Via overloaded operators
logger("corp.mega.d2", level = INFO) >= stdout
DSL design is a matter of taste
Step 5. Use declared classes
References to Log4J classes are string constants
Let’s replace it with Class objects
log4j {� appender("stdout", "org.apache.log4j.ConsoleAppender") {� layout("org.apache.log4j.PatternLayout") {� param("ConversionPattern", "...")� }� }
Step 5. Use declared classes
fun <T : Appender> Log4J.appender(� name : String, � type : KClass<T>, � builder : Log4JAppender.() -> Unit) ...
Step 5. Use declared classes
log4j {� appender("stdout", ConsoleAppender::class) {� layout(PatternLayout::class) {� param("ConversionPattern", "%p\t%d{ISO8601}\t%...")� }� }�}
Step 6*. Use Log4J Objects Directly
layout<PatternLayout> {� // this is a call to PatternLayout#setConversionPattern� conversionPattern = "%p\t%d{ISO8601}\t..."�}
Via reified generics and inline functions in Kotlin
inline fun <reified T : Layout> layout(l : T.() -> Unit) { � val type = T::class // allowed for reified generics
Target Language Examples
A language that is supported by an IDE� Kotlin, Scala, Groovy …
Create a DSL API library, generator, emitter
Get IDE support & refactorings
Create unit tests
Have semantics checks
Package: Use one build tool
No mix of tools � e.g. Ant + Maven
No duplication of settings� e.g. IntelliJ project + Ant
No tricky files or templates
No complex documentation
Trick: Update configuration from Extension
Update Project instance once DSL extension method is called
Pass Project into a DSL extension class
Alternative
update Project in afterEvaluate. May not work
Trick: Running TeamCity2DSL
Obvious plan:� Declare everything onto buildscript classpath� Run classes as-is
Problem: classpath is shared. Can have conflicts� It did not work with jdom which conflicted with Gradle internals
Solution: � Load classes explicitly� Avoid any buildscript classpath level dependencies
Testing the plugin code itself
All tests are integration tests� Running on a gradle script sample
Local maven deployment is used for such tests� A dedicated step is created to publish all dependencies� Local maven added into repositories
Base project is used to setup tests
There are a number of tests for the tool itself, without Gradle
Initial Solution
Implement tasks in Ant (required locally)
Download dependencies (Kotlin), call compiler, and everything is done with Ant� Partly duplicated in IntelliJ settings
Hard to update to newer version
No IDE project included (so a template project was introduced)
Complicated startup
No extensibility supported
TOO COMPLEX!