Eugene Petrenko
JetBrains
@jonnyzzz
Building self-contained toolset with Gradle
2016
About me
PhD in Computer Science
At JetBrains (ReSharper, TeamCity, Upsource, Toolbox App and many more)
Open source projects & plugins
Fun in Kotlin & DSLs
Blogging at http://jonnyzzz.com
Agenda
What is Toolset / self-contained mean?
Building DSLs with Kotlin
A Self-contained DSL Toolset in Gradle
Live demo
What is Toolset
Language
Compiler
Build scripts
Extensibility
IDE support
Tests
Self-contained
An instrument to work with the language (e.g. DSL) with all required operations
Should � not require any complicated setup� be easy to upgrade� have an IDE support� allows extensibility or plugins
Minimalism, implementation details isolation
Toolset for?
Language is most likely a
Domain Specific Language
Examples� Business rules� Configuration management (e.g. Logger, CI settings, Clouds)� Game logic / Lua scripts generators (e.g. Moonscript)� Workflow editors (e.g. JetBrains YouTrack Workflow editor, MPS based)�
YouTrack Workflow Editor / DSL
Tools from screenshot
JetBrains YouTrack workflow editor
JetBrains MPS used as IDE (via Java Web Start)
How DSLs are used
Generate code in target language from a DSL
extend target language with higher level concepts
use better syntax
add more semantics checks or tests
How DSLs are used also
A two way transition
Import code from original language
Change, test, refactor DSL code with help of Toolset & DSL� A DSL should be designed for better refactorings
Generate original language code back�
How to implement a DSL
There are many ways
Re-invent the wheel
Use Languages Workbench, e.g. JetBrains MPS
Use existing language as base, e.g. Groovy, Scala, Kotlin
Target Language
If cannot change target language
May not be fully supported in IDE, no refactoring
Not as powerful as java/kotlin (e.g. no loops, conditions, functions)
Lacking in validation & semantic checks
Lacking in extensibility
DSL in General Purpose Language
Use existing language, author a library for it
better IDE support with no IDE extensions needed
flexible & faster development with loops, conditions, etc
better validation & semantic checks
extensibility is possible
We will use JetBrains Kotlin language for examples
Why Kotlin?
Open source static typed language by JetBrains
Easy to learn & use
Great IDE support (Eclipse & IntelliJ)
Easy to write static typed DSLs
IDE understands types in DSLs
More on kotlinlang.org
Example: Log4j Logger config
Let’s consider trivial log4j configuration
<category name="category2warn">� <priority value="WARN"/>�</category>
Example: Logger configuration
Replace it with a DSL in Kotlin
loggerConfig {� category("category2warn") {� + WARN� }�}
DSLs in Kotlin: Extension Functions
fun String.parenthesize() : String { � return "($this)" �}
//usage�"some text".parenthesize()
DSLs in Kotlin: Higher order Functions
Function type
(String, Long) -> Int
Extension function type
String.(Byte) -> Unit
Example: Logger configuration. Kotlin
An parameter is an anonymous extension function
Last parameter of a function call can be without braces ( )
fun loggerConfig(� closure: LoggerBuilder.() -> Unit)
Example: Logger configuration
Replaced it with a DSL in Kotlin
loggerConfig {� category("category2warn") {� + WARN� }�}
Example: Logger configuration. Use forEach
loggerConfig {� listOf("A", "B", "C").forEach {� category("category.$it") {� + WARN� }� }�}
Example: Logger configuration. Use forEach 2
loggerConfig {� listAllRootPackages().forEach {� category("category.$it") {� + WARN� }� }�}
Example: Logger Configuration. Library Extensibility
Extract common part
loggerConfig {� listAllRootPackages().forEach { � warnCategory(it) � }�}
Example. Logger Configuration. Shared Library
//library�fun Log4jBuilder.warnCategory(n: String) {� category("category.$n") {� +WARN� }�}
DSLs in Kotlin
Static typed
With IDE support, code completion, error highlighting, refactorings
Can add semantic checks easily
Can have extensibility or unit tests
Outcome
DSLs over general purpose language (kotlin) can be useful
We need a Toolset with IDE support
=> Let’s build one with Gradle
DSL Toolset with Gradle
Implementation details can be hidden in a plugin
Easy to reuse & upgrade
Reuse ‘java’ and ‘kotlin’ plugin features
Building a Kotlin DSL Toolset
Toolset for DSL should
generate to original language code (and from)
Be easy to use / install / update
Should contain an IDE support
This is the only way to have success in DSL adoption
Extensibility Support
Extend � DSL & Generator
make IDE support work for it
Benefit from Gradle/Kotlin features
�
Tests support
Once we have a code in Kotlin for DSL
Let’s have tests for such code
May require support from DSL library
Reuse unit tests support from IDE (no extra code)
Example. TeamCity2DSL
My goal was to learn Kotlin & practice in DSL building
I was looking for proper tooling
Sources:�https://github.com/jonnyzzz/TeamCity2DSL
�Note: TeamCity 10 will have DSL support out of the box.
TeamCity in Action
JetBrains TeamCity
Powerful continuous integration server by JetBrains
Configuration objects:� Project� Build Configuration� Build Configuration Template� Meta-Runner� VCS Root (Version Control)
Project settings are in XML, available via Version Control
Why DSL for CI?
A DSL allows to use richer (Kotlin) language instead of XML
Higher level concepts can be used� Project, VCS Root, Build Configuration, etc
Basic semantic checks of XML data is implemented� Smarter that XSD/DTD� Reports domain-related errors
Kotlin language refactorings on TeamCity entities� *Requires right DSL design
Toolset: Workflow
configuration in XMLs
-> generate Kotlin code
-> refactor, compile, test
-> generate configuration XML
The Toolset: Refactoring Tool
TeamCity project settings are in a set of XML files
XML files under version control
‘xml2dsl’ and ‘dsl2xml’ tasks
Bring IDE support
Initial Solution
Implement tasks in Ant (required locally)
Download dependencies (Kotlin), call compiler, and everything is done with Ant� Partly duplicated in IntelliJ settings
Hard to update to newer version
No IDE project included (so a template project was introduced)
Complicated startup
No extensibility supported
TOO COMPLEX!
IDEA: Use one build tool
No duplication of settings� e.g. IntelliJ project + Ant Script
No tricky files or templates
Initial Gradle Build for Toolset
buildscript {� repositories {� mavenCentral()� }� dependencies {� classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"� }�}�apply plugin: 'java'�apply plugin: 'kotlin'�
repositories {� /// DSL apis, Generator apis, kotlin�}��dependencies {� compile "...$DSL_VERSION"�}��task xml2dsl { /*...javaExec…*/ }�task dsl2xml { /*...javaExec…*/ }
TOO MANY DETAILS!
Use Gradle plugin
New plugin syntax � => only 3 lines in build.gradle� all dependencies from jcenter()��Move all details into plugin code
Use Kotlin for the plugin implementation
plugins {� id "ID" version "VER"�}
Toolset in Gradle: Self-contained
This should be enough to implement this all with Gradle
//root build.gradle file
plugins {� id "org.jonnyzzz.teamcity.dsl" version "0.1.15"�}
Gradle plugin case
Easy to use / upgrade
No IDE configuration required
IDE support is out of the box
Test runners from java plugin
�=> A self-contained toolset that can be used to develop DSL code
With IDE support
We created a Gradle plugin that uses standard extensibility
It works in IntelliJ as is
Code is greed, refactorings are available
Tasks are detected
Tricks inside
Dependencies on ‘java’ and ‘kotlin’ gradle plugins
Automatic download of dependencies (DSL, generator, extensions)
Gradle DSL extensions for extensibility
Trick: Implicit Kotlin Setup
From the plugin we know Kotlin version/dependencies that are required
=> declare dependency from Kotlin in plugin’s pom
project.apply { config ->� config.plugin("java")� config.plugin("kotlin")�}
Initial Gradle Build for Toolset
buildscript {� repositories {� mavenCentral()� }� dependencies {� classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"� }�}�apply plugin: 'java'�apply plugin: 'kotlin'�
repositories {� /// same again�}��dependencies {� compile "...$DSL_VERSION"�}��task xml2dsl { /*...javaExec…*/ }�task dsl2xml { /*...javaExec…*/ }
Trick: Downloading Dependencies
Publish to a maven repo
Generate constants from the plugin build
Same dependency names & versions (not hardcoded)
Same components & plugins
In plugin code:
Include generated list of dependencies & repositories
Initial Gradle Build for Toolset
buildscript {� repositories {� mavenCentral()� }� dependencies {� classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"� }�}�apply plugin: 'java'�apply plugin: 'kotlin'�
repositories {� /// same again�}��dependencies {� compile "...$DSL_VERSION"�}��task xml2dsl { /*...javaExec…*/ }�task dsl2xml { /*...javaExec…*/ }
Trick: Running TeamCity2DSL
Obvious plan:� Declare everything onto buildscript classpath� Run classes as-is
Problem: classpath is shared. Can have conflicts� It did not work with jdom which conflicted with Gradle internals
Solution: � Load classes explicitly� Avoid any buildscript level dependencies
Toolset in Gradle: Self-contained
This should be enough to implement this all with Gradle
//root build.gradle file
plugins {� id "org.jonnyzzz.teamcity.dsl" version "0.1.15"�}
Trick: Extensibility support
TeamCity2DSL supports extensions via .jars� e.g. custom DSL for a build runner
We need the way to support extensibility in Gradle Plugin
=> Use Gradle’s DSL extension mechanism.
Extending DSL & Generator
plugins {� id "org.jonnyzzz.teamcity.dsl" version "0.1.15"�}��teamcity2dsl {� extension '...:commandline-runner:0.1.15'�}
Trick: Update configuration from Extension
Update Project instance once DSL extension method is called
Pass Project into a DSL extension class
Alternative
update Project in afterEvaluate. May not work
Trick: Unit Tests
Once Kotlin and Gradle are used
Tests are available out-of-the box
IDE support is also included automatically
add dependency on junit & code
Live Demo
Let’s have a look in IntelliJ on TeamCity project configuration example
Start from scratch
Adding an extension
Adding unit tests
Live Demo
Welcome back
Testing the plugin code itself
All tests are integration tests� Running on a gradle script sample
Local maven deployment is used for such tests� A dedicated step is created to publish all dependencies� Local maven added into repositories
Base project is used to setup tests
There are a number of tests for the tool itself, without Gradle
Outline
It’s possible to build self-contained toolset as easy as
plugins {� id "PLUGIN" version "VERSION"�}
With IDE supported, easy to use, easy to upgrade, extensible
Implementation details are fully hidden
Outline: Self-Contained DSL Toolset in Gradle
Self-contained DSL toolset with IDE support is doable � with JetBrains Kotlin for DSLs� with three-lines in Gradle� can share with non-IT people�
Build your own toolset!
Example implementation & details are here�https://github.com/jonnyzzz/TeamCity2DSL
�JetBrains TeamCity 10.0 will have it’s own DSL support �JetBrains MPS is a DSL languages workbench
TeamCity2DSL: build.gradle
plugins {� id "org.jonnyzzz.teamcity.dsl" version "0.1.15"�}
Extensibility Implementation
Add more .jars for “compile” configuration
Use Java Services API (via resource file)
A Toolset
Consider Domain Specific Languages use case
source files� XML build definitions for CI� Business processes definition files� Logging configuration files�
Those files are tricky to ��
Self-Contained Toolset
This is a tool that
Easy to use, try, upgrade
Includes an IDE support
Agenda
Generating code to simplify changes management
Tricks in creating self-contained Gradle plugin
Improvements
Outcome
Tool. Configuration files editor
Experiment in DSL area & Kotlin
Convert TeamCity build configurations XML files into Kotlin Code
Apply IDE refactorings to author build configuration change
Moves semantics and constraints into Kotlin library
Makes use of IDE to author non-trivial changes to XML
Problem: Implementation details
download dependencies & compiler
setup toolset, directories, classpath
register extensions
configure IDE
run compiler, call tool tasks, run tests, implement tasks
Problem: User experience
Tricky to install & start working
Complicated upgrade
Plugin Tasks
Plugin adds two tasks:
dsl2xml
xml2dsl
Plugin should replace all configuration
It should be easy to install & upgrade
Plugin entrypoint
Plugin is implemented as usual via
class GeneratorPlugin : Plugin<Project> {� override fun apply(project: Project?) { � //code will be here� }�}
I’ll omit this in next slides, assuming
Plugin: Defining Tasks
Tasks are defined in the standard way
Nice trick is to add dependencies in the next statements
project.tasks.create(� "dsl2xml", Dsl2Xml::class.java)
Plugin: Downloading Dependencies
The trick is to create an internal configuration and declare dependencies there
val configuration = project.configurations� .maybeCreate(NAME)� .setVisible(false).setTransitive(true)��project.dependencies.add(� conf.name, “foo.bar:1.2.3”)
Plugin: Avoid duplicating repositories
I use the following trick in plugin#apply
copy all buildscript repositories into build repositories
project.buildscript.repositories.forEach {� project.repositories.add(it) �}
Plugin: Loading Classes
val config = project.configurations� .getByName(NAME) ?: throw failTask(...)��URLClassLoader(� config.files.map{ it.toURI().toURL() }.toTypedArray(), � null)
Plugin: xml2dsl task is ready
Dependencies configured automatically
Task is executing in standalone classloader
Kotlin code is generated
Next: Setting up compiler & implement dsl2xml task
Plugin: Adding Source Root
Generated DSL code should be in a dedicated source root
project.convention� .getPlugin(JavaPluginConvention::class.java)� .sourceSets.getByName("main")� .java.srcDir(PATH)
Plugin: Add tasks dependencies
dsl2xml task should compile DSL prior to execution
Adding dependencies can be done as follows
val dsl2xml = project.tasks.create(...)�dsl2xml.dependsOn(� project.tasks.getByName("classes"))
Plugin: Putting all together
buildscript {� repositories { … }� dependencies {� classpath 'org.jonnyzzz.teamcity.dsl:gradle-plugin:${dsl-version}'� }�}��apply plugin: 'org.jonnyzzz.teamcity.dsl'
//xml2dsl & dsl2xml tasks are registered
Live demo
Open project from scratch
Show tasks and code navigation (in IntelliJ)
Testsing plugin: template
buildscript {� repositories {� mavenLocal()� mavenCentral()� maven { url "http://dl.bintray.com/jonnyzzz/maven" }� }� dependencies {� classpath '${DSL_PLUGIN_CLASSPATH}'� }�}�apply plugin: '${DSL_PLUGIN_NAME}'
Extensibility for the Generator
We use Java Services API
And deal with classloading carefully
Implemented via one line: � fix the gradle plugin to include more packages� via DSL extensions (and custom supply to ‘dependencies’)� via ‘compile’ configuration
Plugin: Extensibility
Register extension into project
project.extensions� .create(� "teamcity2dsl", DSLSettings::class.java)
Plugin: Extensibility. Read
Access extensions from script or task
Use project#afterEvaluate
Add more dependencies to the generator
project.extensions
.findByType(DSLSettings::class.java)
Plugin: Extensions
buildscript {� repositories { … }� dependencies {� classpath 'org.jonnyzzz.teamcity.dsl:gradle-plugin:${dsl-version}'� }�}��apply plugin: 'org.jonnyzzz.teamcity.dsl'��teamcity2dsl {� plugins.add ‘org.jonnyzzz.teamcity.dsl:sample-plugin:0.0.1’�}��
Plugin Extensions: Classpath
‘compile’ configuration should include those classes
An internal configuration should extend compile configuration to add generator implementation
Isolation is nice to have
Simplification: Use same classpath for all, � add dependencies to ‘compile’ configuration
Writing Unit Tests
Just works, just add junit or testng dependency
Running tests will make it compile main sources (including generated DSL)
Will work out of the box
‘xml2dsl’ task may be executed via dependencies
Recall
Create a Gradle plugin to � reduce build or tools complexity, � avoid binaries and copy-pastes and to hide details.
Generic tricks and ideas�
https://github.com/jonnyzzz/TeamCity2DSL
�TeamCity 10.EAP contains another builds DSL implementation
Short build script
buildscript {� repositories { … }� dependencies {� classpath 'org.jonnyzzz.teamcity.dsl:gradle-plugin:${dsl-version}'� }�}��apply plugin: 'org.jonnyzzz.teamcity.dsl'
//exposes xml2dsl and dsl2xml tasks
DSLs in Kotlin
html {� head { title = "Hello Kotlin DSLs" }� body {� div { + "Loren Ipsum" }� }�}
DSLs in Kotlin: Lambda Expressions
Last function-parameter of a method can be without ( )
fun html(builder: HTML.() -> Unit)��interface HTML {� fun head(builder: HEAD.() -> Unit)� fun body(builder: BODY.() -> Unit)�}
DSLs in Kotlin. Explained
html {� head { title = "Hello Kotlin DSLs" }� body {� div { + "Loren Ipsum" }� }�}
DSL Extensibility
TeamCity supports plugins (e.g. build runners)
Extend � DSL & Generator
make IDE support work for it
Example: add syntax for your build runner (e.g. Gradle Build Runner)�
Example: Logger configuration. Kotlin 2
A config function from extension function
Last parameter of a function call can be without braces
�fun category(name: String, � closure: CategoryBuilder.() -> Unit)�
operator fun WARN.unaryPlus()
Questions