Getting Eclipse to work with Google’s Ginormous Codebase™

Frank Kieviet, Google

Contents

• The Google build environment: what’s unusual about it?
• Why is this difficult for Eclipse?
• Solution

​

The Google Build environment

Code management system at Google

• There is only a single code repository
• 99% of Java code (several million files) lives in two directories:
• java (e.g. //java/com/google/common/base/Joiner.java)
• javatests
• Almost no jars checked in
• There are also no jars stored elsewhere
• There are no branches: everybody works at HEAD

​

​

Google’s “build” language

• Each directory represents a package and has a BUILD file
• A BUILD file contains targets
• Each target
• declares dependencies
• produces an artifact, e.g. library, test, or application
• Very easy to add a new dependency
• Consequently, developers add too many dependencies

Google’s build system

• Builds everything
• Yes, EVERYTHING! (up to Unix runtime libraries)
• Everybody is at HEAD
• Yet surprisingly fast:
• uses caching heavily
• can fork out building artifacts to the cloud
• Onlys build what’s new
• Output: multiple Gb for a typical Java-based service

Google developer’s experience

• Virtual file system with several million files
• Directories java, javatests, build-output
• java, javatests contain millions of files
• build-output contains thousands of jars, multiple Gbs of artifacts
• Natively supported workflow: edit file, build, run

​

Using Eclipse at Google

Basic problems

• Eclipse can’t handle a tree with millions of files
• Eclipse needs a classpath
• Shorter classpaths preferred:�Eclipse doesn’t scale well with very long classpaths
• Eclipse needs jars to run/debug tests and applications
• Eclipse needs source files to debug through code in jars

Approaches

General approach: work on a small subset of the source tree

(typically 10000-20000 Java source files)

• Approach 1 ← not covered in this presentation
• extensive tooling in Eclipse
• deep integration with Google’s build system
• add all/most (thousands) jars to classpath (situation two years ago)
• relatively smooth developer experience
• Approach 2 ← covered in this presentation
• minimal tooling
• minimal classpath

Magicjar

Three components:

1. Dependency analyzer generates minimal classpath
2. Eclipse plugin
1. filters source directories
2. finds source files for compiled code
3. Custom classloader for running code

Scope

• Specifies what files in the Ginormous Sourcetree™ should be visible
• Simple “scope-language” describing what directories and their subdirectories to include/exclude
• Contained in a simple text file

Classpath generation

Classpath generation

Dependency analysis

1. Index all jars to build a catalog that maps classname → jar
2. Use Eclipse Batch compiler to find out what classes are needed on classpath to compile all sources in the Scope

​

class Compiler extends

org.eclipse.jdt.internal.compiler.batch.Main

@Override protected void setPaths(...) {

newClasspaths.add(new IndexedClasspath(...));

this.checkedClasspaths = newClasspaths;

}

}

IndexedClasspath

class IndexedClasspath implements

org.eclipse.jdt.internal.compiler.batch.FileSystem.Classpath

​

When the BatchCompiler asks for a class:

• looks up jar in Catalog
• records the name of the class
• loads class from jar and returns “stripped bytecode” to BatchCompiler

​

After compilation finishes:

• returns a list of all required classes

Bytecode stripping

• Removes all implementation details from a class:
• private members
• private methods
• method bodies
• Reduces size
• Reduces dependencies on other classes
• Keeps only relevant parts for compiler
• Implementation: ASM (org.objectweb.asm.ClassReader)

Classpath generation

• List of required classes from IndexedClassloader
• Yields a list of jars
• Typically a few hundred jars
• Further optimization: combine all jars into a handful of bigger jars

Classpath generation: unexpected problem

​

Problem:

• Small difference between Eclipse BatchCompiler and RCP Compiler
• Happens sporadically (about 1% of users), yet significant

​

Solution:

• Further ASM inspection of generated classpath
• Adds additional jars to classpath
• Single level analysis suffices in 99.9% of cases
• Optional: multi-level analysis

Remaining problems

• Batch Compiler’s Annotation Preprocessor blows up randomly
• Bug in Eclipse
• Pessimistic of fix any time soon
• Workaround:
• Include classes generated by Annotation Preprocessor in Build System

Eclipse plugin

Eclipse plugin

Eclipse plugin responsibilities

• Filter source tree to adhere to Scope
• Lookup source for compiled code
• Provide JRE configurations for custom classloader

Scope

• Monitors Scope file
• Hides everything from the Ginormous Sourcetree™ except what is in Scope
• Implemented as a filter

​

<extension

point="org.eclipse.core.resources.filterMatchers">

<filterMatcher

​

public class FilterFileResourceFilter extends

org.eclipse.core.resources.filtermatchers.AbstractFileInfoMatcher {

Lookup sources for compiled code

• Implemented as an editor (since 3.8 uses the classes-without-sources file association)
• Classpath generator writes metadata for each jar
• Uses this metadata + a set of rules to find the source of a Java file given its jar

​

<extension

point="org.eclipse.ui.editors">

​

public class ExtendedClassFileEditor extends

org.eclipse.jdt.internal.ui.javaeditor.ClassFileEditor {

JVM Configurations

• Configures JREs that have the -Djava.system.class.loader=magicjar property
• Makes it possible for the classpath generator to create a complete Eclipse project that requires no customization by the user

​

<extension

point="org.eclipse.jdt.launching.vmInstalls">

​

Custom classloader

Custom classloader

Custom classloader

• Stripped bytecode class files cannot be used to run code within Eclipse
• Classpath does not contain all classes to execute code
• Many implementation classes omitted because the stripped code no longer references the implementation classes.
• Custom classloader loads classes from
• Eclipse’s output directory
• The Ginormous Build-output Directory™
• Uses the Catalog for this (recall Catalog maps classname → jar file)
• Makes running JUnit tests very fast

Problems

• Despite…
• single java source directory
• everybody working at HEAD

there sometimes ARE versioning problems

• Code like this:

if (Class.forName(“...”) != null) {

// we’re apparently in our private testing environment

Solution

• Classloader will construct classpath based on the class-being-run and build-system generated classpath
• Class-being-run is first class loaded from Eclipse generated classes

Conclusion

Merits of approach

• Can be used to construct an efficient classpath for “a Ginormous Pile of Jars™ ”
• Shorter classpath makes Eclipse much more responsive
• Relatively portable approach: is also being used for IntelliJ

Thank you!

Frank Kieviet, Google

http://google.com/+FrankKieviet (requires G+ account)