1 of 22

Dependency Management in Scala: Now and Later

Mark Harrah

Typesafe

2 of 22

Manual dependency management

  • Go to website, download jar, copy to project
  • Figure out dependencies, repeat for each
  • Maintenance: high, collaboration: low

3 of 22

Automatic dependency management

  • Machine readable dependency, product definitions (metadata)
  • Transitively processes dependencies
  • Resolves conflicts
  • Retrieves remote information => requires caching
  • Publishes metadata and artifacts

​

4 of 22

Transitive/Conflict Resolution

demo 1.0

A 1.0

B 2.3

C 3.0

D 1.3

C 4.0

...

...

5 of 22

On-demand access and caching

Need metadata for org.example:A: 1.0

In cache?

Uptodate?

Exists?

<repo>/org/example/A/1.0/pom.xml

​

Download and cache.

Process and continue.

Yes

Yes

Yes

Fail. Cache failure

No

No

No

6 of 22

Motivation for something different

  • Uptodate checking problematic for on-demand, 1:N remote paths
  • Metadata: pom.xml deficient, stagnant
  • Concurrency, repository management, offline installation, ...

7 of 22

Metadata: Universes/Compatibility

  • Scala 2.9/2.10 binary incompatible
  • Mostly source compatible
  • Publish/consume libraries for both 2.9/2.10
  • Need separate universes

8 of 22

Cross-versioning

  • name_2.10
  • fragile: name_2.9 and name_2.10 not related- outside of conflict resolution
  • not general: sbt, Play, other libraries
  • coarse: not able to handle different types of compatibility

9 of 22

Better Cross-versioning

  • Version attributes instead of single version
    • scalaBinaryVersion = 2.10
    • uniqueVersion = 2.11.0-20130208-134329-f1a342c
    • version = 2.11.0-SNAPSHOT
  • Like Ivy's extra attributes
  • Structured information: automation

10 of 22

Richer resolution configuration

  • Akka standard v. instrumented jars for Typesafe Console
  • Renaming organization or module

demo:demo => org.example:demo

  • pom.xml: limited support for "relocation"

11 of 22

Extra metadata: main class

$ dz run org.example.HelloWorld

Getting org.example:hello-world:2.10.0 ...

Running HelloWorld...

​

Hello World!

12 of 22

Management and distribution

  • Barrier to publishing
  • Cache access by concurrent processes
  • Integration with native packagers
  • Offline installation
  • Repository management: mirroring, ...

13 of 22

Proposal

  • Metadata repository
    • DVCS
    • Batch update
  • Artifact repository
    • Separate, file repository
    • 1:1 path:content via hash of content
    • Still on-demand retrieval
  • Metadata format
    • Richer, evolving
    • Requires matching resolution engine

14 of 22

Metadata repository

  • Metadata stored in DVCS (git, ...)
  • All metadata retrieved and locally available

$ dz search http

net.databinder.dispatch:dispatch-core

0.9.5 (scalaVersion: 2.9.2, 2.10)

  • Repository lists instead of centralized repository?
  • Scale?

15 of 22

Metadata repository: DVCS

  • Consistent local cache, remote repository
  • Efficient, incremental updates
  • Lower resource requirements on server?
  • Commit multi-module project in one transaction
  • GitHub, BitBucket, ... < 18 steps to publish?

​

16 of 22

Version controlled metadata

  • Reproducible builds
    • Record repository+commit
    • Can still use features like dynamic revisions
  • Improve metadata at the source
  • Manage with standard DVCS techniques
  • View/diff/audit metadata changes

org.example:demo:2.4

- dependsOn B 1.0

+ dependsOn B 1.1

​

17 of 22

Artifact repository

  • Strong hash of content: 1 hash : 1 jar
  • <repo>/a6/32f978cc4/content
  • Consistent cache, offline uptodate checking
  • Easy to mirror, split, or merge
  • Main problem: human readable names

​

scalac -cp a6/32f978cc4/content A.scala

18 of 22

Metadata core

  • Identifying attributes
  • Version attributes
  • Describing attributes
  • Published artifacts listed by hash
  • Configurations
  • Resolution control: provides, conflicts, overrides, force

19 of 22

Metadata example

organization:

org.example

name:

demo

uniqueVersion

2.11.0-20130208-134329-f1a342c

commonVersion

2.11.0

binaryVersion

2.11

artifacts

9823d8af32, a2ee3291ca (sources)

provides

demo:demo

apiURL

http://example.org/api/

mainClass

org.example.RunDemo

dependsOn

org.example:core, org.example:util

Flattened, simplified, ...

20 of 22

Compatibility

  • Learn from previous attempts
  • Repository-side instead of user-side
  • Convert metadata with a batch tool
  • Incremental improve conversion
  • Publishing difficult: fundamentally lossy
  • Aether, Ivy

21 of 22

Opportunities

  • Identify/specify workflows, requirements
  • Metadata format and dependency resolution
  • Compatibility
  • User interface/command line
  • Metadata repository
  • Tools: search, run main class, verify

22 of 22

Conclusion

  • Opportunities to improve dependency management for Scala
  • Worth it?
  • Involvement key to success
  • Focus on workflow, reliability, automation
  • Session tomorrow?