Dependency Management with Gradle

Part 1 - Fundamentals

Cédric Champeau (@CedricChampeau) & Louis Jacomet (@ljacomet) - Gradle Inc.

Who are we?

who s who

Dependency management team

What is Gradle?

Gradle’s purpose

Gradle is a build and automation tool.

  • JVM based

  • Implemented in Java

  • 100% Free Open Source - Apache Standard License 2.0

Agnostic Build System

  • Java ecosystem

    • Groovy, Kotlin, Scala, …​

  • Native ecosystem

    • C, C++, Swift, …​

  • Android

  • Misc

    • Python, Go, Asciidoctor, …​

Gradle Enterprise

gradle enterprise

Why dependency management?

A long time ago …​

> javac -d out *.java
> cd out/
> jar cf ../../lib/project1.jar *.class
> cd ../../project2
> javac -cp ../lib/project1.jar -d out *.class

Then came tools

  • Helped you build and package

  • Did not handle dependencies

    • Download jars from project website

    • Lived in a lib directory or equivalent

    • Often saved next to code in source control

Source vs published

  • Sources

    • (mostly) reliable

    • (often) slow

    • hard to temper with

    • hard to version

Source vs published

  • Binaries

    • Stable

    • Fast (pre-built)

    • Requires trusted sources

    • Not always metadata

Growing ecosystem

With the growing popularity of Java and the growth of its ecosystem, managing all dependencies of a project turns into a dependency chase.

Thus dependency management …​

  • Ivy or Maven would take care of your dependencies

    • Libraries have known coordinates

    • Published to repositories, with metadata in addition of the binary

    • Automated the transitive aspect of dependency management

Metadata: Maven != Maven Central

  • Maven: a build tool

  • Maven POM: a metadata format

  • Maven repositories: Places where you can find binaries, with Maven metadata

And Gradle joins the party

  • Compatible with all of the previous options

    • lib or custom repositories

    • Ivy metadata

    • Maven metadata

    • Gradle Module Metadata (since 5.3)

    • and more …​

Demo

Getting dependencies

From a repository to your machine

Gradle attaches importance to the source of a dependency, its origin repository.

Gradle has lots of smarts in order to optimise the downloading of files.

Repositories

  • Order of repositories is important

  • Use repository filtering for partitioning sources

  • Avoid mavenLocal()

    • Not strictly a repository, but a cache for Maven

    • Sometimes contains partial module data, this will trip Gradle

  • Metadata and artifacts downloaded concurrently

How Gradle searches for binaries

  • Gradle knows where an artifact comes from

  • Get the SHA1 of the artifact (GET or HEAD)

  • Looks in its dependency cache if something at the same GAV has the same hash

    • If so, uses it

  • Can leverage local maven repository content instead of downloading

    • That is transparent, and different than having mavenLocal() declared as a repository

Consequence: if you get a file from Maven Central, then you change the repository to JCenter, if the artifacts are the same, downloaded only once.

Core concepts

Definitions

  • Module

    • A piece of software that evolves over time

    • Has a group, name and version to identify it

    • Example: org.slf4j:slf4j-api:1.7.2

  • Configuration
    In the context of dependency management, and for most users:

    • Named set of dependencies

    • Provides access to resolved modules and their artifacts

    • Example: implementation

API and implementation

bucket

More bucket configurations

bucket 2

Bucket configurations relationship

bucket 3

API vs implementation

  • To build a module, you need:

    • api, implementation and compileOnly dependencies

  • To run a module, you need:

    • api, implementation and runtimeOnly dependencies

Classpath configurations

classpath

API vs implementation transitively

  • To build against a module, you need:

    • transitive api dependencies

  • To run against a module, you need:

    • transitive api, implementation and runtimeOnly dependencies

Elements configurations

complete

Configurations

  • 3 kinds of configurations

    • "buckets" of dependencies

    • resolvable configurations

    • consumable configurations

Buckets of dependencies

  • A named list of dependencies

  • canBeConsumed=false
    canBeResolved=false

Resolvable configurations

  • Represents a consumer

  • For a specific usage (compiling, runtime, …​)

  • Extends from one or more bucket(s)

  • canBeConsumed=false
    canBeResolved=true

Consumable configurations

  • Represents a variant of a producer

  • Attaches artifacts

  • Extends from one or more bucket

  • canBeConsumed=true
    canBeResolved=false

Matching consumers with producers

Gradle needs a way to perform the matching between consumers and producers.

Attributes and variants are the answer.

More definitions

  • Attribute

    • Named

    • Typed

    • Can have compatibility and disambiguation rules

  • Variant

    • A consumable form of a module

    • Declare artifacts and / or dependencies

    • Identified by its attributes and their values

Compilation against project

compile1

Compilation against project

compile2

Compilation against project

compile3

Compilation against project API

compileapi1

Compilation against project API

compileapi2

Compilation against project API

compileapi3

Runtime against project

runtime1

Runtime against project

runtime2

Runtime against project

runtime3

Runtime against project

runtime4

Against external dependency

external

Gradle attributes

  • org.gradle.usage: indicates the usage of a variant

    • Example: java-api or java-runtime

  • org.gradle.category: indicates the category of a component

    • Example: library or platform

  • org.gradle.dependency.bundling: indicates how dependencies are handled

    • Example: regular or embedded

Gradle JVM attributes

  • org.gradle.jvm.version: indicates the minimal JVM version compatibility

    • Example: 6, 9 or 12

Usage of attributes

  • Can be declared on configurations

    • Defines an expected value on resolvable

    • Defines an exposed value on consumable

  • Can be declared on dependencies

    • Overrides the configuration value (if set)

  • Can be declared when retrieving artifact

Semantics and dependencies

Gradle has a strong modelling of dependencies:

  • Semantic difference between compilation and runtime

  • Semantic difference between building a library and building against a library

  • Ability for a module to produce more than one variant

Controlling dependency versions

Rich version constraints

Meaning of versions

  • What does it mean to say: "I depend on 1.1"

  • Does it mean it doesn’t work using 1.0?

  • Implicit statement: "I should work with 1.1+"

  • What if it’s not true?

Meaning of versions

  • Use latest.release?

  • Dependency on 1.2-beta-3: is beta important?

  • Dependency on snapshots…​

Enter rich version constraints

dependencies {
   implementation(project(":gitutils"))
   implementation("info.picocli:picocli") {
       version {
          strictly("[3.9, 4.0[")
          prefer("3.9.5")
       }
       because("Provides command-line argument parsing")
   }
}

Version constraints

  • Legacy notation (without version block) translates to require

  • require: doesn’t accept any lower version, upgrades are acceptable

  • strictly: if any other version in the graph disagrees, fails

  • prefer: weak constraint, if nobody else cares, choose this version

How to choose?

rich versions 1

How to choose?

rich versions 2

Special versions

  • Snapshots

  • Ranges ([1.0, [, [1.1, 1.4], …​)

  • Head selectors (latest.release, latest.integration)

  • Cached to avoid too many repository lookups

Control over dynamic versions

    configurations.all {
        resolutionStrategy.cacheDynamicVersionsFor 0, 'seconds'
    }

  • Warning: significant impact on performance!

Control over dynamic versions

  • --refresh-dependencies: has impact only on dynamic/changing modules

  • Absolutely no relationship with Maven local repository or Gradle artifact cache

Dependency locking

If you use dynamic versions ([1.0, [, 1.+, latest.release, …​):

  • Builds become non reproducible

  • Solution: dependency locking

Dependency locking

  • Lock a single configuration

configurations {
   compileClasspath {
      resolutionStrategy.activateDependencyLocking()
   }
}

Dependency locking

  • Convenience for locking all configurations

dependencyLocking {
    lockAllConfigurations()
}

Controlling dependency versions: dependency constraints

Direct dependencies vs transitive dependencies

  • Should be used to tell something about the project itself

    • What you directly use in code

  • What if you need to say something about a transitive dependency?

Dependency constraints

  • A dependency constraint tells something about modules found in the graph

  • Doesn’t matter how deep in the graph they are

  • Can be used to upgrade transitives

  • Affects resolution result if and only if module seen in graph

dependencies {
    constraints {
        implementation("org.slf4j:slf4j-api:1.7.26")
    }
}

Dependency constraints

  • Can be used to implement recommendations

dependencies {
    constraints {
        implementation("org.slf4j:slf4j-api:1.7.26")
        implementation("org.apache:commons-lang3:3.3.0")
    }
    dependencies {
        implementation("org.slf4j:slf4j-api") // no version
    }
}

Dependency constraints

  • Participate in the graph

  • They do not override preferences

    • They introduce additional constraints on versions

    • Conflict resolution discussed in 2d part of webinar

Structuring your build

Use case: commons dependencies

  • Often subprojects in a repository have to share the same versions

  • How can we:

    • Define common dependency versions

    • Share the result

Meet the Java Platform plugin

  • A Java Platform defines a set of dependency constraints

  • May have different api and runtime constraints

  • Optionally defines dependencies

Step 1: define a Java Platform subproject

platform/build.gradle.kts
plugins {
   `java-platform`
}

dependencies {
    constraints {
       api("org.slf4j:slf4j-api:1.7.26")
       api("org.slf4j:slf4j-simple:1.7.26")
    }
}

Step 2: refactor subprojects

cli/build.gradle.kts
dependencies {
   api(platform(project(":platform")))

   implementation("org.slf4j:slf4j-api") // <-- no version here
}

Step 3: extract versions

platform/build.gradle.kts
plugins {
   `java-platform`
}

// May be moved to `buildSrc`
object Deps {
   val slf4jVersion = "1.7.26"
}

dependencies {
    constraints {
       api("org.slf4j:slf4j-api:${Deps.slf4jVersion}")
       api("org.slf4j:slf4j-simple:${Deps.slf4jVersion}")
    }
}

Publication

  • When published, platforms are similar to Maven BOMs

  • Actually published as BOMs for Maven consumers

  • and published "as is" for Gradle consumers

Wrap up

Sneak peak at Part 2

  • Handling version conflicts

  • Dealing with bad metadata

  • Dependency alignment

  • Capabilities

  • Resolution rules

  • …​

To be confirmed: June 6th

Questions?