: Dependency Management with Gradle

Dependency Management with Gradle

Publishing and release strategies
Cédric Champeau (@CedricChampeau) & Louis Jacomet (@ljacomet) - Gradle Inc.

Who are we?

Dependency management team

Agenda

Publishing basics
Customizing publications
The Gradle component model
Published metadata
Versioning

Publishing Basics

Publishing building blocks

The plugins: maven-publish and ivy-publish
The concept of a publication
One or more target repositories

This presentation does not cover the legacy publication system, deprecated in Gradle 6.0

Basic publishing demo

Plugin and basic information

plugins {
    `java-library`
    `maven-publish`
}

group = "org.test"
version = "0.0.1"

Adding the publication

publishing {
    publications {
        create<MavenPublication>("main") {
            from(components["java"])
        }
    }
}

Adding the repository

publishing {
    repositories {
        maven {
            name = "artifactory"
            url = uri("http://localhost:8081/artifactory/example-repo-local")
        }
    }
}

Multiple repositories can be defined that way, in case you need to publish to more than one location.

Adding a local test repository

maven {
    name = "testRepo"
    url = uri("$buildDir/repo")
}

This test repository can be useful to review the produced publication files, before uploading them to their final destination.

Publication tasks

The publication plugins will define a number of tasks based on the declared publications and repositories:

publish<Publication>PublicationTo<Repository>Repository to publish a specific publication to a specific repository
publishAllPublicationsTo<Repository>Repository to publish all publications to a specific repository
publish to publish all publications to all repositories

More publication tasks

publishToMavenLocal to publish all maven publications to the Maven local repository
- This is not executed when using publish
generate<MetadataType>FileFor<Publication>Publication to generate the metadata file (Ivy, Maven or Gradle) for a specific publication.

Signing publications

Requires the signing plugin

signing {
    setRequired({ !version.toString().endsWith("-SNAPSHOT") })
    useGpgCmd()
    sign(publishing.publications["main"])
}

Publication result

The artifacts of the selected component
- Usually the JAR file for a Java project
The metadata of the selected component
- Maven or Ivy depending on the repository type
- Gradle Module Metadata by default starting with Gradle 6.0
Extra files
- Checksums: MD5, SHA1, SHA256, SHA512 (last two with Gradle 6.0)
- Signatures if signing configured

Publishing additional artifacts

Use cases

Publication of javadoc and sources JARs
Your project defines additional archives
- Test fixtures
- Optional features

In all the cases above, we want to publish not only the added artifact but also its relevant metadata.

Extra artifacts demo

Publishing javadoc and sources

java {
    withJavadocJar()
    withSourcesJar()
}

Produces and attaches the JAR to the publication …
As variants, with the right variant attributes

Using test fixtures

plugins {
    `java-test-fixtures`
}
dependencies {
    testFixturesApi("org.assertj:assertj-core:3.13.2")
}

Gives you an extra source set: testFixtures
Gives you configurations to declare dependencies: testFixturesApi and testFixturesImplementation
Creates the necessary configurations for compiling, running and consuming the test fixtures

Publishing test fixtures

All wired for you. Nothing to do.

Details in later sections

Optional features

Additional features of your library, that may require:
- Additional dependencies
- Additional artifact
while still using a single project for development

Test fixtures are one use case for this concept

Optional features demo

Optional feature declaration

java {
    registerFeature("optionalFeature") {
        usingSourceSet(sourceSets["main"])
    }
}

Source set choice indicates whether optional aspect is bundled with the main JAR or as an additional artifact
Creates the configurations for dependency declarations and wires them properly

Publishing a feature

All wired for you. Nothing to do.

Details in later sections

Consuming a feature

implementation("org.test:publish-feature:0.0.1") {
    capabilities {
        requireCapability("org.test:publish-feature-optional-feature")
    }
}

Selection happens through the use of capabilities

Handling mutually exclusive features

java {
    registerFeature("mysqlSupport") {
        usingSourceSet(sourceSets["main"])
        capability("org.gradle.demo", "producer-db-support", "1.0")
        capability("org.gradle.demo", "producer-mysql-support", "1.0")
    }
    registerFeature("postgresSupport") {
        usingSourceSet(sourceSets["main"])
        capability("org.gradle.demo", "producer-db-support", "1.0")
        capability("org.gradle.demo", "producer-postgres-support", "1.0")
    }
}

Since only one module can provide a given capability, sharing a capability forces consumer to chose only one capability provider.

Complete documentation

The component model

Java component

from(components["java"])

The component model

Creates safe boundaries between projects

Ground for the publication model

Artifacts are attached to configurations
which are exposed as variants
which belong to a component

Sharing task outputs

Classic (but wrong) answer:

project(":subproject").tasks.getByName("someTask")

Unsafe
Poorly modeled

Safe cross-project publications demo

Cross-project publications

Producer: create an outgoing configuration

val instrumentedJars by configurations.creating {
    isCanBeConsumed = true
    isCanBeResolved = false
    // If you want this configuration to share the same dependencies,
    // otherwise omit this line
    extendsFrom(
            configurations["implementation"],
            configurations["runtimeOnly"])
}

Cross-project publications

Producer: attach the artifact to the outgoing configuration

val instrumentedJar = tasks.register<Jar>("instrumentedJar") {
    archiveClassifier.set("instrumented")
}
instrumentedJars.outgoing.artifact(instrumentedJar)

Cross-project publications

Consumer: declare a resolvable configuration

val instrumentedClasspath by configurations.creating {
    isCanBeConsumed = false
    isCanBeResolved = true
}

Cross-project publications

Consumer: declaring a dependency on the configuration

dependencies {
    instrumentedClasspath(project(mapOf(
        "path" to ":producer",
        "configuration" to "instrumentedJars")))
}

Cross-project publications: model improvement

Let’s improve modeling on the producer side
Producer:

plugins {
    `java-library`
    `instrumented-jars`
}

Cross-project publications: model improvement

Moved logic to buildSrc
Declaration of attributes

private fun Project.createOutgoingConfiguration(): Configuration {
    val instrumentedJars by configurations.creating {
        isCanBeConsumed = true
        isCanBeResolved = false
        attributes {
            attribute(Category.CATEGORY_ATTRIBUTE, namedAttribute(Category.LIBRARY))
            attribute(Usage.USAGE_ATTRIBUTE, namedAttribute(Usage.JAVA_RUNTIME))
            attribute(Bundling.BUNDLING_ATTRIBUTE, namedAttribute(Bundling.EXTERNAL))
            attribute(TargetJvmVersion.TARGET_JVM_VERSION_ATTRIBUTE, JavaVersion.current().majorVersion.toInt())
            attribute(LibraryElements.LIBRARY_ELEMENTS_ATTRIBUTE, namedAttribute("instrumented-jar"))
        }
    }
    return instrumentedJars
}

Cross-project publications: model improvement

Consumer: may now use regular dependency declarations

dependencies {
    implementation(project(":producer"))
    testImplementation("junit:junit:4.12")
}

Cross-project publications: model improvement

Consumer: asks for a different variant for test runtime

configurations {
    testRuntimeClasspath {
        attributes {
            attribute(LibraryElements.LIBRARY_ELEMENTS_ATTRIBUTE,
                    objects.named(LibraryElements::class.java, "instrumented-jar"))
        }
    }
}

Cross-project publications: model improvement

Consumer: now compile classpath and test runtime classpath use different variants

> Task :consumer:resolveInstrumentedClasses
[/producer/build/classes/java/main, junit-4.12.jar, hamcrest-core-1.3.jar]
[producer-instrumented.jar, junit-4.12.jar, hamcrest-core-1.3.jar]

Enriching existing components

Producer can add variants to existing components

        val javaComponent = components.findByName("java") as AdhocComponentWithVariants
        javaComponent.addVariantsFromConfiguration(outgoing) {
            // dependencies for this variant are considered runtime dependencies
            mapToMavenScope("runtime")
            // and also optional dependencies, because we don't want them to leak
            mapToOptional()
        }

Creating adhoc components

Or producer can create a full adhoc component

// create an adhoc component
val adhocComponent = softwareComponentFactory.adhoc("myAdhocComponent")
// add it to the list of components that this project declares
components.add(adhocComponent)
// and register a variant for publication
adhocComponent.addVariantsFromConfiguration(outgoing) {
    mapToMavenScope("runtime")
}

Deeper control of publications

Filtering out variants for publication

Sometimes you don’t need to publish all variants for external consumption
Example: test fixtures can be used between projects but shouldn’t be available from external contributors

val javaComponent = components["java"] as AdhocComponentWithVariants
javaComponent.withVariantsFromConfiguration(
        configurations.testFixturesApiElements) {
   skip()
}

Extra publishing plugins

Artifactory plugin
- Leverages the maven-plugin
- Adds "build information" when uploading to Artifactory

plugins {
  id("com.jfrog.artifactory") version "4.10.0"
}

No support for Gradle Module Metadata yet!

Extra publishing plugins

Bintray plugin
- Leverages the artifactory plugin
- Provides way to publish to JCenter
- Provides way to synchronize to Maven Central (optional)

plugins {
  id("com.jfrog.bintray") version "1.8.4"
}

No support for Gradle Module Metadata yet!

Metadata

Supported metadata formats

Apache Maven™ POM file
- Only in Maven repositories
Apache Ivy™ ivy.xml file
- Only in Ivy repositories
Gradle Module Metadata file
- In both types of repositories in addition to one of the above
- Used by Gradle 6 if available instead of the above
- Carries information on variants, dependency constraints, custom artifacts

Gradle Module Metadata

junit-jupiter-api/
├── junit-jupiter-api-5.6.0.pom ('published-with-gradle-metadata' marker)
├── junit-jupiter-api-5.6.0.module
├── junit-jupiter-api-5.6.0.jar
├── junit-jupiter-api-5.6.0-javadoc.jar
├── junit-jupiter-api-5.6.0-sources.jar

Information in module missing in pom includes:
- Variants with the knowledge about javadoc.jar and sources.jar
- Constraint on other JUnit 5.6.0 modules to align versions

Published variants with artifacts

junit-jupiter-api-5.6.0.module (excerpt)

"variants": [
  { "name": "apiElements" ... },
  { "name": "runtimeElements" ... },
  { "name": "javadocElements",
    "attributes": {
      "org.gradle.category": "documentation",
      "org.gradle.docstype": "javadoc",
    },
    "files": [
      { "url": "junit-jupiter-api-5.6.0-javadoc.jar" ... }
    ]},
  { "name": "sourcesElements" ... },

build.gradle.kts (consumer example)

val javadoc by configurations.creating {
    attributes.attribute(CATEGORY_ATTRIBUTE, objects.named(DOCUMENTATION))
    attributes.attribute(DOCS_TYPE_ATTRIBUTE, objects.named(JAVADOC))
}
dependencies { javadoc("org.junit.jupiter:junit-jupiter-api:5.6.0") }
tasks.create("doSomethingWithJavadocs") { doLast { javadoc.files } }

Published platform dependency

junit-jupiter-api-5.6.0.module (excerpt)

"variants": [
  {
    "name": "apiElements",
    "attributes": { "org.gradle.category": "library", ... },
    "dependencies": [
      {
        "group": "org.junit",
        "module": "junit-bom",
        "version": { "requires": "5.6.0" },
        "attributes": { "org.gradle.category": "platform" },
      }
    ]
  },

Published platform (BOM)

junit-bom-5.6.0.module (excerpt)

"variants": [
  {
    "name": "apiElements",
    "attributes": { "org.gradle.category": "platform", ... },
    "dependencyConstraints": [
      {
        "group": "org.junit.jupiter",
        "module": "junit-jupiter-engine",
        "version": { "requires": "5.6.0" }
      }
    ]
  },

build.gradle.kts (consumer example)

dependencies {
    implementation("org.junit.jupiter:junit-jupiter-api:5.6.0")
    //                         ↑↑↑ brings in 'junit-bom:5.6.0'
    implementation("org.junit.jupiter:junit-jupiter-engine:5.5.2")
    //    ↑↑↑ thanks to 'junit-bom:5.6.0' it will align to 5.6.0
}

Metadata format features

	Gradle	POM	Ivy
Dependency constraints	✔	X*	X
Dependencies on platforms	✔	X*	X
Rich version constraints	✔	X	X

Gradle

POM

Ivy

Dependency constraints

✔

Dependencies on platforms

✔

Rich version constraints

✔

* <dependencyManagement> blocks are added to poms

Metadata format features

	Gradle	POM	Ivy
Component capabilities	✔	X	X
Feature variants	✔	X**	✔
Custom components	✔	X**	X

Gradle

POM

Ivy

Component capabilities

✔

Feature variants

✔

X**

✔

Custom components

✔

X**

** Artifacts have to be addressed directly by consumer (e.g. by classifier)

Metadata with custom variants

guava-28.1.module (how could it look like?)

"variants": [
  { "name": "jdk6ApiElements", "dependencies": [ ... ],
    "attributes": { "org.gradle.jvm.version": 6, ... },
    "files": [{ "url": "../28.1-android/guava-28.1-android.jar" }]
  },
  { "name": "jdk8ApiElements", "dependencies": [ ... ],
    "attributes": { "org.gradle.jvm.version": 8, ... },
    "files": [{ "url": "../28.1-jre/guava-28.1-jre.jar" }]
  },
  { "name": "jdk6RuntimeElements" ... },
  { "name": "jdk8RuntimeElements" ... },

build.gradle.kts (consumer example)

dependencies {
    implementation("com.google.guava:guava:28.1-jre")
}
java {
    // 6 or 7 gets 'android.jar', 8+ gets 'jre.jar', <6 fails
    targetCompatibility = JavaVersion.VERSION_1_6
}

Strategies for publication

Declared vs published

Your build file declares a number of versions
- e.g., commons-lang3:3.3
Not necessarily the resolved one:
- e.g., commons-lang3:3.3 → 3.4 (conflict resolution)

Declared vs published

What about dynamic dependencies?
e.g. commons-lang3:3.+
Gradle lets you choose between two modes: declared or resolved versions
- defaults to the declared versions

Declared versions

Pros
- Effectively what you need
- Maximum expressiveness thanks to rich versions
Cons
- Requires diligence in what is declared
- Result may be very different

Resolved versions

Pros
- What you resolve is what you get
- Stable resolution for consumers (no dynamic)
Cons
- Lossy mapping (ranges disappear, for example)
- Maybe unintentional
- Always approximate (what resolution to use?)
- Maybe incorrect (e.g. substitutions)

Resolved rich versions

Makes the mapping lossy for Gradle as well
flattens to require or strictly
- if the declared version is strictly, the resolved is as well

How to

publishing {
    publications {
        create<MavenPublication>("mavenJava") {
            versionMapping {
                usage("java-api") {
                    fromResolutionOf("runtimeClasspath")
                }
                usage("java-runtime") {
                    fromResolutionResult()
                }
            }
        }
    }
}

Relationship with dependency locking

Locking guarantees same resolution result
But lock files are unknown to consumers
Using locking + resolved result makes sense

Future work on resolved versions

More strategies
- still publish ranges, prefer the resolved version
- fail if resolved version disagrees with declared version

Release strategies

Gradle is agnostic

Version is simply a project property
Open as to where to publish

Which version to release?

-SNAPSHOT, based on the Maven convention
- Changing aspect is a problem for reproducibility
Unique releases
- Keeping up with releases requires work / automation

Where to publish?

Maven local repository
- Convenient but not without issues
- Consider using repository content filtering to limit its impact
Repository (local or remote)
- Requires more infrastructure

Strategies

Live with the -SNAPSHOT limitations
Use dynamic versions and locking for easier automated updates
- Add publication of resolved versions

Integrations

During development, integration can use:
- Publication and releases
- Composite builds
- Source dependencies

Plugins can help

nebula-release-plugin supports multiple release types
- Integrates with Git
- final, candidate, snapshots, unique snapshots, …
- Give it a try!

Multi repository development

This is really where all these concepts come into play inside an organisation.

Stay tuned for a future webinar on this, and how Gradle can help!

Conclusion

Publishing

Switch to the maven-publish or ivy-plugin
Supports publishing POM/Ivy + Gradle Module Metadata
Gradle Module Metadata → richest
Flexible support for custom components
- already used by 3rd party plugins (Kotlin Native, Android, …)