Host your maven artifacts using Google Cloud Storage

If you use Google Cloud and you use Java for your projects then Google Cloud Storage is a great place to host your teams artifacts.

It is easy to setup and pretty cheap. Also it is much simpler than setting one of the existing repository options (jfrog, nexus, archiva etc) if you are not particularly interested in their features.

To get started you need to specify a maven wagon which supports google cloud storage.
We will use the Google storage wagon.

Let’s get started by creating a maven project

mvn archetype:generate -DgroupId=com.test.apps -DartifactId=GoogleWagonTest -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false

We are going to add a simple service.

package com.test.apps;

public class HelloService {

    public String sayHello() {

        return "Hello";
    }
}

Then we are going to add the maven wagon which will upload and fetch our binaries to the google cloud storage.

    <build>
        <extensions>
            <extension>
                <groupId>com.gkatzioura.maven.cloud</groupId>
                <artifactId>google-storage-wagon</artifactId>
                <version>1.0</version>
            </extension>
        </extensions>
    </build>

Then we shall create the google cloud storage bucket which will host our artifacts.

Our bucket shall be called mavenrepository

Now that we have set up our bucket in google we shall set the distribution management on our maven project.

    <distributionManagement>
        <snapshotRepository>
            <id>my-repo-bucket-snapshot</id>
            <url>gs://mavenrepository/snapshot</url>
        </snapshotRepository>
        <repository>
            <id>my-repo-bucket-release</id>
            <url>gs://mavenrepository/release</url>
        </repository>
    </distributionManagement>

From the maven documentation

Where as the repositories element specifies in the POM the location and manner in which Maven may download remote artifacts for use by the current project, distributionManagement specifies where (and how) this project will get to a remote repository when it is deployed. The repository elements will be used for snapshot distribution if the snapshotRepository is not defined.

The next step is the most crucial and this has to to do with authenticating to google cloud.

You need to have the gcloud command line setup in your system and you must issue a login
‘gcloud auth login –brief’ with an account that has access to the bucket we created previously.
The other way is to use the GOOGLE_APPLICATION_CREDENTIALS environmental variable. You can use this GOOGLE_APPLICATION_CREDENTIALS in order to set the path to your google application credentials file.
The credentials file should also be able to access the bucket we created previously.

And now the easiest part which is deploying.

mvn deploy

Now since your artifact has been deployed you can use it in another repo by specifying your repository and your wagon.

    <repositories>
        <repository>
            <id>my-repo-bucket-snapshot</id>
            <url>gs://mavenrepository/snapshot</url>
        </repository>
        <repository>
            <id>my-repo-bucket-release</id>
            <url>gs://mavenrepository/release</url>
        </repository>
    </repositories>

    <build>
        <extensions>
            <extension>
                <groupId>com.gkatzioura.maven.cloud</groupId>
                <artifactId>google-storage-wagon</artifactId>
                <version>1.0</version>
            </extension>
        </extensions>
    </build>

That’s it! Next thing you know, your artifact will be downloaded by maven through google cloud storage and used as a dependency in your new project.

37 thoughts on “Host your maven artifacts using Google Cloud Storage

  1. Hey Emmanouil, Thanks for this helpful article. When you talk about adding the maven wagon, are you adding this to the pom.xml for the project? Or in the Maven settings? same question for distribution management. I am also having trouble seeing how the GoogleStorageWagon code from github comes into play.

  2. Did you get this working at all? I’m stuck at specifying the Google Cloud Storage project using the wagon.gs.project.[repository id] property. I have this set, but it always tries to access my bucket at a project id of 455054200496 which I dont even know where that number is coming from.

    1. I was able to get it working, the issue was with the permissions. It requires storage.buckets.list which is not obvious since the bucket is specified as part of the configuration, I dont know why it needs list permissions

  3. I run into the following exception
    “`
    Could not establish connection with google cloud
    com.google.cloud.storage.StorageException: Anonymous caller does not have storage.buckets.list access to project 116955912135.
    “`
    To clarify, my account does have access to google cloud, and I can run `gcloud` commands from the terminal.

    1. I had to create a new role with the permissions and add my service account user to the role. None of the default craator/view roles container the permission to list buckets

  4. “gcloud auth login” didn’t help.

    A better way to authenticate is to

    1. create a service account in GCP + create & download the access_key file for the service account

    2. make sure(or assign) the service account has the “Storage Object Admin Viewer” permission

    3. on your local machine, gcloud auth activate-service-account –key-file=

    4. verify that the service account has permission to list buckets – run : gsutil ls

    5. finally, “mvn deploy” should be successful.

  5. Hi Emmanouil, thanks for the article this has been really useful. Do you know if it is possible to use the Wagon to store archetypes and then generate from them? I’ve not been able to find information about how I can to this?

    Thanks

    Dan

  6. I am getting this error. Can you provide the solution. I already have issued a login by using this command ‘gcloud auth login –brief’ but still getting below error.

    [ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.7:deploy (default-deploy) on project altaf: Failed to deploy artifacts: Could not transfer artifact com.altaf:altaf:jar:1.0 from/to my-repo-bucket-release (gs://gupshupsms/release): Please configure you google cloud account by logging using gcloud and specify a default project -> [Help 1]

      1. I have authenticated in both the way.

        1. I already login with gcloud.
        2. I created one service account and downloaded JSON file then set the environment path to GOOGLE_APPLICATION_CREDENTIALS

        But still getting the same problem.
        [ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.7:deploy (default-deploy) on project altaf: Failed to deploy artifacts: Could not transfer artifact com.altaf:altaf:jar:1.0 from/to my-repo-bucket-release (gs://gupshupsms/release): Please configure you google cloud account by logging using gcloud and specify a default project -> [Help 1]

        I request you to please create a YouTube video on this topic. It will be very helpful to me.

  7. hello Emmanouil,

    i am unable to download dependencies from gbucket.

    push functionality is working.

    “[ERROR] Unresolveable build extension: Plugin com.gkatzioura.maven.cloud:google-storage-wagon:1.0 or one of its dependencies could not be resolved: Failed to read artifact descriptor for com.gkatzioura.maven.cloud:google-storage-wagon:jar:1.0: Could not transfer artifact com.gkatzioura.maven.cloud:google-storage-wagon:pom:1.0 from/to mirrorId (gs://mavenrepository-a1/): Cannot access gs://mavenrepository-a1/ with type default using the available connector factories: BasicRepositoryConnectorFactory: Cannot access gs://mavenrepository-a1/ using the registered transporter factories: WagonTransporterFactory: java.util.NoSuchElementException”

    1. The maven wagon is located on the maven repository.
      Based on the logs you try to download the maven wagon through your google repository, this should not happen google-storage-wagon should be downloaded through the maven central repo.

  8. I am trying to connect from eclipse/maven running on my windows laptop to maven repo on GCP storage. I am using GOOGLE_APPLICATION_CREDENTIALS env variable with a service account and appropriate permission.
    I have NOT installed gcloud command line in my laptop.
    Initially I was getting the “Anonymous caller” error. Once I corrected the key.json file the error stopped.
    It proceeded to on to give a SSL certificate error. I downloaded the http://www.googleapis.com cert and added it to the trust store to the mvn params. It is still erroring out with no SSL cert found for domain.
    Is it possible to connect without installing gcloud command line or am I missing something?

    1. I am past the SSL issue now.

      I am getting the following exception
      Caused by: java.lang.IndexOutOfBoundsException
      at java.base/java.nio.ByteBuffer.wrap(ByteBuffer.java:395)
      at org.eclipse.aether.transport.wagon.WagonTransferListener.transferProgress(WagonTransferListener.java:64)
      at com.gkatzioura.maven.cloud.listener.TransferListenerContainerImpl.lambda$fireTransferProgress$2(TransferListenerContainerImpl.java:75)
      at java.base/java.util.Vector.forEach(Vector.java:1387)
      at com.gkatzioura.maven.cloud.listener.TransferListenerContainerImpl.fireTransferProgress(TransferListenerContainerImpl.java:75)
      at com.gkatzioura.maven.cloud.transfer.TransferProgressImpl.progress(TransferProgressImpl.java:36)
      at com.gkatzioura.maven.cloud.transfer.TransferProgressFileInputStream.read(TransferProgressFileInputStream.java:65)
      at com.vorstella.shade.com.google.api.client.util.ByteStreams.copy(ByteStreams.java:51)
      at com.vorstella.shade.com.google.api.client.util.IOUtils.copy(IOUtils.java:94)
      at com.vorstella.shade.com.google.api.client.http.AbstractInputStreamContent.writeTo(AbstractInputStreamContent.java:72)
      at com.vorstella.shade.com.google.api.client.http.MultipartContent.writeTo(MultipartContent.java:107)
      at com.vorstella.shade.com.google.api.client.http.GZipEncoding.encode(GZipEncoding.java:49)
      at com.vorstella.shade.com.google.api.client.http.HttpEncodingStreamingContent.writeTo(HttpEncodingStreamingContent.java:51)
      at com.vorstella.shade.com.google.api.client.http.javanet.NetHttpRequest$DefaultOutputWriter.write(NetHttpRequest.java:76)
      at com.vorstella.shade.com.google.api.client.http.javanet.NetHttpRequest.writeContentToOutputStream(NetHttpRequest.java:156)
      at com.vorstella.shade.com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:117)
      at com.vorstella.shade.com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)
      at com.vorstella.shade.com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1011)
      at com.vorstella.shade.com.google.api.client.googleapis.media.MediaHttpUploader.executeCurrentRequestWithoutGZip(MediaHttpUploader.java:548)
      at com.vorstella.shade.com.google.api.client.googleapis.media.MediaHttpUploader.executeCurrentRequest(MediaHttpUploader.java:565)
      at com.vorstella.shade.com.google.api.client.googleapis.media.MediaHttpUploader.directUpload(MediaHttpUploader.java:360)
      at com.vorstella.shade.com.google.api.client.googleapis.media.MediaHttpUploader.upload(MediaHttpUploader.java:334)
      at com.vorstella.shade.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:508)
      at com.vorstella.shade.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
      at com.vorstella.shade.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:549)
      at com.vorstella.shade.com.google.cloud.storage.spi.v1.HttpStorageRpc.create(HttpStorageRpc.java:303)
      at com.vorstella.shade.com.google.cloud.storage.StorageImpl.create(StorageImpl.java:161)
      at com.gkatzioura.maven.cloud.gcs.GoogleStorageRepository.put(GoogleStorageRepository.java:101)

      I tried a simple hello world with
      File file = new File ( “MyFilePath/artifactid-versionno.txt” );
      Repository repository = new Repository(“MyRepo”, “gs://MyBucket/mavenrepository/snapshot”);
      googleStorageWagon.connect(repository);
      googleStorageWagon.put(file, “group.id/artifactid/vesionno/artifactid-versionno.txt”);

      It is working fine.

      But I am getting the above error if I do a mvn deploy

      I am using JDK 11.

  9. Not working for parent poms :
    I created parent pom. Built and deployed it successfully. Now I created a project which extends the parent pom. The deploy fails with
    org.eclipse.aether.transfer.NoTransporterException: java.util.NoSuchElementException
    role: org.apache.maven.wagon.Wagon
    roleHint: gs
    Looks like the maven is not able to download the parent pom from maven central, it is not able to go to gcp since the pom is not fully loaded yet for it to identify the wagon extension.

  10. Even after using the version 1.8 getting the error Checksum validation failed unable to retrieve the artifacts from the bucket .

    1. Yes, it has to do with the java binary from google which executes the operations behind the scenes. It is a warning and is not causing any issues with regards to working as a proper wagon implementation.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.