Upload and Download files to S3 using maven.

Throughout the years I’ve seen many teams using maven in many different ways. Maven can be used for many ci/cd tasks instead of using extra pipeline code or it can be used to prepare the development environment before running some tests.
Generally it is convenient tool, widely used among java teams and will continue so since there is a huge ecosystem around it.

The CloudStorage Maven plugin helps you with using various cloud buckets as a private maven repository. Recently CloudStorageMaven for s3 got a huge upgrade, and you can use it in order to download or upload files from s3, by using it as a plugin.

The plugin assumes that your environment is configured properly to access the s3 resources needed.
This can be achieved individually through aws configure

aws configure

Other ways are through environment variables or by using the appropriate iam role.

Supposing you want to download some certain files from a path in s3.

<build>
        <plugins>
            <plugin>
                <groupId>com.gkatzioura.maven.cloud</groupId>
                <artifactId>s3-storage-wagon</artifactId>
                <version>1.6</version>
                <executions>
                    <execution>
                        <id>download-one</id>
                        <phase>package</phase>
                        <goals>
                            <goal>s3-download</goal>
                        </goals>
                        <configuration>
                            <bucket>your-bucket</bucket>
                            <downloadPath>/local/download/path</downloadPath>
                            <keys>1.txt,2.txt,directory/3.txt</keys>
                        </configuration>
                    </execution>
                <executions>
            <plugin>
        <plugins>
</build>

The files 1.txt,2.txt,directory/3.txt once the execution is finished shall reside in the local directory specified
(/local/download/path).
Be aware that the file discovery on s3 is done with prefix, thus if you have file 1.txt and 1.txt.jpg both files shall be downloaded.

You can also download only one file to one file that you specified locally, as long as it is one to one.

                    <execution>
                        <id>download-prefix</id>
                        <phase>package</phase>
                        <goals>
                            <goal>s3-download</goal>
                        </goals>
                        <configuration>
                            <bucket>your-bucket</bucket>
                            <downloadPath>/path/to/local/your-file.txt</downloadPath>
                            <keys>a-key-to-download.txt</keys>
                        </configuration>
                    </execution>

Apparently files with a prefix that contain directories (they are fakes ones on s3) will downloaded to the directory specified in the form of directories and sub directories

                    <execution>
                        <id>download-prefix</id>
                        <phase>package</phase>
                        <goals>
                            <goal>s3-download</goal>
                        </goals>
                        <configuration>
                            <bucket>your-bucket</bucket>
                            <downloadPath>/path/to/local/</downloadPath>
                            <keys>s3-prefix</keys>
                        </configuration>
                    </execution>

The next part is about uploading files to s3.

Uploading one file

                    <execution>
                        <id>upload-one</id>
                        <phase>package</phase>
                        <goals>
                            <goal>s3-upload</goal>
                        </goals>
                        <configuration>
                            <bucket>your-bucket</bucket>
                            <path>/path/to/local/your-file.txt</path>
                            <key>key-to-download.txt</key>
                        </configuration>
                    </execution>

Upload a directory

                    <execution>
                        <id>upload-one</id>
                        <phase>package</phase>
                        <goals>
                            <goal>s3-upload</goal>
                        </goals>
                        <configuration>
                            <bucket>your-bucket</bucket>
                            <path>/path/to/local/directory</path>
                            <key>prefix</key>
                        </configuration>
                    </execution>

Upload to the root of bucket.

                    <execution>
                        <id>upload-multiples-files-no-key</id>
                        <phase>package</phase>
                        <goals>
                            <goal>s3-upload</goal>
                        </goals>
                        <configuration>
                            <bucket>your-bucket</bucket>
                            <path>/path/to/local/directory</path>
                        </configuration>
                    </execution>

That’s it! Since it is an open source project you can contribute or issue pull requests at github.

Advertisement

Behavioural Design Patterns: Template method

Previously we used the strategy pattern to in order to solve the problem of choosing various speeding algorithms based on the road type. The next behavioural design pattern we are going to use is the template method.
By using the template method we define the skeleton of the algorithm and the implementation of certain steps is done by subclasses.

Thus we have methods with concrete implementations, and methods without any implementation. Those methods will be implemented based on the application logic needed to be achieved.

Imagine the case of a coffee machine. There are many types of coffees and different ways to implement them, however some steps are common and some steps although they vary they also need to be implemented. Processing the beans, boiling, processing the milk, they are all actions that differ based on the type of coffee. Placing to a cup and service however are actions that do no differentiate.

package com.gkatzioura.design.behavioural.template;

public abstract class CoffeeMachineTemplate {

    protected abstract void processBeans();

    protected abstract void processMilk();

    protected abstract void boil();

    public void pourToCup() {
        /**
         * pour to various cups based on the size
         */
    }

    public void serve() {
        processBeans();
        boil();
        processMilk();
        pourToCup();
    }

}

Then we shall add an implementation for the espresso. So here’s our espresso machine.

package com.gkatzioura.design.behavioural.template;

public class EspressoMachine extends CoffeeMachineTemplate {

    @Override
    protected void processBeans() {
        /**
         * Gring the beans
         */
    }

    @Override
    protected void processMilk() {
        /**
         * Use milk to create leaf art
         */
    }

    @Override
    protected void boil() {
        /**
         * Mix water and beans
         */
    }
}

As you see we can create various coffee machine no matter how different some steps might be.
You can find the sourcecode on github.

Behavioural Design Patterns: Strategy

Previously we used the state in order to add some functionality to an application based on the user state. Our next behavioural design pattern is Strategy.
The strategy pattern enables us to select an algorithm at runtime. Based on the instructions our program will pick the most suitable algorithm instead of implementing an algorithm directly. This makes our codebase more flexible and keep it clean from any extra logic.

Our example shall evolve around vehicles and the speeding that is allowed based on the type of road. For example if a vehicle is on a four lane road the speed would be way different than being on an urban area road.
So we are actually going to implement the strategy patterns with regards to speeding.

We will start with the speeding interface.

package com.gkatzioura.design.behavioural.strategy;

public interface Speeding {

    Double adjustSpeed(Double currentSpeed);

}

Then we shall create some implementations based on the road type.
The four lane speeding implementation adjusts the speeding when driving on a four lane.

package com.gkatzioura.design.behavioural.strategy;

public class FourLaneSpeeding implements Speeding {

    private static final Double upperLimit = 50d;

    @Override
    public Double adjustSpeed(Double currentSpeed) {
        if(currentSpeed>upperLimit) {
            currentSpeed = upperLimit;
        }

        System.out.println("Speed adjusted at "+currentSpeed);

        return currentSpeed;
    }

}

The urban area speeding implementation adjusts the speeding when driving on a rural road.

package com.gkatzioura.design.behavioural.strategy;

public class UrbanAreaSpeeding implements Speeding {

    private static final Double upperLimit = 30d;

    @Override
    public Double adjustSpeed(Double currentSpeed) {
        if(currentSpeed>upperLimit) {
            currentSpeed = upperLimit;
        }

        System.out.println("Speed adjusted at "+currentSpeed);

        return currentSpeed;
    }

}

And then we shall create the vehicle class.

package com.gkatzioura.design.behavioural.strategy;

public class Vehicle {

    private Speeding speeding;
    private Double currentSpeed;

    public void drive() {

        speeding.adjustSpeed(currentSpeed);

        /**
         * Driving related actions.
         */
    }

    public void setSpeeding(Speeding speeding) {
        this.speeding = speeding;
    }

    public void setCurrentSpeed(Double currentSpeed) {
        this.currentSpeed = currentSpeed;
    }
}

As you can see the vehicle shall change its speeding strategy based on the road driving.
Let’s put them all together.

package com.gkatzioura.design.behavioural.strategy;

public class Strategy {

    public static void main(String[] args) {
        Vehicle vehicle = new Vehicle();

        vehicle.setCurrentSpeed(70d);
        
        vehicle.drive();
        
        /**
         * Changed route
         */
        
        vehicle.setSpeeding(new FourLaneSpeeding());

        vehicle.drive();

        /**
         * Changed route
         */
        
        vehicle.setSpeeding(new UrbanAreaSpeeding());

        vehicle.drive();
    }
}

That’s all for now! You can find the sourcecode on github.