Run you first gatling load test using scala.

Gatling is a neat tool. You can create your load tests by just coding in scala. Jmeter allows you to do so through a plugin or beanshell but it is not as direct as the way gatling does so.

I will start by adding the gatling plugin

addSbtPlugin("io.gatling" % "gatling-sbt" % "3.0.0")

The next step is to changed the build.sbt


version := "0.1"
scalaVersion := "2.12.8"

enablePlugins(GatlingPlugin)

scalacOptions := Seq(
  "-encoding", "UTF-8", "-target:jvm-1.8", "-deprecation",
  "-feature", "-unchecked", "-language:implicitConversions", "-language:postfixOps")
libraryDependencies += "io.gatling.highcharts" % "gatling-charts-highcharts" % "3.1.2" % "test,it"
libraryDependencies += "io.gatling"            % "gatling-test-framework"    % "3.1.2" % "test,it"

The above are no different than what you can find on the official site when it comes to sbt commands and gatling.

Our next step is to add a simple http test. Be aware that you should add it in the directories src/test or src/it since, as it is instructed from the sbt dependencies for the binaries to take effect on these directories.

I shall put this test on src/test/scala/com/gkatzioura/BasicSimulation.scala

package com.gkatzioura

import io.gatling.core.Predef._
import io.gatling.http.Predef._
import scala.concurrent.duration._

class BasicSimulation extends Simulation {

  val httpConf = http.baseUrl("http://yourapi.com")
      .doNotTrackHeader("1")

  val scn = scenario("BasicSimulation")
    .exec(http("request_1")
    .get("/"))
    .pause(5)

  setUp(scn.inject(atOnceUsers(1))).protocols(httpConf)
}

Afterwards testing is simple. You go to sbt mode and execute the test.

sbt
>gatling:testOnly com.gkatzioura.BasicSimulation
>gatling:test

The first command instructs to run just one test, the second one shall run everything.

That’s it! Pretty simple.

Migrate your elastic beanstalk workers to docker containers

Amazon Elastic Beanstalk is one of the most popular services that aws provides. Elastic beanstalk comes with two options, the worker and the web application.

The worker application consumes the messages from a sqs queue and process them. If the process was successful the message is removed from the queue, if not the message shall remain in the queue or after some failed attempts it will go back to a dead letter queue. If you want to get more into elb I have made a tutorial on deploying you spring application using elb and cloudformation.

Elastic beanstalk workers are really great because they are managed, they can be scaled up/down depending on your workloads and they provide a wide variety of development environments like java, node.js and also you can use docker images.

Although elastic bean can work wonders if you have an aws based infrastructure you might face some issues when you will try to move to a container based infrastructure using a container orchestration engine.

Most probably your containerized worker application will work seamlessly without any extra configuration, however you need to find an alternative for the agent which dispatches the queue messages to your application.

In order to make things simple I implemented a mechanism which retrieves the messages from the queue and sends them to the worker application.

The container-queue-worker projects aims to provide an easy way to migrate your elastic beanstalk workers to a docker orchestration system.

Since the solution is Scala based it can either be used as a standalone jvm application or it can be run in a container using the image from dockerhub.

Once set up what you need to add the routing configurations.

This can be done using environmental variables

WORKER_TYPE=sqs
WORKER_SERVER_ENDPOINT=http://{docker-service}
WORKER_AWS_QUEUE_ENDPOINT=http://{amazon queue endpoint}

Or if you use it as a container you can add a config file on the /etc/worker/worker.conf path.

worker {
  type =  sqs
  server-endpoint = http://{docker-service}
  aws {
    queue-endpoint =  http://{amazon queue endpoint}
  }
}

In order to make thing easier for you I added a docker compose file simulating the desired outcome.

version: '3.5'
networks:
  queue-worker-network:
    name: queue-worker-network
services:
  worker-server:
    build:
      context: ./worker-server
      dockerfile: Dockerfile
    ports:
      - 8080:8080
    networks:
      - queue-worker-network
  elasticmq:
    build:
      context: ./elasticmq
      dockerfile: Dockerfile
    ports:
      - 9324:9324
    networks:
      - queue-worker-network
  container-queue-worker:
    image: gkatzioura/container-queue-worker:0.1
    depends_on:
      - elasticmq
      - worker-server
    environment:
      WORKER_TYPE: sqs
      WORKER_SERVER_ENDPOINT: http://worker-server:8080/
      WORKER_AWS_QUEUE_ENDPOINT: http://elasticmq:9324/queue/test-queue
      AWS_DEFAULT_REGION: eu-west-1
      AWS_ACCESS_KEY_ID: access-key
      AWS_SECRET_ACCESS_KEY: secret-key
    networks:
      - queue-worker-network

Dockerize your Scala application

Dockerizing a Scala application is pretty easy.

The first concern is creating a fat jar. Now we all come from different backgrounds including maven/gradle and different plugins that handle this issue.
If you use sbt the way to go is to use the sbt-assembly plugin.

To use it we should add it to our project/plugins.sbt file. If the file does not exist create it.

logLevel := Level.Warn

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.6")

So by executing

sbt clean assembly

We will end up with a fat jar located at the target/scala-**/**.jar path.

Now the easy part is putting our application inside docker, thus a Dockerfile is needed.

We will use the openjdk alpine as a base image.

FROM openjdk:8-jre-alpine

ADD target/scala-**/your-fat-jar app.jar

ENTRYPOINT ["java","-jar","/app.jar"]

The above approach works ok and gives the control needed to customize your build process.
For a more bootstraping experience you can use the sbt native packager.

All you need to do is to add the plugin to project/plugins.sbt file.

logLevel := Level.Warn

addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.3.4")

Then we specify the main class of our application and enable the Java and Docker plugins from the native packager at the build.sbt file.

mainClass in Compile := Some("your.package.MainClass")

enablePlugins(JavaAppPackaging)
enablePlugins(DockerPlugin)

The next step is to issue the sbt command.

sbt docker:publishLocal

This command will build your application, include the binaries needed to the jar, containerize your application and publish it to your local maven repo.

Run code on startup with Play and Scala

Depending on various projects, sometimes there is the need to execute some actions on initialization just before our application starts to serve requests.

It was a common practice to call the functions that we wanted to get executed through GlobalSettings, however it is not recommended.

The other way around to achieve this is to implement a class which will be injected and thus add the code that we want to get executed on the class constructor.

We might believe that it is sufficient to implement a class which shall use the @Singleton annotation.

For example

@Singleton
class StartUpService {

    //The code that needs to be executed

}

But this will not work as expected since our component instances on play are created lazily when they are needed.

Instances are created lazily when they are needed. If a component is never used by another component, then it won’t be created at all. This is usually what you want. For most components there’s no point creating them until they’re needed. However, in some cases you want components to be started up straight away or even if they’re not used by another component. For example, you might want to send a message to a remote system or warm up a cache when the application starts. You can force a component to be created eagerly by using an eager binding.

To tackle this problem, our singleton has to be initialized eagerly. To achieve an eager initialization we will define an eager binding.

To define an eager binding we have to implement a class that extends the AbstractModule and then bind our service as an eager singleton.

package com.gkatzioura.eager

import com.google.inject.AbstractModule
import com.google.inject.name.Names

// A Module is needed to register bindings
class EagerLoaderModule extends AbstractModule {
  override def configure() = {

    bind(classOf[StartUpService]).asEagerSingleton
  }
}

Then we have to enable our module by declaring so to our conf/application.conf configuration.

play.modules.enabled += "com.gkatzioura.eager.EagerLoaderModule"

The above approach creates a module by defining it explicitly. The other approach is to use the default functionality where Play will load any class called Module that is defined in the root package.

In conclusion, play gives us the option to execute certain functions once the application has started. To do so we need to implement a component as an eager singleton. Skip the GlobalSettings as it is not advised by the official documentation.

Play and SBT basics

Previously we had an introduction to sbt, its default tasks and how to add extra tasks.

Play comes with the sbt console. The SBT console is a development console based on sbt that allows you to manage a Play application’s complete development cycle.

Let us create a play application using sbt and see the commands provided.

sbt new playframework/play-scala-seed.g8

[warn] Executing in batch mode.
[warn]   For better performance, hit [ENTER] to switch to interactive mode, or
[warn]   consider launching sbt without any commands, or explicitly passing 'shell'
[info] Set current project to development (in build file:/home/gkatzioura/Development/)

This template generates a Play Scala project 

name [play-scala-seed]: PlayExample
organization [com.example]: com.gkatzioura
scala_version [2.11.11]: 
play_version [2.5.14]: 
scalatestplusplay_version [2.0.0]: 

The result is a play project named playexample. By opening with an editor the project/plugins.sbt we can see the sbt plugin added to our project.
Therefore we are going to check what are the extra tasks that the sbt plugin provides and some tasks that can be generally helpful.

cd playexample; sbt 
[PlayExample] $ <tab><tab>
Display all 511 possibilities? (y or n)
...
h2-browser  
...                                                                                                                                                                                                                              
playStop                                           
playUpdateSecret     
playGenerateSecret                              
...
stage
...
  • playStop – Stop Play, if it has been started in non blocking mode
  • playGenerateSecret – This will generate a new secret that you can use in your application. For example the application secret can be used for Signing session cookies and CSRF tokens or
    built in encryption utilities
  • playUpdateSecret – Update the application conf to generate an application secret
  • stage – Create a local directory with all the files laid out as they would be in the final distribution.
  • h2-browser – Opens an h2 database browser. Pretty useful if you are using h2 for development

Those are some of the commands that you might use often. However if you want extra information, you can always type help play.

[PlayExample] $ help play

playExternalizeResources

  Whether resources should be externalized into the conf directory when Play is packaged as a distribution.

playCommonClassloader

  The common classloader, is ...
...

I’ve compiled a cheat sheet that lists some helpful sbt commands.
Sign up in the link to receive it.

SBT basics

Sbt is the de facto build tool in the Scala community.
Being used to other build tools you will be familiar with the commands

  • clean – Deletes files produced by the build, such as generated sources, compiled classes, and task caches.
  • compile – Compiles sources
  • test – Executes all tests
  • package – Produces the main artifact, such as a binary jar. This is typically an alias for the task that actually does the packaging.
  • help – Displays this help message or prints detailed help on requested commands (run ‘help ‘).
  • console – Starts the Scala interpreter with the project classes on the classpath.

Then we have extra commands suchs as

    • run – Runs a main class, passing along arguments provided on the command line.
    • tasks – Lists the tasks defined for the current project.
    • reload – (Re)loads the current project or changes to plugins project or returns from it.
    • console – Starts the Scala interpreter with the project classes on the classpath.

A key functionality is the new command.

For example by using new, we can create a project from the template specified (for example scala-seed.g8) using giter8.

sbt new scala/scala-seed.g8

...
Minimum Scala build. 

name [My Something Project]: hello

Template applied in ./hello

The previous snippet creates a project called hello.

The file build.sbt holds a sequence of key-value pairs called setting expressions. The left-hand side is a key and the right hand side is the body.
There are three types of keys.

  • SettingKey[T]: a key for a value computed once (the value is computed
    when loading the subproject, and kept around).
  • TaskKey[T]: a key for a value, called a task, that has to be recomputed
    each time, potentially with side effects.
  • InputKey[T]: a key for a task that has command line arguments as input.

For example if we want to add and extra task, to our previous project,  which prints hello, then we shall add the following lines to the build.sbt file.

import Dependencies._

lazy val hello = taskKey[Unit]("An example task")

lazy val root = (project in file(".")).
settings(
hello := { println("Hello!") },
inThisBuild(List(
organization := "com.example",
scalaVersion := "2.12.2",
version := "0.1.0-SNAPSHOT"
)),
name := "Hello",
libraryDependencies += scalaTest % Test
)

We can either run the task or ask for more info about the task.

>sbt
> hello
Hello!
[success] Total time: 0 s, completed May 1, 2017 6:08:36 PM
> help hello
An example task
> 

Depending on the project and the plugin used, there would be extra tasks and settings defined.

On the next post we will check play and sbt integration, and some basic commands.

I’ve compiled a cheat sheet that lists some helpful sbt commands.
Sign up in the link to receive it.

SQL Data Access in Play using Scala

Today’s modern application frameworks come with a promise of easy sql data access. There is no wonder why we have so many frameworks that make it easier to issue queries and handle transactions. SQL is the lingua franca of most applications when it comes to databases.

Play comes with the JDBC plugin. We encountered the JDBC plugin previously in order to modify our database schema.

The first step is to include the jdbc and the evolutions module.

libraryDependencies += evolutions
libraryDependencies += jdbc

Then we shall define the connection string needed. We will use a simple h2 database. The configuration is added at the application.conf.

db.default.driver=org.h2.Driver
db.default.url="jdbc:h2:/tmp/defaultdatabase"

Then we add a script that creates the users table.

# Users schema

# --- !Ups

CREATE TABLE users (
    id bigint(20) NOT NULL AUTO_INCREMENT,
    email varchar(255) NOT NULL,
    first_name varchar(255) NOT NULL,
    last_name varchar(255) NOT NULL,
    PRIMARY KEY (id),
    UNIQUE KEY (email)
);

# --- !Downs

DROP TABLE users;

Before creating our repository class let’s check what the jdbc plugin provides us with.

We have the plain getConnection method, responsible for returning a jdbc connection. This is similar to the DataSource.getConnection from Java. Thus pay extra attention since you must close the connection.

val connection = db.getConnection()
connection.close()

Next method is withConnection. By using withConnection you get Play to manage the connection for you. All you have to do is pass a block of code with jdbc actions.

  def fetchUsers(): List[User] = {

    db.withConnection { conn =>

      val stmt = conn.createStatement
      var rs = stmt.executeQuery("SELECT*FROM users");
      val listBuffer = ListBuffer[User]()

      while(rs.next()) {

        listBuffer.append(User(Option(rs.getLong("id")),rs.getString("email"),rs.getString("first_name"),rs.getString("last_name")))
      }

      listBuffer.toList
    }
  }

As you can see above, we’ve just returned back a list of our user entries.

And last but not least withTransaction. You’ve guessed right, what you receive back is a connection with autocommit set to false.

  def addUser(user:User): User = {

    db.withTransaction { conn =>
      val stmt = conn.createStatement

      val insertQuery = "INSERT INTO users ( email, first_name, last_name) VALUES( '"+user.email+"', '"+user.firstName+"','"+user.lastName+"') "
      stmt.executeUpdate(insertQuery,Statement.RETURN_GENERATED_KEYS)
      val resultSet = stmt.getGeneratedKeys;
      if(resultSet.next()) {
        val id = resultSet.getLong(1);
        new User(Option(id),user.email,user.firstName,user.lastName)
      } else {
        throw new Exception("User not persisted properly")
      }
    }
  }

In the above example a user is persisted. In case of failure we throw an exception and the transaction is rolled back.

To sum up we have just checked how to access a sql database using play. Also we have checked the extra functions that play api provides apart from the familiar jdbc api.
That’s all for now! Feel free to check the code on github.

Use JSON with Play and Scala

Once getting your hands into typing scala code using play, the first thing that comes to mind is JSON.
Without doubt JSON is one of the most basic components of web applications. Rest apis use json, your angular app has to consume json and the list goes on.

If you are lazy like me, you expect that it is sufficient to just pass back scala objects through your controller or specify a scala class as an argument to your controller. Somehow things don’t get far from that however some adjustments have to be done.

The first step is to specify the json module

libraryDependencies += json

The JSON library is pretty similar to the org.json library for java but with extra capabilities. The types we have out of the box are

JsString
JsNumber
JsBoolean
JsObject
JsArray
JsNull

However the key functionality comes from the Reads and Writes converters which can be used to marshal or unmarshal our data structures.

Suppose we have a class called User

case class User(id:Option[Long],email:String,firstName:String,lastName:String)

We want to use this class to pass data to our controllers or use it as a response, once our action has finished.

Thus we need to create a Reader and writer for the User object.

  implicit val userWrites = new Writes[User] {
    def writes(user: User) = Json.obj(
      "id" -> user.id,
      "email" -> user.email,
      "firstName" -> user.firstName,
      "lastName" -> user.lastName
    )
  }

  implicit val userReads: Reads[User] = (
    (__ \ "id").readNullable[Long] and
      (__ \ "email").read[String] and
      (__ \ "firstName").read[String] and
      (__ \ "lastName").read[String]
    )(User.apply _)

Most probably you’ve noticed that the id is optional. We do so in order to be able to either pass the id of the user or not.

Now let’s put them together in a controller.

package controllers

import javax.inject.Inject

import com.google.inject.Singleton
import play.api.libs.json._
import play.api.mvc.{Action, Controller}
import play.api.libs.functional.syntax._

/**
  * Created by gkatzioura on 4/26/17.
  */
case class User(id:Option[Long],email:String,firstName:String,lastName:String)

@Singleton
class UserController @Inject() extends Controller {

  def all = Action { implicit request =>
    val users = Seq(
      User(Option(1L),"gkazoura@example.com","Emmanouil","Gkatziouras"),
      User(Option(2L),"john@doe.com","John","Doe"),
      User(Option(3L),"john2@doe.com","John2","Doe2")
    )
    Ok(Json.toJson(users))
  }

  def greet = Action

  def add = Action { implicit request =>

    val user  = Json.fromJson[User](request.body.asJson.get).get
    val newUser = User(Option(4L),user.email,user.firstName,user.lastName)
    Ok(Json.toJson(newUser))
  }

  implicit val userWrites = new Writes[User] {
    def writes(user: User) = Json.obj(
      "id" -> user.id,
      "email" -> user.email,
      "firstName" -> user.firstName,
      "lastName" -> user.lastName
    )
  }

  implicit val userReads: Reads[User] = (
    (__ \ "id").readNullable[Long] and
      (__ \ "email").read[String] and
      (__ \ "firstName").read[String] and
      (__ \ "lastName").read[String]
    )(User.apply _)
}

And also the roots configuration

GET     /user/                   controllers.UserController.all
POST    /user/                   controllers.UserController.add

As we can see the all method returns a list of user objects in Json format while the add method is supposed to persist a user object and assign an id to it.

Let’s do a curl request and check our results

curl http://localhost:9000/user/

....

[{"id":1,"email":"gkazoura@example.com","firstName":"Emmanouil","lastName":"Gkatziouras"},{"id":2,"email":"john@doe.com","firstName":"John","lastName":"Doe"},{"id":3,"email":"john2@doe.com","firstName":"John2","lastName":"Doe2"}]

curl -H "Content-Type: application/json" -X POST -d '{"email":"emmanouil@egkatzioura.com","firstName":"Emmanouil","lastName":"Gkatziouras"}' http://localhost:9000/user/

...

{"id":4,"email":"emmanouil@egkatzioura.com","firstName":"Emmanouil","lastName":"Gkatziouras"}

So we didn’t get into any special json handling or reading instead we used only objects.
That’s it! Now your are ready for more JSON related action!

You can check the sourcecode on github.

Database Initialization with play and Scala

Once starting your play prototype application one of the priorities is to initialize your database and also manage the database schema changes.

Play provides us with evolutions. By utilizing evolutions we are able to create our database and to manage any futures changes to the schema.

To get started we need  to add the jdbc dependency and the evolutions dependency.

libraryDependencies += evolutions
libraryDependencies += jdbc

Then we shall use a simple h2 database persisted on disk, as our play application’s default database.
We edit the conf/application.conf file and add the following lines.

db.default.driver=org.h2.Driver
db.default.url="jdbc:h2:/tmp/defaultdatabase"

Pay extra attention that our database location is at the tmp directory thus all change shall be deleted once we reboot our workstation.

Once we have configured our database we are ready to create our first sql statement.
Our scripts should be located at the conf/evolutions/{your database name} directory, thus in our case
/conf/evolutions/default.

Our first script ‘1.sql’, shall create the users table.

# Users schema

# --- !Ups

CREATE TABLE users (
    id bigint(20) NOT NULL AUTO_INCREMENT,
    email varchar(255) NOT NULL,
    first_name varchar(255) NOT NULL,
    last_name varchar(255) NOT NULL,
    PRIMARY KEY (id),
    UNIQUE KEY (email)
);

# --- !Downs

DROP TABLE users;

As we can see we got ups and downs. What do they stand for? As you have guessed ups describe the transformations while downs describe how to revert them.
So the next question would be, how this functionality comes in use?
Suppose you have two developers working on the 2.sql. Locally they have successfully migrated their database once they are done, however the merge result is far different than the file they executed on their database.
What evolutions do is detect if the file is different and reverts the old revision by applying downs and then applying the up to date revision.

Now we are all set to run our application.

sbt run

Once we navigate at localhost:9000 we shall be presented with a screen that forces us to run the evolutions detected.

Let us go one step further and see what has been done to our database schema. We can easily explore a h2 database using dbeaver or your ide .
By issuing show tables the results contain one extra table.


>SHOW TABLES;

TABLE_NAME,TABLE_SCHEMA
PLAY_EVOLUTIONS,PUBLIC
USERS,PUBLIC

The PLAY_EVOLUTIONS table keeps track of our changes

Id is the number of the evolution script that we created. The fields apply and revert are the ups and downs sql statements we created previously.
The field hash is used in order to detect changes to our file. In case of an evolution that has a different hash from the one applied the previous evolution is reverted and applies the new script.

For example let’s enhance our previous script and add one more field. The field username.

# Users schema

# --- !Ups

CREATE TABLE users (
    id bigint(20) NOT NULL AUTO_INCREMENT,
    email varchar(255) NOT NULL,
    username varchar(255) NOT NULL,
    first_name varchar(255) NOT NULL,
    last_name varchar(255) NOT NULL,
    PRIMARY KEY (id),
    UNIQUE KEY (email)
);

# --- !Downs

DROP TABLE users;

Once we start our application we will be presented with a screen that forces us to issue an evolution for our different revision. If we hit apply the users table shall contain the username field.

So the process of a new revision is pretty straight forward.
The hash from the new 1.sql file is extracted. Then a query checks if the 1.sql file has already been applied. If it has been applied a check is issued in case the hashes are the same. If they are not then the downs script from the current database entry is executed. Once finished the new script is applied.

Your first Web application with Play and Scala

Today we are going to develop a simple play application using Scala.

To do so we must have sbt installed to our system.

Once installed we issue the command

sbt new playframework/play-scala-seed.g8

Then we are presented with an interactive terminal in order to pass valuable information.

name [play-scala-seed]: PlayStarter
organization [com.example]: com.gkatzioura
scala_version [2.11.8]: 
scalatestplusplay_version [2.0.0]: 
play_version [2.5.13]: 

Then let us check what we have just created

cd playstarter
sbt run

Navigate to http://localhost:9000 and you have a basic Play hello world.

By looking to our project structure, as expected, we have a directory with our controllers.
Consider our request being handled as an action. We issue a request and we receive an html view.

  def index = Action { implicit request =>
    Ok(views.html.index())
  }

As you can see we the html that is rendered is located at the views directory. Play comes with Twirl as a template engine.

At conf/routes we can see how the route is configured to the index action

Let’s add a simple action to that controller that returns a text body.

  def greet(name: String) = Action {
    Ok("Hello " + name)
  }

We have to edit the routes file to specify the new route and the get parameter

GET     /greet                      controllers.HomeController.greet(name)

Then issue a request at http://localhost:9000/greet?john

On the next step we shall add a new route with a path param

Suppose we want to retrieve the total logins for a user.
We implement an action that send a fake number

  def loginCount(userId: String) = Action {
    Ok(14)
  }

And then we register the route

GET     /user/:userId/login/count          controllers.HomeController.loginCount(userId)

By issuing the request http://localhost:9000/user/18/login/count
we shall receive the number 14.

To sum up we just implemented our first Play application. We also implemented some basic actions to our controller and achieved to pass some path and request parameters.