Play and SBT basics

Previously we had an introduction to sbt, its default tasks and how to add extra tasks.

Play comes with the sbt console. The SBT console is a development console based on sbt that allows you to manage a Play application’s complete development cycle.

Let us create a play application using sbt and see the commands provided.

sbt new playframework/play-scala-seed.g8

[warn] Executing in batch mode.
[warn]   For better performance, hit [ENTER] to switch to interactive mode, or
[warn]   consider launching sbt without any commands, or explicitly passing 'shell'
[info] Set current project to development (in build file:/home/gkatzioura/Development/)

This template generates a Play Scala project 

name [play-scala-seed]: PlayExample
organization [com.example]: com.gkatzioura
scala_version [2.11.11]: 
play_version [2.5.14]: 
scalatestplusplay_version [2.0.0]: 

The result is a play project named playexample. By opening with an editor the project/plugins.sbt we can see the sbt plugin added to our project.
Therefore we are going to check what are the extra tasks that the sbt plugin provides and some tasks that can be generally helpful.

cd playexample; sbt 
[PlayExample] $ <tab><tab>
Display all 511 possibilities? (y or n)
...
h2-browser  
...                                                                                                                                                                                                                              
playStop                                           
playUpdateSecret     
playGenerateSecret                              
...
stage
...
  • playStop – Stop Play, if it has been started in non blocking mode
  • playGenerateSecret – This will generate a new secret that you can use in your application. For example the application secret can be used for Signing session cookies and CSRF tokens or
    built in encryption utilities
  • playUpdateSecret – Update the application conf to generate an application secret
  • stage – Create a local directory with all the files laid out as they would be in the final distribution.
  • h2-browser – Opens an h2 database browser. Pretty useful if you are using h2 for development

Those are some of the commands that you might use often. However if you want extra information, you can always type help play.

[PlayExample] $ help play

playExternalizeResources

  Whether resources should be externalized into the conf directory when Play is packaged as a distribution.

playCommonClassloader

  The common classloader, is ...
...

SBT basics

Sbt is the de facto build tool in the Scala community.
Being used to other build tools you will be familiar with the commands

  • clean – Deletes files produced by the build, such as generated sources, compiled classes, and task caches.
  • compile – Compiles sources
  • test – Executes all tests
  • package – Produces the main artifact, such as a binary jar. This is typically an alias for the task that actually does the packaging.
  • help – Displays this help message or prints detailed help on requested commands (run ‘help ‘).
  • console – Starts the Scala interpreter with the project classes on the classpath.

Then we have extra commands suchs as

    • run – Runs a main class, passing along arguments provided on the command line.
    • tasks – Lists the tasks defined for the current project.
    • reload – (Re)loads the current project or changes to plugins project or returns from it.
    • console – Starts the Scala interpreter with the project classes on the classpath.

A key functionality is the new command.

For example by using new, we can create a project from the template specified (for example scala-seed.g8) using giter8.

sbt new scala/scala-seed.g8

...
Minimum Scala build. 

name [My Something Project]: hello

Template applied in ./hello

The previous snippet creates a project called hello.

The file build.sbt holds a sequence of key-value pairs called setting expressions. The left-hand side is a key and the right hand side is the body.
There are three types of keys.

  • SettingKey[T]: a key for a value computed once (the value is computed
    when loading the subproject, and kept around).
  • TaskKey[T]: a key for a value, called a task, that has to be recomputed
    each time, potentially with side effects.
  • InputKey[T]: a key for a task that has command line arguments as input.

For example if we want to add and extra task, to our previous project,  which prints hello, then we shall add the following lines to the build.sbt file.

import Dependencies._

lazy val hello = taskKey[Unit]("An example task")

lazy val root = (project in file(".")).
settings(
hello := { println("Hello!") },
inThisBuild(List(
organization := "com.example",
scalaVersion := "2.12.2",
version := "0.1.0-SNAPSHOT"
)),
name := "Hello",
libraryDependencies += scalaTest % Test
)

We can either run the task or ask for more info about the task.

>sbt
> hello
Hello!
[success] Total time: 0 s, completed May 1, 2017 6:08:36 PM
> help hello
An example task
> 

Depending on the project and the plugin used, there would be extra tasks and settings defined.

On the next post we will check play and sbt integration, and some basic commands.

SQL Data Access in Play using Scala

Today’s modern application frameworks come with a promise of easy sql data access. There is no wonder why we have so many frameworks that make it easier to issue queries and handle transactions. SQL is the lingua franca of most applications when it comes to databases.

Play comes with the JDBC plugin. We encountered the JDBC plugin previously in order to modify our database schema.

The first step is to include the jdbc and the evolutions module.

libraryDependencies += evolutions
libraryDependencies += jdbc

Then we shall define the connection string needed. We will use a simple h2 database. The configuration is added at the application.conf.

db.default.driver=org.h2.Driver
db.default.url="jdbc:h2:/tmp/defaultdatabase"

Then we add a script that creates the users table.

# Users schema

# --- !Ups

CREATE TABLE users (
    id bigint(20) NOT NULL AUTO_INCREMENT,
    email varchar(255) NOT NULL,
    first_name varchar(255) NOT NULL,
    last_name varchar(255) NOT NULL,
    PRIMARY KEY (id),
    UNIQUE KEY (email)
);

# --- !Downs

DROP TABLE users;

Before creating our repository class let’s check what the jdbc plugin provides us with.

We have the plain getConnection method, responsible for returning a jdbc connection. This is similar to the DataSource.getConnection from Java. Thus pay extra attention since you must close the connection.

val connection = db.getConnection()
connection.close()

Next method is withConnection. By using withConnection you get Play to manage the connection for you. All you have to do is pass a block of code with jdbc actions.

  def fetchUsers(): List[User] = {

    db.withConnection { conn =>

      val stmt = conn.createStatement
      var rs = stmt.executeQuery("SELECT*FROM users");
      val listBuffer = ListBuffer[User]()

      while(rs.next()) {

        listBuffer.append(User(Option(rs.getLong("id")),rs.getString("email"),rs.getString("first_name"),rs.getString("last_name")))
      }

      listBuffer.toList
    }
  }

As you can see above, we’ve just returned back a list of our user entries.

And last but not least withTransaction. You’ve guessed right, what you receive back is a connection with autocommit set to false.

  def addUser(user:User): User = {

    db.withTransaction { conn =>
      val stmt = conn.createStatement

      val insertQuery = "INSERT INTO users ( email, first_name, last_name) VALUES( '"+user.email+"', '"+user.firstName+"','"+user.lastName+"') "
      stmt.executeUpdate(insertQuery,Statement.RETURN_GENERATED_KEYS)
      val resultSet = stmt.getGeneratedKeys;
      if(resultSet.next()) {
        val id = resultSet.getLong(1);
        new User(Option(id),user.email,user.firstName,user.lastName)
      } else {
        throw new Exception("User not persisted properly")
      }
    }
  }

In the above example a user is persisted. In case of failure we throw an exception and the transaction is rolled back.

To sum up we have just checked how to access a sql database using play. Also we have checked the extra functions that play api provides apart from the familiar jdbc api.
That’s all for now! Feel free to check the code on github.

Use JSON with Play and Scala

Once getting your hands into typing scala code using play, the first thing that comes to mind is JSON.
Without doubt JSON is one of the most basic components of web applications. Rest apis use json, your angular app has to consume json and the list goes on.

If you are lazy like me, you expect that it is sufficient to just pass back scala objects through your controller or specify a scala class as an argument to your controller. Somehow things don’t get far from that however some adjustments have to be done.

The first step is to specify the json module

libraryDependencies += json

The JSON library is pretty similar to the org.json library for java but with extra capabilities. The types we have out of the box are

JsString
JsNumber
JsBoolean
JsObject
JsArray
JsNull

However the key functionality comes from the Reads and Writes converters which can be used to marshal or unmarshal our data structures.

Suppose we have a class called User

case class User(id:Option[Long],email:String,firstName:String,lastName:String)

We want to use this class to pass data to our controllers or use it as a response, once our action has finished.

Thus we need to create a Reader and writer for the User object.

  implicit val userWrites = new Writes[User] {
    def writes(user: User) = Json.obj(
      "id" -> user.id,
      "email" -> user.email,
      "firstName" -> user.firstName,
      "lastName" -> user.lastName
    )
  }

  implicit val userReads: Reads[User] = (
    (__ \ "id").readNullable[Long] and
      (__ \ "email").read[String] and
      (__ \ "firstName").read[String] and
      (__ \ "lastName").read[String]
    )(User.apply _)

Most probably you’ve noticed that the id is optional. We do so in order to be able to either pass the id of the user or not.

Now let’s put them together in a controller.

package controllers

import javax.inject.Inject

import com.google.inject.Singleton
import play.api.libs.json._
import play.api.mvc.{Action, Controller}
import play.api.libs.functional.syntax._

/**
  * Created by gkatzioura on 4/26/17.
  */
case class User(id:Option[Long],email:String,firstName:String,lastName:String)

@Singleton
class UserController @Inject() extends Controller {

  def all = Action { implicit request =>
    val users = Seq(
      User(Option(1L),"gkazoura@example.com","Emmanouil","Gkatziouras"),
      User(Option(2L),"john@doe.com","John","Doe"),
      User(Option(3L),"john2@doe.com","John2","Doe2")
    )
    Ok(Json.toJson(users))
  }

  def greet = Action

  def add = Action { implicit request =>

    val user  = Json.fromJson[User](request.body.asJson.get).get
    val newUser = User(Option(4L),user.email,user.firstName,user.lastName)
    Ok(Json.toJson(newUser))
  }

  implicit val userWrites = new Writes[User] {
    def writes(user: User) = Json.obj(
      "id" -> user.id,
      "email" -> user.email,
      "firstName" -> user.firstName,
      "lastName" -> user.lastName
    )
  }

  implicit val userReads: Reads[User] = (
    (__ \ "id").readNullable[Long] and
      (__ \ "email").read[String] and
      (__ \ "firstName").read[String] and
      (__ \ "lastName").read[String]
    )(User.apply _)
}

And also the roots configuration

GET     /user/                   controllers.UserController.all
POST    /user/                   controllers.UserController.add

As we can see the all method returns a list of user objects in Json format while the add method is supposed to persist a user object and assign an id to it.

Let’s do a curl request and check our results

curl http://localhost:9000/user/

....

[{"id":1,"email":"gkazoura@example.com","firstName":"Emmanouil","lastName":"Gkatziouras"},{"id":2,"email":"john@doe.com","firstName":"John","lastName":"Doe"},{"id":3,"email":"john2@doe.com","firstName":"John2","lastName":"Doe2"}]

curl -H "Content-Type: application/json" -X POST -d '{"email":"emmanouil@egkatzioura.com","firstName":"Emmanouil","lastName":"Gkatziouras"}' http://localhost:9000/user/

...

{"id":4,"email":"emmanouil@egkatzioura.com","firstName":"Emmanouil","lastName":"Gkatziouras"}

So we didn’t get into any special json handling or reading instead we used only objects.
That’s it! Now your are ready for more JSON related action!

You can check the sourcecode on github.

Database Initialization with play and Scala

Once starting your play prototype application one of the priorities is to initialize your database and also manage the database schema changes.

Play provides us with evolutions. By utilizing evolutions we are able to create our database and to manage any futures changes to the schema.

To get started we need  to add the jdbc dependency and the evolutions dependency.

libraryDependencies += evolutions
libraryDependencies += jdbc

Then we shall use a simple h2 database persisted on disk, as our play application’s default database.
We edit the conf/application.conf file and add the following lines.

db.default.driver=org.h2.Driver
db.default.url="jdbc:h2:/tmp/defaultdatabase"

Pay extra attention that our database location is at the tmp directory thus all change shall be deleted once we reboot our workstation.

Once we have configured our database we are ready to create our first sql statement.
Our scripts should be located at the conf/evolutions/{your database name} directory, thus in our case
/conf/evolutions/default.

Our first script ‘1.sql’, shall create the users table.

# Users schema

# --- !Ups

CREATE TABLE users (
    id bigint(20) NOT NULL AUTO_INCREMENT,
    email varchar(255) NOT NULL,
    first_name varchar(255) NOT NULL,
    last_name varchar(255) NOT NULL,
    PRIMARY KEY (id),
    UNIQUE KEY (email)
);

# --- !Downs

DROP TABLE users;

As we can see we got ups and downs. What do they stand for? As you have guessed ups describe the transformations while downs describe how to revert them.
So the next question would be, how this functionality comes in use?
Suppose you have two developers working on the 2.sql. Locally they have successfully migrated their database once they are done, however the merge result is far different than the file they executed on their database.
What evolutions do is detect if the file is different and reverts the old revision by applying downs and then applying the up to date revision.

Now we are all set to run our application.

sbt run

Once we navigate at localhost:9000 we shall be presented with a screen that forces us to run the evolutions detected.

Let us go one step further and see what has been done to our database schema. We can easily explore a h2 database using dbeaver or your ide .
By issuing show tables the results contain one extra table.


>SHOW TABLES;

TABLE_NAME,TABLE_SCHEMA
PLAY_EVOLUTIONS,PUBLIC
USERS,PUBLIC

The PLAY_EVOLUTIONS table keeps track of our changes

Id is the number of the evolution script that we created. The fields apply and revert are the ups and downs sql statements we created previously.
The field hash is used in order to detect changes to our file. In case of an evolution that has a different hash from the one applied the previous evolution is reverted and applies the new script.

For example let’s enhance our previous script and add one more field. The field username.

# Users schema

# --- !Ups

CREATE TABLE users (
    id bigint(20) NOT NULL AUTO_INCREMENT,
    email varchar(255) NOT NULL,
    username varchar(255) NOT NULL,
    first_name varchar(255) NOT NULL,
    last_name varchar(255) NOT NULL,
    PRIMARY KEY (id),
    UNIQUE KEY (email)
);

# --- !Downs

DROP TABLE users;

Once we start our application we will be presented with a screen that forces us to issue an evolution for our different revision. If we hit apply the users table shall contain the username field.

So the process of a new revision is pretty straight forward.
The hash from the new 1.sql file is extracted. Then a query checks if the 1.sql file has already been applied. If it has been applied a check is issued in case the hashes are the same. If they are not then the downs script from the current database entry is executed. Once finished the new script is applied.

Your first Web application with Play and Scala

Today we are going to develop a simple play application using Scala.

To do so we must have sbt installed to our system.

Once installed we issue the command

sbt new playframework/play-scala-seed.g8

Then we are presented with an interactive terminal in order to pass valuable information.

name [play-scala-seed]: PlayStarter
organization [com.example]: com.gkatzioura
scala_version [2.11.8]: 
scalatestplusplay_version [2.0.0]: 
play_version [2.5.13]: 

Then let us check what we have just created

cd playstarter
sbt run

Navigate to http://localhost:9000 and you have a basic Play hello world.

By looking to our project structure, as expected, we have a directory with our controllers.
Consider our request being handled as an action. We issue a request and we receive an html view.

  def index = Action { implicit request =>
    Ok(views.html.index())
  }

As you can see we the html that is rendered is located at the views directory. Play comes with Twirl as a template engine.

At conf/routes we can see how the route is configured to the index action

Let’s add a simple action to that controller that returns a text body.

  def greet(name: String) = Action {
    Ok("Hello " + name)
  }

We have to edit the routes file to specify the new route and the get parameter

GET     /greet                      controllers.HomeController.greet(name)

Then issue a request at http://localhost:9000/greet?john

On the next step we shall add a new route with a path param

Suppose we want to retrieve the total logins for a user.
We implement an action that send a fake number

  def loginCount(userId: String) = Action {
    Ok(14)
  }

And then we register the route

GET     /user/:userId/login/count          controllers.HomeController.loginCount(userId)

By issuing the request http://localhost:9000/user/18/login/count
we shall receive the number 14.

To sum up we just implemented our first Play application. We also implemented some basic actions to our controller and achieved to pass some path and request parameters.

A journey with Scala

To those who are regular visitors of this blog, it is well known that when it comes to developing code I am a Spring/Java guy. Also I use different technologies like node or python but this depends largely on the project’s needs.

Due to some recent projects and courses involving Spark, stumbling on Scala was inevitable. After some investigation I decided to adopt it, as one of my main tools and there are many reasons for that.

From a Java Developers perspective

  • It evolves faster
  • It has Flexible syntax
  • It is static typed
  • It is pretty recent thus exciting, but in a JVM flavor

I remember back then when I was anticipating the release of Java 7. Lambdas, streams and put some functional programming into action. All those features that Scala provided. Unfortunately lambdas were dropped from Java 7, and released as part of Java 8. Thus it took 3 years to get your hands on lambdas 😦

The Scala syntax is great and increases productivity. I really fancy the fact that you can skip a lot of boilerplate that your had to deal with java. Let alone the options like tuples, switches and parameter name specification on function calls. The list could go on and on.

I used both dynamic and static typed languages however by developing mainly on Java I am a bit biased. I believe that static typing is a good choice because you can detect errors early, build reliable code and increase maintainability. Also it makes collaboration with others much easier.

Every new technology is exciting and feels like a new toy. However adopting a new technology comes with the lack of libraries and frameworks (node anyone?) that were essential for your development process. Fortunately you can always bet on the JVM and use your Java libraries with your Scala source code.

From a ecosystem perspective

If you come from Java EE or Spring MVC, you have an already prooved and tested framework for your web application.  Luckily Play comes to the rescue. Is Play sufficient for all your needs? I doubt. But frameworks evolve and since we live in a  microservices-architecture era having some Java components does the work.

From  a project perspective.

The cloud and the overall technological burst has brought a variety of different types of applications. Developing applications has become more challenging and some of them involve big data processing, streaming and machine learning. Scala is first place on this type of applications, thus if your objective is to work on the big data world mastering Scala will definitely assist you.

It does the filtering for you. As mentioned previously Spring/Java can be used for various type of projects, batch Applications, CMS apps, Rest apis etc. Scala has a more specific identity and targets certain type of projects. By searching for opportunities and contracts that include Scala, you already have a sense of the project’s nature and the challenges ahead.

At last

Talk is cheap! There are many tutorials ahead to author and document my experience. Stay tuned for more Scala content.

Spring Data with JPA and @NamedQueries

If you use Spring Data and @NamedQuery annotations at your JPA entity you can easily use them in a more convenient way using the spring data repository.

On a previous blog we created a spring data project using spring boot and docker. We will use the pretty same project and enhance our repository’s functionality.

We will implement a named query that will fetch employees only if their Last Name has as many characters as the ones specified.

package com.gkatzioura.springdata.jpa.persistence.entity;

import javax.persistence.*;

/**
 * Created by gkatzioura on 6/2/16.
 */
@Entity
@Table(name = "employee", schema="spring_data_jpa_example")
@NamedQuery(name = "Employee.fetchByLastNameLength",
        query = "SELECT e FROM Employee e WHERE CHAR_LENGTH(e.lastname) =:length "
)
public class Employee {

    @Id
    @Column(name = "id")
    @GeneratedValue(strategy = GenerationType.SEQUENCE)
    private Long id;

    @Column(name = "firstname")
    private String firstName;

    @Column(name = "lastname")
    private String lastname;

    @Column(name = "email")
    private String email;

    @Column(name = "age")
    private Integer age;

    @Column(name = "salary")
    private Integer salary;

    public Long getId() {
        return id;
    }

    public void setId(Long id) {
        this.id = id;
    }

    public String getFirstName() {
        return firstName;
    }

    public void setFirstName(String firstName) {
        this.firstName = firstName;
    }

    public String getLastname() {
        return lastname;
    }

    public void setLastname(String lastname) {
        this.lastname = lastname;
    }

    public String getEmail() {
        return email;
    }

    public void setEmail(String email) {
        this.email = email;
    }

    public Integer getAge() {
        return age;
    }

    public void setAge(Integer age) {
        this.age = age;
    }

    public Integer getSalary() {
        return salary;
    }

    public void setSalary(Integer salary) {
        this.salary = salary;
    }
}

Pay extra attention to the query name and the convention we follow @{EntityName}.{queryName}.
Then we will add the method to our spring data repository.

package com.gkatzioura.springdata.jpa.persistence.repository;

import com.gkatzioura.springdata.jpa.persistence.entity.Employee;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.data.repository.query.Param;
import org.springframework.stereotype.Repository;

import java.util.List;

/**
 * Created by gkatzioura on 6/2/16.
 */
@Repository
public interface EmployeeRepository extends JpaRepository<Employee,Long>, EmployeeRepositoryCustom {

    List<Employee> fetchByLastNameLength(@Param("length") Long length);
}

And last but not least add some functionality to our controller.

package com.gkatzioura.springdata.jpa.controller;

import com.gkatzioura.springdata.jpa.persistence.entity.Employee;
import com.gkatzioura.springdata.jpa.persistence.repository.EmployeeRepository;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

import java.util.List;

/**
 * Created by gkatzioura on 6/2/16.
 */
@RestController
public class TestController {

    @Autowired
    private EmployeeRepository employeeRepository;

    @RequestMapping("/employee")
    public List<Employee> getTest() {

        return employeeRepository.findAll();
    }

    @RequestMapping("/employee/filter")
    public List<Employee> getFiltered(String firstName,@RequestParam(defaultValue = "0") Double bonusAmount) {

        return employeeRepository.getFirstNamesLikeAndBonusBigger(firstName,bonusAmount);
    }

    @RequestMapping("/employee/lastnameLength")
    public List<Employee> fetchByLength(Long length) {
        return employeeRepository.fetchByLastNameLength(length);
    }

}

You can find the source code on github.

Run WordCount with Scala and Spark on HDInsight

Previously we tried to solve the word count problem with a Scala and Spark approach.
The next step is to deploy our solution to HDInsight using spark, hdfs, and scala

We shall provision a Sprak cluster.

screenshot-from-2017-02-22-23-12-22

Since we are going to use HDInsight we can utilize hdfs and therefore use the azure storage.

screenshot-from-2017-02-22-23-12-59

Then we choose our instance types.

screenshot-from-2017-02-22-23-13-21

And we are ready to create the Spark cluster.

screenshot-from-2017-02-22-23-13-55

Our data shall be uploaded to the hdfs file system
To do so we will upload our text files to the azure storage account which is integrated with hdfs.

For more information on managing a storage account with azure cli check the official guide. Any text file will work.

azure storage blob upload mytextfile.txt sparkclusterscala  example/data/mytextfile.txt

Since we use hdfs we shall make some changes to the original script

val text = sc.textFile("wasb:///example/data/mytextfile.txt")
val counts = text.flatMap(line => line.split(" ")).map(word => (word,1)).reduceByKey(_+_)
counts.collect

Then we can upload our scala class to the head node using ssh

scp WordCountscala.scala demon@{your cluster}-ssh.azurehdinsight.net:/home/demo/WordCountscala.scala

Again in order to run the script, things are pretty straightforward.

spark-shell -i WordCountscala.scala

And once the task is done we are presented with the spark prompt. Plus we can now save our results to the hdfs file system.

scala> counts.saveAsTextFile("/wordcount_results")

And do a quick check.

hdfs dfs -ls /wordcount_results/
hdfs dfs -text /wordcount_results/part-00000