Query DynamoDB Items with Node.js

On a previous post we proceeded on inserting data on a DynamoDB database.

On this tutorial we will issue some basic queries against our DynamoDB tables.

The main rule is that every query has to use the hash key.

The simplest form of query is using the hash key only. We will query the Users table on this one. There would be only one result, therefore there is no use on iterating the Items list.

var getUser = function(email,callback) {
	
	var docClient = new AWS.DynamoDB.DocumentClient();
	
	var params = {
		    TableName: "Users",
		    KeyConditionExpression: "#email = :email",
		    ExpressionAttributeNames:{
		        "#email": "email"
		    },
		    ExpressionAttributeValues: {
		        ":email":email
		    }
		};
	
	docClient.query(params,callback);
};

However we can issue more complex queries using conditions.
Logins Table suits well for an example. We will issue a query that will fetch login attempts between to dates.

var queryLogins = function(email,from,to,callback) {

	var docClient = new AWS.DynamoDB.DocumentClient();
	
	var params = {
	    TableName:"Logins",
	    KeyConditionExpression:"#email = :emailValue and #timestamp BETWEEN :from AND :to",
	    ExpressionAttributeNames: {
	    	"#email":"email",
	    	"#timestamp":"timestamp"
	    },
	    ExpressionAttributeValues: {
	    	":emailValue":email,
	    	":from": from.getTime(),
	    	":to":to.getTime()
	    }			
	};
	
	var items = []
	
	var queryExecute = function(callback) {
	
		docClient.query(params,function(err,result) {

			if(err) {
				callback(err);
			} else {
			
				console.log(result)
				
				items = items.concat(result.Items);
			
				if(result.LastEvaluatedKey) {

					params.ExclusiveStartKey = result.LastEvaluatedKey;
					queryExecute(callback);				
				} else {
					callback(err,items);
				}	
			}
		});
	}
	
	queryExecute(callback);
};

Keep in mind that DynamoDB Fetches data in pages, therefore you have to issue the same request more than once in case of multiple pages. Therefore you have to use the last evaluated key to your next request. In case of many entries be aware that you should handle the call stack size.

Last but not least querying on indexes is one of the basic actions. It is the same routine either for local or global secondary indexes.
Keep in mind that the results fetched depend on the projection type we specified once creating the Table. In our case the projection type is for all fields.

We shall use the Supervisors table.

	var docClient = new AWS.DynamoDB.DocumentClient();
	
	var params = {
		    TableName: "Supervisors",
		    IndexName: "FactoryIndex",
		    KeyConditionExpression:"#company = :companyValue and #factory = :factoryValue",
		    ExpressionAttributeNames: {
		    	"#company":"company",
		    	"#factory":"factory"
		    },
		    ExpressionAttributeValues: {
		    	":companyValue": company,
		    	":factoryValue": factory
		    }
		};

	docClient.query(params,callback);

You can find full source code with unit tests on github.

Query DynamoDB Items with Java

On a previous post we proceeded on inserting data on a DynamoDB database.

On this tutorial we will issue some basic queries against our DynamoDB tables.

The main rule is that every query has to use the hash key.

The simplest form of query is using the hash key only. We will query the Users table on this one. There would be only one result, therefore there is no use on iterating the Items list.

    public Map<String,AttributeValue> getUser(String email) {

        Map<String,String> expressionAttributesNames = new HashMap<>();
        expressionAttributesNames.put("#email","email");

        Map<String,AttributeValue> expressionAttributeValues = new HashMap<>();
        expressionAttributeValues.put(":emailValue",new AttributeValue().withS(email));

        QueryRequest queryRequest = new QueryRequest()
                .withTableName(TABLE_NAME)
                .withKeyConditionExpression("#email = :emailValue")
                .withExpressionAttributeNames(expressionAttributesNames)
                .withExpressionAttributeValues(expressionAttributeValues);

        QueryResult queryResult = amazonDynamoDB.query(queryRequest);

        List<Map<String,AttributeValue>> attributeValues = queryResult.getItems();

        if(attributeValues.size()>0) {
            return attributeValues.get(0);
        } else {
            return null;
        }
    }

However we can issue more complex queries using conditions.
Logins Table suits well for an example. We will issue a query that will fetch login attempts between to dates.

    public List<Map<String ,AttributeValue>> queryLoginsBetween(String email, Date from, Date to) {

        List<Map<String,AttributeValue>> items = new ArrayList<>();

        Map<String,String> expressionAttributesNames = new HashMap<>();
        expressionAttributesNames.put("#email","email");
        expressionAttributesNames.put("#timestamp","timestamp");

        Map<String,AttributeValue> expressionAttributeValues = new HashMap<>();
        expressionAttributeValues.put(":emailValue",new AttributeValue().withS(email));
        expressionAttributeValues.put(":from",new AttributeValue().withN(Long.toString(from.getTime())));
        expressionAttributeValues.put(":to",new AttributeValue().withN(Long.toString(to.getTime())));

        QueryRequest queryRequest = new QueryRequest()
                .withTableName(TABLE_NAME)
                .withKeyConditionExpression("#email = :emailValue and #timestamp BETWEEN :from AND :to ")
                .withExpressionAttributeNames(expressionAttributesNames)
                .withExpressionAttributeValues(expressionAttributeValues);

        Map<String,AttributeValue> lastKey = null;

        do {

            QueryResult queryResult = amazonDynamoDB.query(queryRequest);
            List<Map<String,AttributeValue>> results = queryResult.getItems();
            items.addAll(results);
            lastKey = queryResult.getLastEvaluatedKey();
            queryRequest.setExclusiveStartKey(lastKey);
        } while (lastKey!=null);

        return items;
    }

Keep in mind that DynamoDB Fetches data in pages, therefore you have to issue the same request more than once in case of multiple pages. Therefore you have to use the last evaluated key to your next request.

Last but not least querying on indexes is one of the basic actions. It is the same routine either for local or global secondary indexes.
Keep in mind that the results fetched depend on the projection type we specified once creating the Table. In our case the projection type is for all fields.

We shall use the Supervisors table.

    public Map<String ,AttributeValue> getSupervisor(String company,String factory) {

        List<Map<String,AttributeValue>> items = new ArrayList<>();

        Map<String,String> expressionAttributesNames = new HashMap<>();
        expressionAttributesNames.put("#company","company");
        expressionAttributesNames.put("#factory","factory");

        Map<String,AttributeValue> expressionAttributeValues = new HashMap<>();
        expressionAttributeValues.put(":company",new AttributeValue().withS(company));
        expressionAttributeValues.put(":factory",new AttributeValue().withS(factory));

        QueryRequest queryRequest = new QueryRequest()
                .withTableName(TABLE_NAME)
                .withKeyConditionExpression("#company = :company and #factory = :factory ")
                .withIndexName("FactoryIndex")
                .withExpressionAttributeNames(expressionAttributesNames)
                .withExpressionAttributeValues(expressionAttributeValues);

        QueryResult queryResult = amazonDynamoDB.query(queryRequest);

        List<Map<String,AttributeValue>> attributeValues = queryResult.getItems();

        if(attributeValues.size()>0) {
            return attributeValues.get(0);
        } else {
            return null;
        }
    }

You can find full source code with unit tests on github.

I’ve compiled a cheat sheet that lists dynamodb functions in Java
Sign up in the link to receive it.

Insert Items to DynamoDB Tables using Node.js

On a previous article we learned how to create DynamoDB Tables using Node.js.

Next step is to insert items to the DynamoDB Tables previously created.

Keep in mind that for the insert action the most basic step is to specify the the primary key.
For the table users the primary key is the attribute email. You can add as many attributes as you want however the cumulative size should not surpass 400 KB.

var AWS = require("aws-sdk");

	var dynamodb = new AWS.DynamoDB();
	var params = {
			TableName:"Users",
		    Item:{
		    	email : { S:"jon@doe.com"},
		        fullname: { S:"Jon Doe"}
		    }
		};
	
	dynamodb.putItem(params,callback);

DynamoDB also supports Batch writes. In this case the main benefit lies on less I/O, however nothing changes regarding consumed capacity. In our case we will add a batch of login attempts.

var AWS = require("aws-sdk");

var insetBatchLogins = function(callback) {
	
	var dynamodb = new AWS.DynamoDB();
	var batchRequest = {
			RequestItems: {
				"Logins": [
				           {
				        	   PutRequest: { 
				        		   Item: {
				        			   "email": { S: "jon@doe.com" },
				        			   "timestamp": { N: "1467041009976" }
				        			   }
				           }},
				           {
				        	   PutRequest: { 
				        		   Item: {
				        			   "email": { S: "jon@doe.com" },
				        			   "timestamp": { N: "1467041019976" }
				        			   }
				           }}]
		    }
		};

	dynamodb.batchWriteItem(batchRequest,callback);
};

In case of an insert with a global/local secondary index all you have to do is to specify the corresponding attributes for the index. Take into consideration that you can have empty index related attributes or even duplicates.

	var dynamodb = new AWS.DynamoDB();
	
	var params = {
			TableName:"Supervisors",
		    Item:{
		    	name: { S:"Random SuperVisor"},
		    	company: { S:"Random Company"},
		    	factory: { S:"Jon Doe"}
		    }
		};
	
	dynamodb.putItem(params,callback);

You can find the sourcecode on github.

Insert Items to DynamoDB Tables using Java

On a previous article we learned how to create DynamoDB Tables using Java.

Next step is to insert items to the DynamoDB Tables previously created.

Keep in mind that for the insert action the most basic step is to specify the the primary key.
For the table users the primary key is the attribute email. You can add as many attributes as you want however the cumulative size should not surpass 400 KB.

 Map<String,AttributeValue> attributeValues = new HashMap<>();
        attributeValues.put("email",new AttributeValue().withS("jon@doe.com"));
        attributeValues.put("fullname",new AttributeValue().withS("Jon Doe"));

        PutItemRequest putItemRequest = new PutItemRequest()
                .withTableName("Users")
                .withItem(attributeValues);

        PutItemResult putItemResult = amazonDynamoDB.putItem(putItemRequest);

DynamoDB also supports Batch writes. In this case the main benefit lies on less I/O, however nothing changes regarding consumed capacity. In our case we will add a batch of login attempts.

        Map<String,AttributeValue> firstAttributeValues = new HashMap<>();
        firstAttributeValues.put("email",new AttributeValue().withS("jon@doe.com"));

        Long date = new Date().getTime();

        firstAttributeValues.put("timestamp",new AttributeValue().withN(Long.toString(date)));

        PutRequest firstPutRequest = new PutRequest();
        firstPutRequest.setItem(firstAttributeValues);

        WriteRequest firstWriteRequest = new WriteRequest();
        firstWriteRequest.setPutRequest(firstPutRequest);

        Map<String,AttributeValue> secondAttributeValues = new HashMap<>();
        secondAttributeValues.put("email",new AttributeValue().withS("jon@doe.com"));
        secondAttributeValues.put("timestamp",new AttributeValue().withN(Long.toString(date+100)));

        PutRequest secondPutRequest = new PutRequest();
        secondPutRequest.setItem(secondAttributeValues);

        WriteRequest secondWriteRequest = new WriteRequest();
        secondWriteRequest.setPutRequest(secondPutRequest);

        List<WriteRequest> batchList = new ArrayList<WriteRequest>();
        batchList.add(firstWriteRequest);
        batchList.add(secondWriteRequest);

        Map<String, List<WriteRequest>> batchTableRequests = new HashMap<String, List<WriteRequest>>();
        batchTableRequests.put("Logins",batchList);

        BatchWriteItemRequest batchWriteItemRequest = new BatchWriteItemRequest();
        batchWriteItemRequest.setRequestItems(batchTableRequests);

        amazonDynamoDB.batchWriteItem(batchWriteItemRequest);

In case of an insert with a global/local secondary index all you have to do is to specify the corresponding attributes for the index. Take into consideration that you can have empty index related attributes or even duplicates.

        Map<String,AttributeValue> attributeValues = new HashMap<>();
        attributeValues.put("name",new AttributeValue().withS("Random SuperVisor"));
        attributeValues.put("company",new AttributeValue().withS("Random Company"));
        attributeValues.put("factory",new AttributeValue().withS("Jon Doe"));

        PutItemRequest putItemRequest = new PutItemRequest()
                .withTableName("Supervisors")
                .withItem(attributeValues);

        PutItemResult putItemResult = amazonDynamoDB.putItem(putItemRequest);

You can find the sourcecode on github.

I’ve compiled a cheat sheet that lists dynamodb functions in Java
Sign up in the link to receive it.

Create DynamoDB tables with Node.js

On this post we will create Tables on a DynamoDB Database using node.js

Before getting started we need to have local dynamodb installed since we want to avoid any costs for dynamodb usage. There was a previous post on local dynamodb.

In case you use docker you can find a local dynamodb image or you can create one on you own as described here.

Using local DynamoDB and node.js is extremely handy for debugging. Local dynamodb provides as with an web user interface on http://localhost:8000/shell. The local dynamodb shell is a javascript shell, therefore the actions for node.js can be issued straight to the DynamoDB shell.

The actions would be the same as described on the corresponding java tutorial.

First step is to create a table with a hash key. In this case the email of the user would be the hash key.

var createUsers = function(callback) {

	var dynamodb = new AWS.DynamoDB();

	var params = {
	    TableName : "Users",
	    KeySchema: [       
	        { AttributeName: "email", KeyType: "HASH"}
	    ],
	    AttributeDefinitions: [       
	        { AttributeName: "email", AttributeType: "S" }
	    ],
	    ProvisionedThroughput: {       
	        ReadCapacityUnits: 5, 
	        WriteCapacityUnits: 5
		   }
		};

	dynamodb.createTable(params, callback);	
};

The next table will be called Logins. Logins should keep track each time the user logged in. To do so apart from using a hash key we will also use a range key for the date it occurred.

var createLogins = function(callback) {

	var dynamodb = new AWS.DynamoDB();

	var params = {
	    TableName : "Logins",
	    KeySchema: [       
	        { AttributeName: "email", KeyType: "HASH"},
	        { AttributeName: "timestamp", KeyType: "RANGE"}
		],
	    AttributeDefinitions: [       
	        { AttributeName: "email", AttributeType: "S" },
	        { AttributeName: "timestamp", AttributeType: "N" }
	    ],
	    ProvisionedThroughput: {       
	        ReadCapacityUnits: 5, 
	        WriteCapacityUnits: 5
		   }
		};

	dynamodb.createTable(params, callback);	
};

Next table is Supervisors. The hash key of Supervisor would be his name. A supervisor will work for a company. The company will be our global secondary index. Since the companies own more than one factories the field factory would be the range key.

var createSupervisors = function(callback) {

	var dynamodb = new AWS.DynamoDB();

	var params = {
	    TableName : "Supervisors",
	    KeySchema: [       
	        { AttributeName: "name", KeyType: "HASH"}
		],
	    AttributeDefinitions: [       
	        { AttributeName: "name", AttributeType: "S" },
	        { AttributeName: "company", AttributeType: "S" },
	        { AttributeName: "factory", AttributeType: "S" }    
	    ],
	    ProvisionedThroughput: {       
	        ReadCapacityUnits: 5, 
	        WriteCapacityUnits: 5
		   },
		GlobalSecondaryIndexes: [{
				IndexName: "FactoryIndex",
				KeySchema: [
				    {
				    	AttributeName: "company",
				    	KeyType: "HASH"
				    },
					{
						AttributeName: "factory",
						KeyType: "RANGE"
					}
				],
				Projection: {
					ProjectionType: "ALL"
				},
				ProvisionedThroughput: {
					ReadCapacityUnits: 1,
					WriteCapacityUnits: 1
				}
		    }]
	};

	dynamodb.createTable(params, callback);	
};

Next table would be the table Companies. The hash key would be the parent company and the range key the subsidiary company. Each company has a CEO. The CEO would be the range key for the local secondary index.

var createCompanies = function(callback) {

	var dynamodb = new AWS.DynamoDB();

	var params = {
	    TableName : "Companies",
	    KeySchema: [       
	        { AttributeName: "name", KeyType: "HASH"},
	        { AttributeName: "subsidiary", KeyType: "RANGE"}
		],
	    AttributeDefinitions: [       
	        { AttributeName: "name", AttributeType: "S" },
	        { AttributeName: "subsidiary", AttributeType: "S" },
	        { AttributeName: "ceo", AttributeType: "S" }    
	    ],
	    ProvisionedThroughput: {       
	        ReadCapacityUnits: 5, 
	        WriteCapacityUnits: 5
		   },
		LocalSecondaryIndexes: [{
				IndexName: "CeoIndex",
				KeySchema: [
				    {
				    	AttributeName: "name",
				    	KeyType: "HASH"
				    },
					{
						AttributeName: "ceo",
						KeyType: "RANGE"
					}
				],
				Projection: {
					ProjectionType: "ALL"
				}
		    }]
	};

	dynamodb.createTable(params, callback);	
};

You can find the source code on github.

Create DynamoDB tables with Java

On this post we will create Tables on a DynamoDB Database the java way.

Before getting started we need to have local dynamodb installed since we want to avoid any costs for dynamodb usage. There was a previous post on local dynamodb.

In case you use docker you can find a local dynamodb image or you can create one on you own as described here.

The dynamodb java sdk gives us the ability to create dynamodb tables using java code.

The most basic action is to create a table with a hash key. In this case the email of the user would be the hash key.

List<KeySchemaElement> elements = new ArrayList<KeySchemaElement>();
        KeySchemaElement keySchemaElement = new KeySchemaElement()
                .withKeyType(KeyType.HASH)
                .withAttributeName("email");
        elements.add(keySchemaElement);

        List<AttributeDefinition> attributeDefinitions = new ArrayList<>();

        attributeDefinitions.add(new AttributeDefinition()
                .withAttributeName("email")
                .withAttributeType(ScalarAttributeType.S));

        CreateTableRequest createTableRequest = new CreateTableRequest()
                .withTableName("Users")
                .withKeySchema(elements)
                .withProvisionedThroughput(new ProvisionedThroughput()
                        .withReadCapacityUnits(5L)
                        .withWriteCapacityUnits(5L))
                .withAttributeDefinitions(attributeDefinitions);

        amazonDynamoDB.createTable(createTableRequest);   

What we did is creating the Users table using his email for a hash key.

The next table will be called Logins. Logins should keep track each time the user logged in. To do so apart from using a hash key we will also use a range key.

        List<KeySchemaElement> elements = new ArrayList<KeySchemaElement>();
        KeySchemaElement hashKey = new KeySchemaElement()
                .withKeyType(KeyType.HASH)
                .withAttributeName("email");
        KeySchemaElement rangeKey = new KeySchemaElement()
                .withKeyType(KeyType.RANGE)
                .withAttributeName("timestamp");
        elements.add(hashKey);
        elements.add(rangeKey);


        List<AttributeDefinition> attributeDefinitions = new ArrayList<>();

        attributeDefinitions.add(new AttributeDefinition()
                .withAttributeName("email")
                .withAttributeType(ScalarAttributeType.S));


        attributeDefinitions.add(new AttributeDefinition()
                .withAttributeName("timestamp")
                .withAttributeType(ScalarAttributeType.N));

        CreateTableRequest createTableRequest = new CreateTableRequest()
                .withTableName("Logins")
                .withKeySchema(elements)
                .withProvisionedThroughput(new ProvisionedThroughput()
                        .withReadCapacityUnits(5L)
                        .withWriteCapacityUnits(5L))
                .withAttributeDefinitions(attributeDefinitions);


        amazonDynamoDB.createTable(createTableRequest);

By using the email as a hash key we can query for the logins of the specific user.
By using the date that the login occured as a range key with can find sort the login entries or perform advanced queries based on the login date for a specific user.

However most of the times a hash key and range key are not enough for our needs.
DynamoDB provides us with Global Secondary indexes and Local secondary Indexes.

We will create the table SupervisorS. The hash key of Supervisor would be his name. A supervisor will work for a company. The company will be our global secondary index. Since the companies own more than one factories the field factory would be the range key.

List<KeySchemaElement> elements = new ArrayList<>();
        KeySchemaElement hashKey = new KeySchemaElement()
                .withKeyType(KeyType.HASH)
                .withAttributeName("name");
        elements.add(hashKey);

        List<GlobalSecondaryIndex> globalSecondaryIndices = new ArrayList<>();

        ArrayList<KeySchemaElement> indexKeySchema = new ArrayList<>();

        indexKeySchema.add(new KeySchemaElement()
                .withAttributeName("company")
                .withKeyType(KeyType.HASH));  //Partition key
        indexKeySchema.add(new KeySchemaElement()
                .withAttributeName("factory")
                .withKeyType(KeyType.RANGE));  //Sort key


        GlobalSecondaryIndex factoryIndex = new GlobalSecondaryIndex()
                .withIndexName("FactoryIndex")
                .withProvisionedThroughput(new ProvisionedThroughput()
                        .withReadCapacityUnits((long) 10)
                        .withWriteCapacityUnits((long) 1))
                .withKeySchema(indexKeySchema)
                .withProjection(new Projection().withProjectionType(ProjectionType.ALL));
        globalSecondaryIndices.add(factoryIndex);

        List<AttributeDefinition> attributeDefinitions = new ArrayList<>();

        attributeDefinitions.add(new AttributeDefinition()
                .withAttributeName("name")
                .withAttributeType(ScalarAttributeType.S));
        attributeDefinitions.add(new AttributeDefinition()
                .withAttributeName("company")
                .withAttributeType(ScalarAttributeType.S));
        attributeDefinitions.add(new AttributeDefinition()
                .withAttributeName("factory")
                .withAttributeType(ScalarAttributeType.S));

        CreateTableRequest createTableRequest = new CreateTableRequest()
                .withTableName("Supervisors")
                .withKeySchema(elements)
                .withProvisionedThroughput(new ProvisionedThroughput()
                        .withReadCapacityUnits(5L)
                        .withWriteCapacityUnits(5L))
                .withGlobalSecondaryIndexes(factoryIndex)
                .withAttributeDefinitions(attributeDefinitions);

        amazonDynamoDB.createTable(createTableRequest);

Next table would be the table Companies. The hash key would be the parent company and the range key the subsidiary company. Each company has a CEO. The CEO would be the range key for the local secondary index.

List<KeySchemaElement> elements = new ArrayList<>();
        KeySchemaElement hashKey = new KeySchemaElement()
                .withKeyType(KeyType.HASH)
                .withAttributeName("name");
        KeySchemaElement rangeKey = new KeySchemaElement()
                .withKeyType(KeyType.RANGE)
                .withAttributeName("subsidiary");

        elements.add(hashKey);
        elements.add(rangeKey);

        List<LocalSecondaryIndex> localSecondaryIndices = new ArrayList<>();

        ArrayList<KeySchemaElement> indexKeySchema = new ArrayList<>();

        indexKeySchema.add(new KeySchemaElement()
                .withAttributeName("name")
                .withKeyType(KeyType.HASH));
        indexKeySchema.add(new KeySchemaElement()
                .withAttributeName("ceo")
                .withKeyType(KeyType.RANGE));

        LocalSecondaryIndex ceoIndex = new LocalSecondaryIndex()
                .withIndexName("CeoIndex")
                .withKeySchema(indexKeySchema)
                .withProjection(new Projection().withProjectionType(ProjectionType.ALL));
        localSecondaryIndices.add(ceoIndex);

        List<AttributeDefinition> attributeDefinitions = new ArrayList<>();

        attributeDefinitions.add(new AttributeDefinition()
                .withAttributeName("name")
                .withAttributeType(ScalarAttributeType.S));
        attributeDefinitions.add(new AttributeDefinition()
                .withAttributeName("subsidiary")
                .withAttributeType(ScalarAttributeType.S));
        attributeDefinitions.add(new AttributeDefinition()
                .withAttributeName("ceo")
                .withAttributeType(ScalarAttributeType.S));

        CreateTableRequest createTableRequest = new CreateTableRequest()
                .withTableName("Companies")
                .withKeySchema(elements)
                .withProvisionedThroughput(new ProvisionedThroughput()
                        .withReadCapacityUnits(5L)
                        .withWriteCapacityUnits(5L))
                .withLocalSecondaryIndexes(localSecondaryIndices)
                .withAttributeDefinitions(attributeDefinitions);

        amazonDynamoDB.createTable(createTableRequest);

You can find the source code on github.

I’ve compiled a cheat sheet that lists dynamodb functions in Java
Sign up in the link to receive it.

Implement a DynamoDB docker Image

When you use DynamoDB and you have good codebase test coverage, chances are that you tend to use a lot local DynamoDB.
Docker comes really in handy in order to distribute a pre-configured local dynamo db among your dev teams or your Continuous integration server.

I will use a Centos image.

We will need Java.
I prefer the oracle jdk therefore I have to accept the license and download locally the java rpm.

In case you want open jdk you can just install it through yum

So we create the Dockerfile.
I will use the default port which is 8000 so I will expose port 8000.
jdk-8u91-linux-x64.rpm is the oracle java I downloaded previously.

FROM centos

ADD jdk-8u91-linux-x64.rpm /

RUN rpm -Uvh jdk-8u91-linux-x64.rpm

RUN rm /jdk-8u91-linux-x64.rpm

RUN mkdir /opt/DynamoDB

RUN curl -O -L http://dynamodb-local.s3-website-us-west-2.amazonaws.com/dynamodb_local_latest.tar.gz

RUN mv dynamodb_local_latest.tar.gz /opt/DynamoDB/

RUN cd /opt/DynamoDB && tar xvf dynamodb_local_latest.tar.gz && rm dynamodb_local_latest.tar.gz

EXPOSE 8000

ENTRYPOINT ["java","-Djava.library.path=/opt/DynamoDB/DynamoDBLocal_lib","-jar","/opt/DynamoDB/DynamoDBLocal.jar","-sharedDb"]

Then we build our image

docker build -t dynamodb .

No we run the container on the background

docker run -p 8000:8000 -d dynamodb

AWS SQS and Spring JMS integration

Amazon WEB Services provide us with the SQS messaging service. The java sdk for sqs is compatible with JMS.

Therefore instead of using SQS as a simple spring bean we can integrate it with the JMS integration framework that spring provides.

I will use spring-boot and gradle

The gradle file

group 'com.gkatzioura.sqstesting'
version '1.0-SNAPSHOT'

buildscript {
    repositories {
        mavenCentral()
    }
    dependencies {
        classpath("org.springframework.boot:spring-boot-gradle-plugin:1.2.7.RELEASE")
    }
}

apply plugin: 'java'
apply plugin: 'idea'
apply plugin: 'spring-boot'

sourceCompatibility = 1.8

repositories {
    mavenCentral()
}

dependencies {
    compile "org.springframework.boot:spring-boot-starter-thymeleaf"
    compile "com.amazonaws:aws-java-sdk:1.10.55"
    compile "org.springframework:spring-jms"
    compile "com.amazonaws:amazon-sqs-java-messaging-lib:1.0.0"
    compile 'org.slf4j:slf4j-api:1.6.6'
    compile 'ch.qos.logback:logback-classic:1.0.13'
    testCompile "junit:junit:4.11"
}

The application class

package com.gkatzioura.sqstesting;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

/**
 * Created by gkatziourasemmanouil on 8/26/15.
 */
@SpringBootApplication
public class Application {

    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }

}

And the application yml file

queue:
endpoint: http://localhost:9324
name: sample-queue

I specify a localhost endpoint since I use ElasticMq.

The SQSConfig class is a configuration class in order to have a SQS client as a spring bean available.

package com.gkatzioura.sqstesting.config;

import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.sqs.AmazonSQSClient;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

/**
 * Created by gkatziourasemmanouil on 25/02/16.
 */
@Configuration
public class SQSConfig {

    @Value("${queue.endpoint}")
    private String endpoint;

    @Value("${queue.name}")
    private String queueName;

    @Bean
    public AmazonSQSClient createSQSClient() {

        AmazonSQSClient amazonSQSClient = new AmazonSQSClient(new BasicAWSCredentials("",""));
        amazonSQSClient.setEndpoint(endpoint);

        amazonSQSClient.createQueue(queueName);

        return amazonSQSClient;
    }

}

The SQSListener is a listener class implementing the JMS MessageListener interface.

package com.gkatzioura.sqstesting.listeners;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;

import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageListener;
import javax.jms.TextMessage;

/**
 * Created by gkatziourasemmanouil on 25/02/16.
 */
@Component
public class SQSListener implements MessageListener {

    private static final Logger LOGGER = LoggerFactory.getLogger(SQSListener.class);

    public void onMessage(Message message) {

        TextMessage textMessage = (TextMessage) message;

        try {
            LOGGER.info("Received message "+ textMessage.getText());
        } catch (JMSException e) {
            LOGGER.error("Error processing message ",e);
        }
    }
}

The JMSSQSConfig class contains configuration for the JmsTemplate and the DefaultMessageListenerContainer. Through the JMSSQSConfig class we register the JMS MessageListeners.

package com.gkatzioura.sqstesting.config;

import com.amazon.sqs.javamessaging.SQSConnectionFactory;
import com.amazonaws.auth.*;
import com.gkatzioura.sqstesting.listeners.SQSListener;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.jms.core.JmsTemplate;
import org.springframework.jms.listener.DefaultMessageListenerContainer;

/**
 * Created by gkatziourasemmanouil on 25/02/16.
 */
@Configuration
public class JMSSQSConfig {

    @Value("${queue.endpoint}")
    private String endpoint;

    @Value("${queue.name}")
    private String queueName;

    @Autowired
    private SQSListener sqsListener;

    @Bean
    public DefaultMessageListenerContainer jmsListenerContainer() {

        SQSConnectionFactory sqsConnectionFactory = SQSConnectionFactory.builder()
                .withAWSCredentialsProvider(new DefaultAWSCredentialsProviderChain())
                .withEndpoint(endpoint)
                .withAWSCredentialsProvider(awsCredentialsProvider)
                .withNumberOfMessagesToPrefetch(10).build();

        DefaultMessageListenerContainer dmlc = new DefaultMessageListenerContainer();
        dmlc.setConnectionFactory(sqsConnectionFactory);
        dmlc.setDestinationName(queueName);

        dmlc.setMessageListener(sqsListener);

        return dmlc;
    }

    @Bean
    public JmsTemplate createJMSTemplate() {

        SQSConnectionFactory sqsConnectionFactory = SQSConnectionFactory.builder()
                .withAWSCredentialsProvider(awsCredentialsProvider)
                .withEndpoint(endpoint)
                .withNumberOfMessagesToPrefetch(10).build();

        JmsTemplate jmsTemplate = new JmsTemplate(sqsConnectionFactory);
        jmsTemplate.setDefaultDestinationName(queueName);
        jmsTemplate.setDeliveryPersistent(false);


        return jmsTemplate;
    }

    private final AWSCredentialsProvider awsCredentialsProvider = new AWSCredentialsProvider() {
        @Override
        public AWSCredentials getCredentials() {
            return new BasicAWSCredentials("", "");
        }

        @Override
        public void refresh() {

        }
    };

}

MessageService is a service that uses JMSTemplate in order to send messages to the queue

package com.gkatzioura.sqstesting;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.jms.core.JmsTemplate;
import org.springframework.jms.core.MessageCreator;
import org.springframework.stereotype.Service;

import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.Session;

/**
 * Created by gkatziourasemmanouil on 28/02/16.
 */
@Service
public class MessageService {

    @Autowired
    private JmsTemplate jmsTemplate;

    @Value("${queue.name}")
    private String queueName;

    private static final Logger LOGGER = LoggerFactory.getLogger(MessageService.class);

    public void sendMessage(final String message) {

        jmsTemplate.send(queueName, new MessageCreator() {
            @Override
            public Message createMessage(Session session) throws JMSException {
                return session.createTextMessage(message);
            }
        });
    }

}

Last but not least a Controller is added. The controller sends the post request body to the queue as a message.

package com.gkatzioura.sqstesting;

import com.amazonaws.util.IOUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.io.InputStream;

/**
 * Created by gkatziourasemmanouil on 24/02/16.
 */
@Controller
@RequestMapping("/main")
public class MainController {

    @Autowired
    private MessageService messageService;

    @RequestMapping(value = "/write",method = RequestMethod.POST)
    public void write(HttpServletRequest servletRequest,HttpServletResponse servletResponse) throws IOException {

        InputStream inputStream = servletRequest.getInputStream();

        String message = IOUtils.toString(inputStream);

        messageService.sendMessage(message);
    }

}

You can download the source code here.

Testing Amazon Web Services Codebase: SQS

SQS is a que messaging service. The sqs client is based on http and polling instead of following a listener approach therefore it is easy to use mocking utils to mock its functionality. However there is an alternative closer to the actual service.

ElasticMq is a queueing system written in scala which is compatible with the sqs interfaces.

By issuing

java -jar elasticmq-server-0.8.12.jar

you have an ElasticMQ instance up and running on port 9324.

Since it is compatible with the amazon sqs you will use exactly the same procedure to establish a connection and send messages.

AmazonSQSClient amazonSQSClient = new AmazonSQSClient(new BasicAWSCredentials("",""));
amazonSQSClient.setEndpoint("http://localhost:9324");
amazonSQSClient.createQueue("testque");

SQS will definetely come in handy for any Integration tests or even unit tests.

Testing Amazon Web Services Codebase: DynamoDB and S3

When switching to an amazon web services infrastructure, one of the main challenges is testing.

Components such as DynamoDB and S3 come in handy however they come with a cost.
When it comes to continuous integration you will end up spending resources if you use the amazon components.

Some of these components have their clones that are capable of running locally.

You can use DynamoDB locally.

By issuing

java -Djava.library.path=./DynamoDBLocal_lib -jar DynamoDBLocal.jar -sharedDb

you will have a local DynamoDB instance up and running.

Also on http://localhost:8000/shell you have a DynamoDB Shell (based on javascript) which will help you to get started.

In order to connect to the local instance you need to set the endpoint on your DynamoDB client.

On Java

AmazonDynamoDBClient client = new AmazonDynamoDBClient();
client.setEndpoint("http://localhost:8000"); 

On Node.js

var AWS = require('aws-sdk');
var config = {"endpoint":"http://localhost:8000"};
var client = new AWS.DynamoDB(config);

Another base component of Amazon Web Services is the Simple Storage Service (S3).

Luckily we have fake-s3 . Fake-S3 a lightweight server clone of amazon S3, exists.

Installing and running fake-s3 is pretty simple

gem install fakes3
fakes3 -r /mnt/fakes3_root -p 4567

In order to connect you have to specify the endpoint

On Java

AmazonS3 client = new AmazonS3Client();
client.setEndpoint("http://localhost:8000"); 

On Node.js

var AWS = require('aws-sdk');
var config = {"endpoint":"http://localhost:8000"};
var client = new AWS.S3(config);

These tools will come in handy during the development face, especially when you get started and want a simple example. By running them locally you avoid overhead of permissions and configurations that come with each component you upload on amazon.