Apache Ignite on your Kubernetes Cluster Part 4: Deployment explained

Previously we saw the Ignite configuration that comes with the Kubernetes installation.
The default configuration does not have persistence enabled so we won’t focus on any storage classes provided by the helm chart.

The default installation uses a stateful set. You can find more information on a stateful set on the Kubernetes documentation.

> kubectl get statefulset ignite-cache -o yaml
apiVersion: apps/v1
kind: StatefulSet
  creationTimestamp: 2020-04-09T12:29:04Z
  generation: 1
    app.kubernetes.io/instance: ignite-cache
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ignite
    helm.sh/chart: ignite-1.0.1
  name: ignite-cache
  namespace: default
  resourceVersion: "281390"
  selfLink: /apis/apps/v1/namespaces/default/statefulsets/ignite-cache
  uid: fcaa7bef-84cd-4e7c-aa33-a4312a1d47a9
  podManagementPolicy: OrderedReady
  replicas: 2
  revisionHistoryLimit: 10
      app: ignite-cache
  serviceName: ignite-cache
      creationTimestamp: null
        app: ignite-cache
      - env:
        - name: IGNITE_QUIET
          value: "false"
        - name: JVM_OPTS
          value: -Djava.net.preferIPv4Stack=true
        - name: OPTION_LIBS
          value: ignite-kubernetes,ignite-rest-http
        image: apacheignite/ignite:2.7.6
        imagePullPolicy: IfNotPresent
        name: ignite
        - containerPort: 11211
          protocol: TCP
        - containerPort: 47100
          protocol: TCP
        - containerPort: 47500
          protocol: TCP
        - containerPort: 49112
          protocol: TCP
        - containerPort: 10800
          protocol: TCP
        - containerPort: 8080
          protocol: TCP
        - containerPort: 10900
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        - mountPath: /opt/ignite/apache-ignite/config
          name: config-volume
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: ignite-cache
      serviceAccountName: ignite-cache
      terminationGracePeriodSeconds: 30
      - configMap:
          defaultMode: 420
          - key: ignite-config.xml
            path: default-config.xml
          name: ignite-cache-configmap
        name: config-volume
      partition: 0
    type: RollingUpdate
  replicas: 0

As you can see the Ingite configuration has been mounted through the configmap. Also you can see that this pod will use a specific service account.
Through the environment variables certain libraries are enabled which provide more features on the Ignite cluster. Also the ports needed for the communication and various protocols are being specified.

The last step is the service. All the ignite nodes shall be load balancer behind the Kubernetes service.

> kubectl get svc ignite-cache -o yaml
apiVersion: v1
kind: Service
  creationTimestamp: 2020-04-09T12:29:04Z
    app: ignite-cache
  name: ignite-cache
  namespace: default
  resourceVersion: "281389"
  selfLink: /api/v1/namespaces/default/services/ignite-cache
  uid: 5be68e28-a57c-4cb5-b610-b708bff80da7
  clusterIP: None
  - name: jdbc
    port: 11211
    protocol: TCP
    targetPort: 11211
  - name: spi-communication
    port: 47100
    protocol: TCP
    targetPort: 47100
  - name: spi-discovery
    port: 47500
    protocol: TCP
    targetPort: 47500
  - name: jmx
    port: 49112
    protocol: TCP
    targetPort: 49112
  - name: sql
    port: 10800
    protocol: TCP
    targetPort: 10800
  - name: rest
    port: 8080
    protocol: TCP
    targetPort: 8080
  - name: thin-clients
    port: 10900
    protocol: TCP
    targetPort: 10900
    app: ignite-cache
  sessionAffinity: None
  type: ClusterIP
  loadBalancer: {}

Whether you add a new node or you add an ignite client node your ignite cluster shall be reached through this Kubernetes service. Apart from, that based on the Kubernetes services you can make this cache public or internal.

Apache Ignite on your Kubernetes Cluster Part 3: Configuration explained

Previously we had a look on the RBAC needed for and ignite cluster in Kubernetes.
This blogs focuses on the deployment and the configuration of the cache.

The default ignite installation uses and xml based configuration. It is easy to mount files using configmaps.

> kubectl get configmap ignite-cache-configmap -o yaml
NAME                     DATA   AGE
ignite-cache-configmap   1      32d
gkatzioura@MacBook-Pro-2 templates % kubectl get configmap ignite-cache-configmap -o yaml
apiVersion: v1
  ignite-config.xml: "....\n"
kind: ConfigMap
  creationTimestamp: 2020-03-07T22:23:50Z
  name: ignite-cache-configmap
  namespace: default
  resourceVersion: "137521"
  selfLink: /api/v1/namespaces/default/configmaps/ignite-cache-configmap
  uid: ff530e3d-10d6-4708-817f-f9845886c1b0

Since viewing the xml from the configmap is cumbersome this is the actual xml

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
	   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="
		       http://www.springframework.org/schema/beans        http://www.springframework.org/schema/beans/spring-beans.xsd">
	<bean class="org.apache.ignite.configuration.IgniteConfiguration">
				name="peerClassLoadingEnabled" value="false"/>
		<property name="dataStorageConfiguration">
			<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
			<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
				<property name="ipFinder">
					<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder">
						<property name="namespace" value="default"/>
								name="serviceName" value="ignite-cache"/>

The default DataStorageConfiguration is being used.
What you can see different from other ignite installations is TCP discovery. The tcp discover used is using the Kubernetes TCP based discovery.

The next blog focuses on the services and the deployment.

Apache Ignite on your Kubernetes Cluster Part 2: RBAC Explained

So previously we had a vanilla installation of Apache Ignite on Kubernetes.

You had a cache service running however all you did was installing a helm chart.
In this blog we shall evaluate what is installed and take notes for our futures helm charts.

The first step would be to view the helm chart.

> helm list
NAME        	NAMESPACE	REVISION	UPDATED                             	STATUS  	CHART       	APP VERSION
ignite-cache	default  	1       	2020-03-07 22:23:49.918924 +0000 UTC	deployed	ignite-1.0.1	2.7.6

Now let’s download it

> helm fetch stable/ignite
> tar xvf ignite-1.0.1.tgz
> cd ignite/; ls -R
Chart.yaml	README.md	templates	values.yaml

NOTES.txt			account-role.yaml		persistence-storage-class.yaml	service-account.yaml		svc.yaml
_helpers.tpl			configmap.yaml			role-binding.yaml		stateful-set.yaml		wal-storage-class.yaml

Reading through the template files is a bit challenging (well they are tempaltes :P) so we shall just check what was installed through our previous blog.

Let’s get started with the account-role. The cluster role that ignite shall use needs to be able to get/list/watch the pods and the endpoints. It makes sense since there is a need for discovery between the nodes.

> kubectl get ClusterRole ignite-cache -o yaml
kind: ClusterRole
  creationTimestamp: 2020-03-07T22:23:50Z
  name: ignite-cache
  resourceVersion: "137525"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/ignite-cache
  uid: 0cad0689-2f94-4b74-87bc-b468e2ac78ae
- apiGroups:
  - ""
  - pods
  - endpoints
  - get
  - list
  - watch

In order to use this role you need a service account. A service account is create with a token.

> kubectl get serviceaccount ignite-cache -o yaml
apiVersion: v1
kind: ServiceAccount
  creationTimestamp: 2020-03-07T22:23:50Z
  name: ignite-cache
  namespace: default
  resourceVersion: "137524"
  selfLink: /api/v1/namespaces/default/serviceaccounts/ignite-cache
  uid: 7aab67e5-04db-41a8-b73d-e76e34ca1d8e
- name: ignite-cache-token-8rln4

Then we have the role binding. We have a new service account called the ignite-cache which has the role ignite-cache.

> kubectl get ClusterRoleBinding ignite-cache -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
  creationTimestamp: 2020-03-07T22:23:50Z
  name: ignite-cache
  resourceVersion: "137526"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/ignite-cache
  uid: 1e180bd1-567f-4979-a278-ba2e420ed482
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ignite-cache
- kind: ServiceAccount
  name: ignite-cache
  namespace: default

It is important for you ignite workloads to use this service account and its token. By doing so they have the permissions to discover the other nodes in your cluster.

The next blog focuses on the configuration.

Apache Ignite on your Kubernetes Cluster Part 1: Vanilla installation

By all means apache Ignite is an Amazing Open Source project.
Don’t assume it’s just a  Cache. It provides way more.


Kubernetes gets more popular by the day and is also a very convenient tool.
In this tutorial we shall integrate ignite and Kubernetes.

The first step would be to spin up Minikube.

To get ignite on your Kubernetes installation the first step would be to install the helm chart.

>helm repo add stable https://kubernetes-charts.storage.googleapis.com
>helm install ignite-cache stable/ignite
NAME: ignite-cache
LAST DEPLOYED: Sat Mar  7 22:23:49 2020
NAMESPACE: default
STATUS: deployed
To check cluster state please run:

kubectl exec -n default ignite-cache-0 -- /opt/ignite/apache-ignite/bin/control.sh --state

Eventually after this command is issued it is expected to have an ignite cache setup on your kubernetes cluster.

>kubectl get pods
ignite-cache-0   1/1     Running   0          79s
ignite-cache-1   1/1     Running   0          13s
>kubectl get svc ignite-cache
ignite-cache   ClusterIP   None         <none>        11211/TCP,47100/TCP,47500/TCP,49112/TCP,10800/TCP,8080/TCP,10900/TCP   6m24s

To those familiar with Kubernetes an ignite cache has just been spinned up in your kubernetes cluster and your applications can use the ignite service within the cluster.
The next blog focuses on the service account needed.

Use local docker image on minikube.

You use Minikube and you want to run your development images that you create locally. This might seem tricky since Minikube needs to download your images from a registry however you images are being uploaded on your local registry.

In any case you can still use you local images with Minikube so let’s get started.

Before running any container let’s issue.

> eval $(minikube docker-env)

This actually reuses the docker host from Minikube for your current bash session.

See for yourself.

> minikube docker-env
export DOCKER_HOST="tcp://"
export DOCKER_CERT_PATH="/Users/gkatzioura/.minikube/certs"
# Run this command to configure your shell:
# eval $(minikube docker-env)

Then spin up an nginx image. Most of the commands are taken from this tutorial.

>docker run -d -p 8080:80 --name my-nginx nginx
>docker ps --filter name=my-nginx
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                  NAMES
128ce006ecae        nginx               "nginx -g 'daemon of…"   13 seconds ago      Up 12 seconds>80/tcp   my-nginx

Now let’s create an image from the running container.

docker commit 128ce006ecae dockerimage:version1

Then let’s run our custom image on minikube.

kubectl create deployment test-image --image=dockerimage:version1

Let’s also expose the service

kubectl expose deployment test-image --type=LoadBalancer --port=80

Let’s take to the next level and try to wget our service

> kubectl exec -it podwithbinbash /bin/bash
bash-4.4# wget test-image
Connecting to test-image (
index.html           100% |***********************************************************************************************************|   612  0:00:00 ETA
bash-4.4# cat index.html
<!DOCTYPE html>
<title>Welcome to nginx!</title>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>

Take extra attention that the above will work only on the terminal that you executed the command

eval $(minikube docker-env)

If you want to you can just setup your bash_profile to do it for every terminal but this is up to you.
Eventually this is one of the quick ways to use you local images on Minikube and most probably there are others available.

Read replicas and Spring Data Part 2: Configuring the base project

In our previous post we set up multiple PostgreSQL instances with the same data.
Our next step would be to configure our spring project by using the both servers.

As stated previously we shall use some of the code taken from the Spring Boot JPA post, since we use exactly the same database.

This shall be our gradle build file

plugins {
	id 'org.springframework.boot' version '2.1.9.RELEASE'
	id 'io.spring.dependency-management' version '1.0.8.RELEASE'
	id 'java'

group = 'com.gkatzioura'
version = '0.0.1-SNAPSHOT'
sourceCompatibility = '1.8'

repositories {

dependencies {
	implementation 'org.springframework.boot:spring-boot-starter-data-jpa'
	implementation 'org.springframework.boot:spring-boot-starter-web'
	implementation "org.postgresql:postgresql:42.2.8"
	testImplementation 'org.springframework.boot:spring-boot-starter-test'

Now let’s proceed on creating the model based on the table created on the previous blog.

package com.gkatzioura.springdatareadreplica.entity;

import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;
import javax.persistence.Table;

@Table(name = "employee", catalog="spring_data_jpa_example")
public class Employee {

    @Column(name = "id")
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    @Column(name = "firstname")
    private String firstName;

    @Column(name = "lastname")
    private String lastname;

    @Column(name = "email")
    private String email;

    @Column(name = "age")
    private Integer age;

    @Column(name = "salary")
    private Integer salary;

    public Long getId() {
        return id;

    public void setId(Long id) {
        this.id = id;

    public String getFirstName() {
        return firstName;

    public void setFirstName(String firstName) {
        this.firstName = firstName;

    public String getLastname() {
        return lastname;

    public void setLastname(String lastname) {
        this.lastname = lastname;

    public String getEmail() {
        return email;

    public void setEmail(String email) {
        this.email = email;

    public Integer getAge() {
        return age;

    public void setAge(Integer age) {
        this.age = age;

    public Integer getSalary() {
        return salary;

    public void setSalary(Integer salary) {
        this.salary = salary;


And the next step is to create a spring data repository.

package com.gkatzioura.springdatareadreplica.repository;

import org.springframework.data.jpa.repository.JpaRepository;
import com.gkatzioura.springdatareadreplica.entity.Employee;

public interface EmployeeRepository extends JpaRepository<Employee,Long> {

Also we are going to add a controller.

package com.gkatzioura.springdatareadreplica.controller;

import java.util.List;

import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import com.gkatzioura.springdatareadreplica.entity.Employee;
import com.gkatzioura.springdatareadreplica.repository.EmployeeRepository;

public class EmployeeContoller {

    private final EmployeeRepository employeeRepository;

    public EmployeeContoller(EmployeeRepository employeeRepository) {
        this.employeeRepository = employeeRepository;

    public List<Employee> getEmployees() {
        return employeeRepository.findAll();


All that it takes is to just add the right properties in you application.yaml

    platform: postgres
    driverClassName: org.postgresql.Driver
    username: db-user
    password: your-password
    url: jdbc:postgresql://

Spring boot has made it possible nowadays not to bother with any JPA configurations.

This is all you need in order to run the application. Once your application is running just try to fetch the employees.

curl http://localhost:8080/employee

As you have seen we did not do any JPA configuration. Since Spring Boot 2 specifying the database url is sufficient for the auto configuration to kick in and do all this configuration for you.

However in our case we want to have multiple datasource and entity manager configurations. In the next post we shall configure the entity managers for our application.

Read replicas and Spring Data Part 1: Configuring the Databases

This is a series of blog posts on our quest to increase our application’s performance by utilizing read replicas.

For this project our goal is to set up our spring data application and use read repositories for writes and
repositories based on read replicas for reads.

In order to simulate this environment we shall use PostgreSQL instances through Docker.

The motives are simple. Your Spring application has become increasingly popular and you want it to handle more requests. Most of the applications out there have a higher demand for read operations rather than write operations. Thus I assume that your application falls into the same category.
Although SQL databases are not horizontally scalable on their own, you can work you way with them by using read replicas.

Our goal is not to make an actual Read replication in PostgreSQL

thereforeinstead of configuring any replication

we will just copy some data from both databases

This is the script we shall use to populate the databases.

set -e

psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" &amp;lt;&amp;lt;-EOSQL
    create schema spring_data_jpa_example;

    create table spring_data_jpa_example.employee(
        firstname   TEXT    NOT NULL,
        lastname    TEXT    NOT NULL,
        email       TEXT    not null,
        age         INT     NOT NULL,
        salary         real,

    insert into spring_data_jpa_example.employee (firstname,lastname,email,age,salary)
    values ('John','Doe 1','john1@doe.com',18,1234.23);
    insert into spring_data_jpa_example.employee (firstname,lastname,email,age,salary)
    values ('John','Doe 2','john2@doe.com',19,2234.23);
    insert into spring_data_jpa_example.employee (firstname,lastname,email,age,salary)
    values ('John','Doe 3','john3@doe.com',20,3234.23);
    insert into spring_data_jpa_example.employee (firstname,lastname,email,age,salary)
    values ('John','Doe 4','john4@doe.com',21,4234.23);
    insert into spring_data_jpa_example.employee (firstname,lastname,email,age,salary)
    values ('John','Doe 5','john5@doe.com',22,5234.23);

Since we shall use and Docker and Docker Compose the script above shall be used in order to initialize the database.
Now on to create our Docker Compose stack.

version: '3.5'

    image: postgres
    restart: always
      POSTGRES_USER: db-user
      POSTGRES_PASSWORD: your-password
      POSTGRES_DB: postgres
      - postgresql-network
      - ""
      - $PWD/init-db-script.sh:/docker-entrypoint-initdb.d/init-db-script.sh
    image: postgres
    restart: always
      POSTGRES_USER: db-user
      POSTGRES_PASSWORD: your-password
      POSTGRES_DB: postgres
      - postgresql-network
      - ""
      - $PWD/init-db-script.sh:/docker-entrypoint-initdb.d/init-db-script.sh
    name: postgresql-network

As you see our configuration is pretty simple. If you are careful enough you would see that I gave the number one to the read-db. This is because in the future we will add more replicas to it.

What I also did is bounding the machines to different local ips.

If you have problem binding addresses like 127.0.0.*:5432
You should try

sudo ifconfig lo0 alias up
sudo ifconfig lo0 alias up

If you are unsuccessful then just change the ports and it will work. It might not be as convenient but it’s still ok.

So let’s get up and running our Docker Compose stack.

docker-compose -f ./postgresql-stack.yaml up

We must be able to query data in both postgresql instances.

docker exec -it deploy_read-db-1_1 /bin/bash
root@07c502968cb3:/# psql -v --username "$POSTGRES_USER" --dbname "$POSTGRES_DB"
db-user=# select*from spring_data_jpa_example.employee;
 id | firstname | lastname |     email     | age | salary
  1 | John      | Doe 1    | john1@doe.com |  18 | 1234.23
  2 | John      | Doe 2    | john2@doe.com |  19 | 2234.23
  3 | John      | Doe 3    | john3@doe.com |  20 | 3234.23
  4 | John      | Doe 4    | john4@doe.com |  21 | 4234.23
  5 | John      | Doe 5    | john5@doe.com |  22 | 5234.23
(5 rows)

We pretty much set up for our next step. We have some databases up and running and we are going to spin up a spring application running upon them. The next blog focuses on implementing an application running upon our primary database.

Using Minikube on osx

Docker compose is making for me wonders when it comes to run some simple components on my workstation.

Spawning and simulating an infrastructure locally is fast and takes no time. Also it is lightweight.
However most teams nowadays use Kubernetes.
If you want to simulate a Kubernetes environment locally the tool to use is Minikube.

With Minikube you need to have a vm running on your workstation. This is normal, after all your workloads in a Kubernetes environment are running on multiple VMs.

So if you use OSX it’s very simple, provided you have a hypervisor installed (my default one is VirtualBox)
I just went for the local library option.

curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-darwin-amd64 \
  && chmod +x minikube
sudo mv minikube /usr/local/bin

Depending on your workstation your installation varies but is still an easy one.

So let’s get started

minikube start

Now you might face a challenge with Minikube on osx.

Progress state: NS_ERROR_FAILURE
VBoxManage: error: Failed to create the host-only adapter
VBoxManage: error: VBoxNetAdpCtl: Error while adding new interface: failed to open /dev/vboxnetctl: No such file or directory
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component HostNetworkInterfaceWrap, interface IHostNetworkInterface
VBoxManage: error: Context: "RTEXITCODE handleCreate(HandlerArg *)" at line 94 of file VBoxManageHostonly.cpp

As you can understand you need to reinstall VirtualBox. However you might face a challenge with the installation. I was getting the error ‘The installation failed’. As explained here you need to edit your osx security settings and allow ‘System software from Oracle’ to load.

At the end just do.

> minikube status
host: Running
kubelet: Running
apiserver: Running
kubectl: Correctly Configured: pointing to minikube-vm at

and you are ready to go.

Debug your container by overriding the command.

The main problem with docker on debugging has to do with images that already have the DockerImageCMD command specified.

If something goes wrong and has to do with the filesystem, or some commands that should have taken effect and they did not you need to do some troubleshooting.

Overriding the command which is executed should be helpful. I do this all the time on custom images since I need some bash in order to be able to troubleshoot.

Supposing I need to troubleshoot something on the default nginx image.

Then nginx image should run like this.

docker run --rm nginx

Now instead of running our server, we shall override the command with a bash session (enter a shell by executing /bin/sh).

docker run --rm -it --entrypoint "/bin/sh" nginx

If you also want to pass arguments to the executable specified there is also another workaround.

docker run --rm -it --entrypoint "ls" nginx -l



Docker compose: run stack dynamically

I use docker compose every day for my local development needs.


During the day I might turn on/off various databases or servers thus I need to do it fast and in a managed way.

Usually your docker-compose files contains the configuration for many containers, network, volumes etc.


version: '3.5'

    image: mongo
    restart: always
      - 27017:27017
    image: mongo-express
    restart: always
      - 8081:8081

This works if you always want the same services up and running.

However it does have a cost on resources, and most of the times you don’t need the full stack.

What you can do in this cases, would be to split them into files and choose what to use.


version: '3.5'

    image: mongo
    restart: always
      - 27017:27017


version: '3.5'

    image: mongo-express
    restart: always
      - 8081:8081

Then choosing what to use becomes very easy, just omit the file

docker-compose -f mongo.yaml -f express.yaml up