Using Minikube on osx

Docker compose is making for me wonders when it comes to run some simple components on my workstation.

Spawning and simulating an infrastructure locally is fast and takes no time. Also it is lightweight.
However most teams nowadays use Kubernetes.
If you want to simulate a Kubernetes environment locally the tool to use is Minikube.

With Minikube you need to have a vm running on your workstation. This is normal, after all your workloads in a Kubernetes environment are running on multiple VMs.

So if you use OSX it’s very simple, provided you have a hypervisor installed (my default one is VirtualBox)
I just went for the local library option.

curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-darwin-amd64 \
  && chmod +x minikube
sudo mv minikube /usr/local/bin

Depending on your workstation your installation varies but is still an easy one.

So let’s get started

minikube start

Now you might face a challenge with Minikube on osx.

Progress state: NS_ERROR_FAILURE
VBoxManage: error: Failed to create the host-only adapter
VBoxManage: error: VBoxNetAdpCtl: Error while adding new interface: failed to open /dev/vboxnetctl: No such file or directory
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component HostNetworkInterfaceWrap, interface IHostNetworkInterface
VBoxManage: error: Context: "RTEXITCODE handleCreate(HandlerArg *)" at line 94 of file VBoxManageHostonly.cpp

As you can understand you need to reinstall VirtualBox. However you might face a challenge with the installation. I was getting the error ‘The installation failed’. As explained here you need to edit your osx security settings and allow ‘System software from Oracle’ to load.

At the end just do.

> minikube status
host: Running
kubelet: Running
apiserver: Running
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.100

and you are ready to go.

Debug your container by overriding the command.

The main problem with docker on debugging has to do with images that already have the DockerImageCMD command specified.

If something goes wrong and has to do with the filesystem, or some commands that should have taken effect and they did not you need to do some troubleshooting.

Overriding the command which is executed should be helpful. I do this all the time on custom images since I need some bash in order to be able to troubleshoot.

Supposing I need to troubleshoot something on the default nginx image.

Then nginx image should run like this.

docker run --rm nginx

Now instead of running our server, we shall override the command with a bash session (enter a shell by executing /bin/sh).


docker run --rm -it --entrypoint "/bin/sh" nginx

If you also want to pass arguments to the executable specified there is also another workaround.


docker run --rm -it --entrypoint "ls" nginx -l

 

 

Docker compose: run stack dynamically

I use docker compose every day for my local development needs.

DockerImage

During the day I might turn on/off various databases or servers thus I need to do it fast and in a managed way.

Usually your docker-compose files contains the configuration for many containers, network, volumes etc.

stack.yaml

version: '3.5'

services:
  mongo:
    image: mongo
    restart: always
    ports:
      - 27017:27017
    environment:
      MONGO_INITDB_ROOT_USERNAME: username
      MONGO_INITDB_ROOT_PASSWORD: password
  mongo-express:
    image: mongo-express
    restart: always
    ports:
      - 8081:8081
    environment:
      ME_CONFIG_MONGODB_ADMINUSERNAME: username
      ME_CONFIG_MONGODB_ADMINPASSWORD: password

This works if you always want the same services up and running.

However it does have a cost on resources, and most of the times you don’t need the full stack.

What you can do in this cases, would be to split them into files and choose what to use.

mongo.yaml

version: '3.5'

services:
  mongo:
    image: mongo
    restart: always
    ports:
      - 27017:27017
    environment:
      MONGO_INITDB_ROOT_USERNAME: username
      MONGO_INITDB_ROOT_PASSWORD: password

express.yaml

version: '3.5'

services:
  mongo-express:
    image: mongo-express
    restart: always
    ports:
      - 8081:8081
    environment:
      ME_CONFIG_MONGODB_ADMINUSERNAME: username
      ME_CONFIG_MONGODB_ADMINPASSWORD: password

Then choosing what to use becomes very easy, just omit the file

docker-compose -f mongo.yaml -f express.yaml up

Pass multiple commands on Docker run

Docker apart form serving our workloads efficiently is also an amazing tool when it comes to not installing additional binaries to your workstation.

DockerImage

Eventually you will find it very easy and simple to just run only one command on docker.

For example I want to run a hello world in go.

My source code is going to be the simple hello world.

package main

import "fmt"

func main() {
    fmt.Println("hello world")
}

Pretty simple! The file shall be named hello_world.go

Now let’s run this in a container.

docker run -v $(pwd):/go/src/app --rm --name helloworld golang:1.8 go run src/app/hello_world.go

How about installing some go packages and the run our application in one liner?

If you try to do so, you shall realise that docker won’t interpret the commands the way you want, thus heres’ s how to get the result that you want.

If your image contains the /bin/bash or bin/sh binary you can pass the commands you wan to execute as a string.

docker run -v $(pwd):/go/src/app --rm --name helloworld golang:1.8 /bin/bash -c "cd src/app;go get https:yourpackage;go run src/app/hello_world.go

That’s it! Now you can run complex bash one-liners without worrying on installing additional software on your workstations

Spin up an InfluxDB instance with docker for testing.

It is a reality that we tend to make things harder than they might be when we try to use and connect various databases.
Since docker came out things became a lot easier.

Most databases like Mongodb, InfluxDB etc come with the binaries needed to spin up the database but also with the clients needed in order to connect. Actually it pretty much starts to become a standard.

We will make a showcase of this by using InfluxDB’s docker image and the data walkthrough.

Let’s start with spinning up the instance.

docker run --rm -p 8086:8086 --name influxdb-local influxdb

We have an influxDB instance running on port 8086 under the name influxdb-local. Once the container is stopped it will also be deleted.

First step is to connect to an influxDB shell and interact with the database.

docker exec -it influxdb-local influx
CREATE DATABASE NOAA_water_database
> exit

Now let’s import some data

docker exec -it influxdb-local /bin/bash
curl https://s3.amazonaws.com/noaa.water-database/NOAA_data.txt -o NOAA_data.txt
influx -import -path=NOAA_data.txt -precision=s -database=NOAA_water_database
rm NOAA_data.txt

Next step is to connect to the shell and query some data.

docker exec -it influxdb-local influx -precision rfc3339 -database NOAA_water_database
Connected to http://localhost:8086 version 1.4.x
InfluxDB shell 1.4.x
> SHOW measurements
name: measurements
name
----
average_temperature
h2o_feet
h2o_pH
h2o_quality
h2o_temperature
>

As you can see we just created an InfluxDB instance with data ready to execute queries and have some tests! Pretty simple and clean. Once we are done by stopping the container all data and the container included shall be removed.

Docker basics: Docker compose

Docker Compose is a tool that allows you to run multi-container applications. With compose we can use yaml files to configure our application’ services and then using a single command create and start all of the configured services. I use this tool a lot when it comes to local development in a microservice environment. It is also lightweight and needs just a small effort. Instead of managing how to run each service while developing you can have the environment and services needed preconfigured and focus on the service that you currently develop.

With docker compose we can configure a network for our services, volumes, mount-points, environmental variables just about everything.
To showcase this we are going to solve a problem. Our goal would be to extract data from mongodb using grafana. Grafana does not have out of the box support for MongoDB therefore we will have to use a plugin.

First step we shall create our networks. Creating a network is not necessary since your services once started will join the default network. We shall make a showcase of using custom networks. We shall have a network for backend services and a network for frontend services. Apparently network configuration can get more advanced and specify custom network drivers or even configure static addresses.

version: '3.5'

networks:
  frontend:
    name: frontend-network
  backend:
    name: backend-network
    internal: true

The backend network is going to be internal therefore there won’t be any outbound connectivity to the containers attached to it.

Then we shall setup our mongodb instance.

version: '3.5'

services:
  mongo:
    image: mongo
    restart: always
    environment:
      MONGO_INITDB_ROOT_USERNAME: ${MONGO_USER}
      MONGO_INITDB_ROOT_PASSWORD: ${MONGO_PASSWORD}
    volumes:
      - ${DB_PATH}:/data/db
    networks:
      - backend

As you see we specified a volume. Volumes can also be specified separately and attach them to a service.
Also we used environmental variables for the root account, you might as well have spotted that the password is going to be provided through environmental variables ie. MONGO_USER=root MONGO_PASSWORD=root docker-compose -f stack.yaml up. The same applies for the volume path too. You can have a more advanced configuration for volumes in your compose configuration and reference them from your service.

Our next goal is to setup the proxy server which shall be in the middle of our grafana and mongodb server. Since it needs a custom Dockerfile to create it, we shall do it through docker-compose. Compose has the capability to spin up a service by specifying the docker file.

So let’s start with the Dockerfile.

FROM node

WORKDIR /usr/src/mongografanaproxy

COPY . /usr/src/mongografanaproxy

EXPOSE 3333

RUN cd /usr/src/mongografanaproxy
RUN npm install
ENTRYPOINT ["npm","run","server"]

Then let’s add it to compose

version: '3.5'

services:
  mongo-proxy:
    build:
      context: .
      dockerfile: ProxyDockerfile
    restart: always
    networks:
      - backend

And the same shall be done to the Grafana image that we want to use. In stead of using a ready grafana image we shall create one with the plugin preinstalled.

FROM grafana/grafana

COPY . /var/lib/grafana/plugins/mongodb-grafana

EXPOSE 3000

version: '3.5'

services:
  grafana:
    build:
      context: .
      dockerfile: GrafanaDockerfile
    restart: always
    ports:
      - 3000:3000
    networks:
      - backend
      - frontend

Let’s wrap them all together

version: '3.5'

services:
  mongo:
    image: mongo
    restart: always
    environment:
      MONGO_INITDB_ROOT_USERNAME: ${MONGO_USER}
      MONGO_INITDB_ROOT_PASSWORD: ${MONGO_PASSWORD}
    volumes:
      - ${DB_PATH}:/data/db
    networks:
      - backend
  mongo-proxy:
    build:
      context: .
      dockerfile: ProxyDockerfile
    restart: always
    networks:
      - backend
  grafana:
    build:
      context: .
      dockerfile: GrafanaDockerfile
    restart: always
    ports:
      - 3000:3000
    networks:
      - backend
      - frontend
networks:
  frontend:
    name: frontend-network
  backend:
    name: backend-network
    internal: true

So let’s run them all together.

docker-compose -f stack.yaml build
MONGO_USER=root MONGO_PASSWORD=root DB_PATH=~/grafana-mongo  docker-compose -f stack.yaml up

The above can be found on github.

You might as well find the Docker Images, Docker Containers and Docker registry posts useful.

Kubernetes and Secrets

This is going to be a small post since it has to deal with kubernetes and secrets. Yet it is a very useful once since adding secrets is so common yet so easy to forget (guilty as charged).

So we will cover username and password, key/values, file uploading, secrets.

Upload username and password using command line.

kubectl create secret generic accountpassword --from-literal=username=yourusername --from-literal=password=yourpassword

Upload just a key

kubectl create secret generic application-key --from-literal=key=yourusername

Upload username and password through files

printf "yourusername" > username.txt
printf "yourpassword" > password.txt
kubectl create secret generic accountpassword --from-file=./username.txt --from-file=./password.txt

Then let’s upload a secret. Be aware that this secret can be used with your secret rules.

kubectl create secret tls your-server-tls --key ./privkey.pem --cert ./fullchain.pem

Another step is to upload a file. This file can then be used by being mounted on your container.

kubectl create secret generic secretfile --from-file=key.json=./secret_json.yaml

Then you can mount it to the pod

    spec:
      volumes:
      - name: secret-json
        secret:
          secretName: secretfile
      containers:
      - name: containername
        volumeMounts:
        - name: secret-json
          mountPath: /var/secrets/json

That’s all! The full docs can be found here.

Migrate your elastic beanstalk workers to docker containers

Amazon Elastic Beanstalk is one of the most popular services that aws provides. Elastic beanstalk comes with two options, the worker and the web application.

The worker application consumes the messages from a sqs queue and process them. If the process was successful the message is removed from the queue, if not the message shall remain in the queue or after some failed attempts it will go back to a dead letter queue. If you want to get more into elb I have made a tutorial on deploying you spring application using elb and cloudformation.

Elastic beanstalk workers are really great because they are managed, they can be scaled up/down depending on your workloads and they provide a wide variety of development environments like java, node.js and also you can use docker images.

Although elastic bean can work wonders if you have an aws based infrastructure you might face some issues when you will try to move to a container based infrastructure using a container orchestration engine.

Most probably your containerized worker application will work seamlessly without any extra configuration, however you need to find an alternative for the agent which dispatches the queue messages to your application.

In order to make things simple I implemented a mechanism which retrieves the messages from the queue and sends them to the worker application.

The container-queue-worker projects aims to provide an easy way to migrate your elastic beanstalk workers to a docker orchestration system.

Since the solution is Scala based it can either be used as a standalone jvm application or it can be run in a container using the image from dockerhub.

Once set up what you need to add the routing configurations.

This can be done using environmental variables

WORKER_TYPE=sqs
WORKER_SERVER_ENDPOINT=http://{docker-service}
WORKER_AWS_QUEUE_ENDPOINT=http://{amazon queue endpoint}

Or if you use it as a container you can add a config file on the /etc/worker/worker.conf path.

worker {
  type =  sqs
  server-endpoint = http://{docker-service}
  aws {
    queue-endpoint =  http://{amazon queue endpoint}
  }
}

In order to make thing easier for you I added a docker compose file simulating the desired outcome.

version: '3.5'
networks:
  queue-worker-network:
    name: queue-worker-network
services:
  worker-server:
    build:
      context: ./worker-server
      dockerfile: Dockerfile
    ports:
      - 8080:8080
    networks:
      - queue-worker-network
  elasticmq:
    build:
      context: ./elasticmq
      dockerfile: Dockerfile
    ports:
      - 9324:9324
    networks:
      - queue-worker-network
  container-queue-worker:
    image: gkatzioura/container-queue-worker:0.1
    depends_on:
      - elasticmq
      - worker-server
    environment:
      WORKER_TYPE: sqs
      WORKER_SERVER_ENDPOINT: http://worker-server:8080/
      WORKER_AWS_QUEUE_ENDPOINT: http://elasticmq:9324/queue/test-queue
      AWS_DEFAULT_REGION: eu-west-1
      AWS_ACCESS_KEY_ID: access-key
      AWS_SECRET_ACCESS_KEY: secret-key
    networks:
      - queue-worker-network

Secure a docker registry using ssl

As mentioned on a previous article having a registry with a username and password is not secure if the registry is not ssl configured.

DockerImage

So we are going to add the ssl certificates to our registry. To make things easier we will use let’s encrypt which is free.

Once we have generated the credentials we have to add them to the registry. We will create a directory called certificates which will contain the certificate pem file and the key pem file. Then we will move the generated certificates on the certificates directory with the names crt.pem and key.crt.

We will follow exactly the same steps we followed in the previous article to generate the password.

docker run --entrypoint htpasswd registry:2 -Bbn {your-user} {your-password} > auth/password-file

Now we are ready to create our registry by also specifying the certificates. To do so we will mount the certificates directory to our docker container. The we will specify where the registry is going to find the credentials on the containers filesystem

docker run -d -p 5000:5000 --restart=always --name registry -v `pwd`/auth:/auth -v `pwd`/certificates:/certificates -e "REGISTRY_AUTH=htpasswd" -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/password-file -e REGISTRY_HTTP_TLS_CERTIFICATE=/certificates/crt.pem -e REGISTRY_HTTP_TLS_KEY=/certificates/key.pem registry:2

So your registry will pickup the credentials specified and will also use the certificates created.
Next step is to do the dns mapping and add a dns entry which directs your subdomain to your registry’s ip.

However if you just wan’t to test it, you can run your registry locally and just change your /etc/hosts and add this entry.

127.0.0.1 registry.{your certificate's domain }

Once you navigate through your browser to https://registry.{your certificate’s domain }:5000
you will get a 200 status code and your browser will identify your connection as secure.

Docker basics: Docker Registry

By default when using docker you pull the images from the Dockerhub docker registry. Most probably you have your own docker images for you application and you want to distribute them and do so in a secure way. One way to do so is to go with the already set options such as a paid plan from Dockerhub or the registries provided by cloud providers like amazon, azure etc.

The other option is setting up your own docker registry.
In any case since you use docker you need to have a registry to distribute your images so that they can make it into production.
There are many benefits on managing your own registry but be aware that it requires effort on your side on provisioning and maintaining it.
Therefore we will create our docker registry

docker run -d -p 5000:5000 --restart=always --name registry registry:2

So we have a docker registry running on port 5000 and the registry will always restart.

Now let’s test our registry and push an image.
First I will build a simple image with no specific purpose.

FROM ubuntu
ENTRYPOINT top

It is just a dummy image printing top.

so we are gonna build it

docker build --tag top-ubuntu:1.0 .

The key is to tag your image based on the domain under which your registry runs.
Currently our registry runs on the localhost therefore by tagging we also specify the location of the registry.

docker tag top-ubuntu:1.0 localhost:5000/top-ubuntu:1.0

And no we push our image

docker push localhost:5000/top-ubuntu:1.0

Now let’s remove our images and see if our image will be downloaded from our running registry

docker rmi top-ubuntu:1.0
docker rmi localhost:5000/top-ubuntu:1.0

And let’s pull

docker pull localhost:5000/top-ubuntu:1.0

As you can see our image has been downloaded from our local registry and is ready to be used.

So far so good. The next step is securing our registry with a username and password.

Let’s start by setting the username and password

First let’s create a directory which shall contain our credentials

mkdir auth

The we shall creae

docker run --entrypoint htpasswd registry:2 -Bbn {your-user} {your-password} > auth/password-file

The file shall contain your username and password information. The password shall be hashed.

Now let’s run our secured registry

 docker run -d -p 5000:5000 --restart=always --name registry -v `pwd`/auth:/auth -e "REGISTRY_AUTH=htpasswd" -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/password-file registry:2

As you can see we mounted the credentials file to the docker container and we specified the location of the password-file.

Let’s try to push our image

docker push localhost:5000/top-ubuntu:1.0
.
.
.

059ad60bcacf: Preparing 
8db5f072feec: Preparing 
67885e448177: Preparing 
ec75999a0cb1: Preparing 
65bdd50ee76a: Preparing 
no basic auth credentials

It’s time to login to our registry

docker login localhost:5000

Once your have provided your credentials you will be able to push the image to your local repository.

docker push localhost:5000/top-ubuntu:1.0

Be aware that our registry is not secure. Having your registry secured with credentials does not make it secure since you need to have ssl encryption.

On the next tutorial we will secure a docker registry with ssl.