Containerising and Deploying with k8s-Logical Progression

This started less an article or blog, more a journal of my many failed attempts at containerising and deploying a small app. I was keen to practice with the technologies I use every day, but without the guardrails of a fantastic platform team with all the tools and support they provide.

The following describes the (eventual!) successful steps to…

Package a Kotlin web application with environment secrets in a Docker image
Upload the image to a container registry, and run both manually and as part of a CI build pipeline
Create a Kubernetes (k8s) cluster and deploy the app, both manually and as part of the CI build pipeline
Expose the application to the internet

The code for the Tempo-Valence application used can be seen here

1. Package app in a docker image

Google Kubernetes Engine (GKE) accepts Docker images as the application deployment format.

Solution here uses gcr.io (Google) container registry, as this was the most straightforward to connect to and authenticate with.

Initial steps to get started…

Enable Cloud Registry API in Google Cloud Platform (GCP) console
Set up a new Service Account in GCP Service Accounts - IAM & Admin section so that this can be used to authenticate and access the Google container registry (gcr). This service account will need to have a Storage Admin role
Download key from this GCP service account and add it to a gitlab-ci variable - called GOOGLE_CREDENTIALS in my example, which is accessed in the gitlab-ci.yml file below
(The key can also be saved to a local, non-committed keyfile somewhere if you wish to run the jib docker image locally to check)
Update build step of gitlab-ci.yml to…

build:
    <<: *gradle-image
    stage: build
    variables:
        CLIENT_KEY: $CLIENT_KEY
        GOOGLE_CREDENTIALS: $GOOGLE_CREDENTIALS
    before_script:
        - echo "$GOOGLE_CREDENTIALS" > keyfile.json
    script:
        - export CLIENT_KEY="$CLIENT_KEY"
        - gradle jib -Djib.console=plain
        - gradle clean build
    only:
        - master

The jib library was used as support for packaging up a JVM application into a docker image was better than the Docker tools (see also outtakes sections for obstacles with this)

Add jib config in build.gradle.kts file of the project as follows…

jib {
    container {
        mainClass = application.mainClassName
        environment = mapOf(Pair("CLIENT_KEY", System.getenv("CLIENT_KEY")))
        ports = listOf("8000")
    }
    from {
        image = "openjdk:11"
    }
    to {
        image = "gcr.io/inductive-seer-256007/tempo-valence:latest"
        auth {
            username = "_json_key"
            password = file("keyfile.json").readText()
        }
    }
}

(The CLIENT_KEY environment variable above refers to a secret env var required to access an external api from the application. The value of this env var is stored as a Variable in Gitlab CI so the pipeline can access to run the app.)

2. Upload docker image to a container registry

See previous step
To run locally…

docker run --rm -p 9000:9000 gcr.io/inductive-seer-256007/tempo-valence:latest

# in new terminal tab
curl http://localhost:9090
# Welcome to TempoValence!

3. Create a k8s container cluster

The following manual commands create the cluster, deploy the application and expose on an external IP

gcloud auth list

gcloud config set account m.........@gmail.com

gcloud config set project inductive-seer-256007

gcloud config set compute/zone europe-west2-a

gcloud container clusters create tempo-valence-cluster

gcloud compute instances list

kubectl create deployment tempo-valence --image=gcr.io/inductive-seer-256007/tempo-valence:latest

kubectl get deployment

kubectl get pods

kubectl scale deployment tempo-valence --replicas=2

kubectl autoscale deployment tempo-valence --cpu-percent=80 --min=1 --max=5

kubectl expose deployment tempo-valence --name=tempo-valence-service --type=LoadBalancer --port 9000 --target-port 9000
# the ports here matter - think it has something to do with how we've set up the application in the image

kubectl get service

# and use the EXTERNAL_IP of tempo-valence-service to access, e.g...
http://35.234.150.122:9000/

4. Deploy

See step 3 for manual deployment
In order to deploy from the gitlab-ci pipeline, the gitlab-ci.yml looked like this for the deploy stage…

deploy:
stage: deploy
variables:
    CI_SERVICE_ACCOUNT: $CI_SERVICE_ACCOUNT
image:
    name: kiwigrid/gcloud-kubectl-helm
before_script:
    - echo "$CI_SERVICE_ACCOUNT" > key.json
    - gcloud auth activate-service-account --key-file=key.json
    - gcloud config set project inductive-seer-256007
    - gcloud config set container/cluster tempo-valence-cluster
    - gcloud config set compute/zone europe-west2-a
    - gcloud container clusters get-credentials tempo-valence-cluster --zone europe-west2-a
script:
    - kubectl create deployment tempo-valence --image=gcr.io/inductive-seer-256007/tempo-valence:latest
    - kubectl expose deployment tempo-valence --name=tempo-valence-service --type=LoadBalancer --port 9000 --target-port 9000
    # Wait for the deployment to be applied
    - kubectl rollout status deployment tempo-valence --watch

Few things to note here!

Set up another service account in GCP, to be used by gitlab ci to authenticate against both the container registry and k8s cluster (may be possible just to amend the roles of previous service account, but I added a new one to keep it separate)
- Give the service account 2 project roles -> Kubernetes Engine Developer and Storage Object Admin
- Download the key associated to the account and add in to a new CI variable in gitlab
The image is the image used to run the deployment - bit of trial and error here, as this image needed to have both kubectl and gcloud cli tools loaded (and I couldn’t be bothered to create one and store it somewhere that didn’t need authentication!) The image included above is open and does the job.
For this to work, the tempo-valence-cluster of course needs to be up and running

5. Expose to the internet

See step 3 for ip address exposed to internet
I have not yet configured for a static ip and custom domain name, this will be my next step, and I will update here soon

Steps to run

As this cluster and application will not be continually running here are the steps needed before this will be available on the internet

To setup

gcloud container clusters create tempo-valence-cluster

# Run ci pipeline, or...
kubectl create deployment tempo-valence --image=gcr.io/inductive-seer-256007/tempo-valence:latest

kubectl scale deployment tempo-valence --replicas=2

kubectl autoscale deployment tempo-valence --cpu-percent=80 --min=1 --max=5

kubectl expose deployment tempo-valence --name=tempo-valence-service --type=LoadBalancer --port 9000 --target-port 9000

To teardown

kubectl delete service tempo-valence-service

gcloud container clusters delete tempo-valence-cluster

Links and article credits

A lot of ideas from this Google k8s tutorial
And this GCP tutorial
One of our own from Matt Dowds on Kubernetes deployment form Gitlab CI