HTTPS on Kubernetes with Spring Boot, Istio and Cert Manager

Blue Team Code Fortification Hacking 101
by TechnoHerder : Leave a comment

In this article, you will learn how to create secure HTTPS gateways on Kubernetes. We will use Cert Manager to generate TLS/SSL certificates. With Istio we can create secure HTTPS gateways and expose them outside a Kubernetes cluster. Our test application is built on top of Spring Boot. We will consider two different ways of securing that app. In the first of them, we are going to set mutual TLS on the gateway and a plain port on the app side. Then, we run a scenario with a secure 2-way SSL configured on the Spring Boot app. Let’s see how it looks.

There are several topics you should be familiar with before start reading the following article. At least, it is worth reading about the basics related to service mesh on Kubernetes with Istio and Spring Boot here. You may also be interested in the article about security best practices for Spring Boot apps.

This tutorial guides you on how to try the exercise on OpenShift with OpenShift Service Mesh (the implementation of Istio). However, you can as well run it on the vanilla Kubernetes cluster like minikube or kind.

Prerequisites

Before we begin we need to prepare a test cluster. Personally, I’m using a local OpenShift cluster that simplifies the installation of several tools we need. But you can as well use any other Kubernetes distribution like minikube or kind. No matter which version you choose, you need to install at least:

  1. Istio – you can do it in several ways, here’s an instruction with Istio CLI
  2. Cert Manager – the simplest way to install it with the kubectl
  3. Skaffold (optionally) – a CLI tool to simplify deploying the Spring Boot app on Kubernetes and applying all the manifests in that exercise using a single command. You can find installation instructions here
  4. Helm – used to install additional tools on Kubernetes (or OpenShift)

Since I’m using OpenShift I can install everything using the UI console and Operator Hub. Here’s the list of my tools:

Source Code

If you would like to try this exercise yourself, you may always take a look at my source code. In order to do that, you need to clone the GitHub repository. Then switch to the tls branch. After that, you should just follow my instructions. Let’s begin.

Generate Certificates with Cert Manager

You can as well generate TLS/SSL certs using e.g. openssl and then apply them on Kubernetes. However, Cert Manager simplifies that process. It automatically creates a Kubernetes Secret with all the required staff. Before we create a certificate we should configure a default ClusterIssuer. I’m using the simplest option – self-signed.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-cluster-issuer
spec:
  selfSigned: {}

In the next step, we are going to create the Certificate object. It is important to set the commonName parameter properly. It should be the same as the hostname of our gateway. Since I’m running the local instance of OpenShift the default domain suffix is apps-crc.testing. The name of the application is sample-spring-kotlin in that case. We will also need to generate a keystore and truststore for configuring 2-way SSL in the second scenario with the passthrough gateway. Finally, we should have the same Kubernetes Secret with a certificate placed in the namespace with the app (demo-apps) and the namespace with Istio workloads (istio-system in our case). We can sync secrets across namespaces in several different ways. I’ll use the built-in feature of Cert Manager based on the secretTemplate field and a tool called reflector. Here’s the final version of the Certificate object:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: sample-spring-boot-cert
  namespace: demo-apps
spec:
  keystores:
    jks:
      passwordSecretRef:
        name: jks-password-secret
        key: password
      create: true
  issuerRef:
    name: selfsigned-cluster-issuer
    group: cert-manager.io
    kind: ClusterIssuer
  privateKey:
    algorithm: ECDSA
    size: 256
  dnsNames:
    - sample-spring-kotlin.apps-crc.testing
  secretName: sample-spring-boot-cert
  commonName: sample-spring-kotlin.apps-crc.testing
  secretTemplate:
    annotations:
      reflector.v1.k8s.emberstack.com/reflection-allowed: "true"  
      reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces: "istio-system"
      reflector.v1.k8s.emberstack.com/reflection-auto-enabled: "true"
      reflector.v1.k8s.emberstack.com/reflection-auto-namespaces: "istio-system"

Before we apply the Certificate object we need to create the Secret with a keystore password:

kind: Secret
apiVersion: v1
metadata:
  name: jks-password-secret
  namespace: demo-apps
data:
  password: MTIzNDU2
type: Opaque

Of course, we also need to install the reflector tool on our Kubernetes cluster. We can easily do it using the following Helm commands:

$ helm repo add emberstack https://emberstack.github.io/helm-charts
$ helm repo update
$ helm upgrade --install reflector emberstack/reflector

Here’s the final result. There is the Secret sample-spring-kotlin-cert in the demo-apps namespace. It contains a TLS certificate, private key, CA, JKS keystore, and truststore. You can also verify that the same Secret is available in the istio-system namespace.

Create Istio Gateway with Mutual TLS

Let’s begin with our first scenario. In order to create a gateway with mTLS, we should set MUTUAL as a mode and set the name of the Secret containing the certificate and private key. The name of the Istio Gateway host is sample-spring-kotlin.apps-crc.testing. Gateway is available on Kubernetes under the default HTTPS port.

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: microservices-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 443
        name: https
        protocol: HTTPS
      hosts:
        - sample-spring-kotlin.apps-crc.testing
      tls:
        mode: MUTUAL
        credentialName: sample-spring-boot-cert

The Spring Boot application is available under the HTTP port. Therefore we should create a standard Istio VirtualService that refers to the already created gateway.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: sample-spring-boot-vs-via-gw
spec:
  hosts:
    - sample-spring-kotlin.apps-crc.testing
  gateways:
    - microservices-gateway
  http:
    - route:
        - destination:
            host: sample-spring-kotlin-microservice
            port:
              number: 8080

Finally, we can run the Spring Boot app on Kubernetes using skaffold. We need to activate the istio-mutual profile that exposes the HTTP (8080) port.

$ skaffold dev -p istio-mutual

OpenShift Service Mesh automatically exposes Istio Gateway as the Route object. Let’s just verify if everything works fine. Assuming that the app has started successfully we can display a list of Istio gateways in the demo-apps namespace:

$ kubectl get gw -n demo-apps
NAME                    AGE
microservices-gateway   3m4s

Then let’s display a list of Istio virtual services in the same namespace:

$ kubectl get vs -n demo-apps
NAME                           GATEWAYS                    HOSTS                                       AGE
sample-spring-boot-vs-via-gw   ["microservices-gateway"]   ["sample-spring-kotlin.apps-crc.testing"]   4m2s

If you are running it on OpenShift you should also check the Route object in the istio-system namespace:

$ oc get route -n istio-system | grep sample-spring-kotlin

If you test it on Kubernetes you just need to set the Host header on your request. Here’s the curl command for testing our secure gateway. Since we enabled mutual TLS auth we need to provide the client key and certificate. We can copy them to the local machine from Kubernetes Secret generated by the Cert Manager.

Let’s call the REST endpoint GET /persons exposed by our sample Spring Boot app:

$ curl -v https://sample-spring-kotlin.apps-crc.testing/persons \
    --key tls.key \
    --cert tls.crt

You will probably receive the following response:

Ok, we forgot to add our CA to the trusted certificates. Let’s do that. Alternatively, we can set the parameter --cacert on the curl command.

Now, we can run again the same curl command as before. The secure communication with our app should work perfectly fine. We can proceed to the second scenario.

Mutual Auth for Spring Boot and Passthrough Istio Gateway

Let’s proceed to the second scenario. Now, our sample Spring Boot is exposing the HTTPS port with the client cert verification. In order to enable it on the app side, we need to provide some configuration settings.

Here’s our application.yml file. Firstly, we need to enable SSL and set the default port to 8443. It is important to force client certificate authentication with the server.ssl.client-auth property. As a result, we also need to provide a truststore file location. Finally, we don’t want to expose Spring Boot Actuator endpoint over SSL, so we force to expose them under the default plain port.

server.port: 8443
server.ssl.enabled: true
server.ssl.key-store: /opt/secret/keystore.jks
server.ssl.key-store-password: ${PASSWORD}

server.ssl.trust-store: /opt/secret/truststore.jks
server.ssl.trust-store-password: ${PASSWORD}
server.ssl.client-auth: NEED

management.server.port: 8080
management.server.ssl.enabled: false

In the next step, we need to inject both keystore.jks and truststore.jks into the app container. Therefore we need to modify Deployment to mount a secret generated by the Cert Manager as a volume. Once again, the name of that Secret is sample-spring-boot-cert. The content of that Secret would be available for the app under the /opt/secret directory. Of course, we also need to expose the port 8443 outside and inject a secret with the keystone and truststore password.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-spring-kotlin-microservice
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: sample-spring-kotlin-microservice
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "true"
      labels:
        app.kubernetes.io/name: sample-spring-kotlin-microservice
    spec:
      containers:
      - image: piomin/sample-spring-kotlin
        name: sample-spring-kotlin-microservice
        ports:
        - containerPort: 8080
          name: http
        - containerPort: 8443
          name: https
        env:
          - name: PASSWORD
            valueFrom:
              secretKeyRef:
                key: password
                name: jks-password-secret
        volumeMounts:
          - mountPath: /opt/secret
            name: sample-spring-boot-cert
      volumes:
        - name: sample-spring-boot-cert
          secret:
            secretName: sample-spring-boot-cert

Here’s the definition of our Istio Gateway:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: microservices-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 443
        name: https
        protocol: HTTPS
      hosts:
        - sample-spring-kotlin.apps-crc.testing
      tls:
        mode: PASSTHROUGH

The definition of the Istio VirtualService is slightly different than before. It contains the TLS route with the name of the host to match SNI.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: sample-spring-boot-vs-via-gw
spec:
  hosts:
    - sample-spring-kotlin.apps-crc.testing
  gateways:
    - microservices-gateway
  tls:
    - route:
        - destination:
            host: sample-spring-kotlin-microservice
            port:
              number: 8443
          weight: 100
      match:
        - port: 443
          sniHosts:
            - sample-spring-kotlin.apps-crc.testing

Finally, we can deploy the current version of the app using the following skaffold command to enable HTTPS port for the app running on Kubernetes:

$ skaffold dev -p istio-passthrough

Now, you can repeat the same steps as in the previous section to verify that the current configuration works fine.

Final Thoughts

In this article, I showed how to use some interesting tools that simplify the configuration of HTTPS for apps running on Kubernetes. You could see how Istio, Cert Manager, or reflector can work together. We considered two variants of making secure HTTPS Istio gateways for the Spring Boot application.

Leave a Reply

Your email address will not be published. Required fields are marked *