In this article, you will learn how to renew certificates in your Spring Boot apps on Kubernetes with cert-manager and Stakater Reloader. We are going to run two simple Spring Boot apps that communicate with each other over SSL. The TLS cert used in that communication will be automatically generated by Cert Manager. With Cert Manager we can easily rotate certs after a certain time. In order to automatically use the latest TLS certs we need to restart our apps. We can achieve it with Stakater Reloader.
Before we start, it is worth reading the following article. It shows how to use cert-manager together with Istio to create secure gateways on Kubernetes.
Source Code
If you would like to try this exercise yourself, you may always take a look at my source code. In order to do that, you need to clone this GitHub repository. Then switch to the ssl
directory. You will find two Spring Boot apps: secure-callme
and secure-caller
. After that, you should just follow my instructions. Let’s begin.
How it works
Before we go into the technical details, let me write a little bit more about the architecture of our solution. Our challenge is pretty common. We need secure SSL/TLS communication between the services running on Kubernetes. Instead of manually generating and replacing certs inside the apps, we need an automatic approach.
Here come the cert-manager and the Stakater Reloader. Cert Manager is able to generate certificates automatically, based on the provided CRD object. It also ensures the certificates are valid and up-to-date and will attempt to renew certificates before expiration. It puts all the required inside Kubernetes Secret
. On the other hand, Stakater Reloader is able to watch if any changes happen in ConfigMap
or Secret
. Then it performs a rolling upgrade on pods, which use the particular ConfigMap
or Secret
. Here is the visualization of the described architecture.
Prerequisites
Of course, you need to have a Kubernetes cluster. In this exercise, I’m using Kubernetes on Docker Desktop. But you can as well use any other local distribution like minikube, kind, or a cloud-hosted instance. No matter which distribution you choose you also need to have:
- Skaffold (optionally) – a CLI tool to simplify deploying the Spring Boot app on Kubernetes and applying all the manifests in that exercise using a single command. You can find installation instructions here
- Helm – used to install additional tools on Kubernetes like Stakater Reloader or cert-manager
Install Cert Manager and Stakater Reloader
In order to install both cert-manager and Reloader on Kubernetes we will use Helm charts. We don’t need any specific settings just defaults. Let’s begin with the cert-manager. Before installing the chart we have to add CRD resources for the latest version 1.10.1
:
$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.10.1/cert-manager.crds.yaml
Then, we need to add the jetstack
chart repository:
$ helm repo add jetstack https://charts.jetstack.io
After that, we can install the chart using the following command:
$ helm install my-release cert-manager jetstack/cert-manager
The same with the Stakater Reloader – first we need to add the stakater
charts repository:
$ helm repo add stakater https://stakater.github.io/stakater-charts
Then, we can install the latest version of the chart:
$ helm install my-reloader stakater/reloader
In order to verify that the installation finished successfully we can display a list of running pods:
$ kubectl get po
NAME READY STATUS RESTARTS AGE
my-cert-manager-578884c6cf-f9ppt 1/1 Running 0 1m
my-cert-manager-cainjector-55d4cd4bb6-6mgjd 1/1 Running 0 1m
my-cert-manager-webhook-5c68bf9c8d-nz7sd 1/1 Running 0 1m
my-reloader-reloader-7566fdc68c-qj9l4 1/1 Running 0 1m
That’s all. Now we can proceed to the implementation.
HTTPS with Spring Boot
Our first app secure-callme
exposes a single endpoint GET /callme
over HTTP. That endpoint will be called by the secure-caller
app. Here’s the @RestController
implementation:
@RestController
public class SecureCallmeController {
@GetMapping("/callme")
public String call() {
return "I'm `secure-callme`!";
}
}
Now our goal is to enable HTTPS for that app, and of course, make it work properly on Kubernetes. First, we should change the default server port for the Spring Boot app to 8443. Then we have to enable SSL and provide locations of key stores. Additionally, we will force verification of the client’s certificate with the server.ssl.client-auth
property. Here’s the configuration for our Spring Boot inside the application.yml
file.
server.port: 8443
server.ssl:
enabled: true
key-store: ${CERT_PATH}/keystore.jks
key-store-password: ${PASSWORD}
trust-store: ${CERT_PATH}/truststore.jks
trust-store-password: ${PASSWORD}
client-auth: NEED
We will set the values of CERT_PATH
and PASSWORD
at the level of Kubernetes Deployment
. Now, let’s switch to the secure-caller
implementation. We have to configure SSL on the REST client side. Since we use Spring RestTemplate
for calling services, we need to add customize its default behavior. Firstly, let’s include the Apache HttpClient
dependency.
<dependency>
<groupId>org.apache.httpcomponents.client5</groupId>
<artifactId>httpclient5</artifactId>
</dependency>
Now, we will use Apache HttpClient
as a low-level client for the Spring RestTemplate
. We need to define a key store and trust store for the client since a server-side requires and verifies client cert. In order to create RestTempate
@Bean
we use RestTemplateBuilder
.
@SpringBootApplication
public class SecureCaller {
public static void main(String[] args) {
SpringApplication.run(SecureCaller.class, args);
}
@Autowired
ClientSSLProperties clientSSLProperties;
@Bean
RestTemplate builder(RestTemplateBuilder builder) throws
GeneralSecurityException, IOException {
final SSLContext sslContext = new SSLContextBuilder()
.loadTrustMaterial(
new File(clientSSLProperties.getTrustStore()),
clientSSLProperties.getTrustStorePassword().toCharArray())
.loadKeyMaterial(
new File(clientSSLProperties.getKeyStore()),
clientSSLProperties.getKeyStorePassword().toCharArray(),
clientSSLProperties.getKeyStorePassword().toCharArray()
)
.build();
final SSLConnectionSocketFactory sslSocketFactory =
SSLConnectionSocketFactoryBuilder.create()
.setSslContext(sslContext)
.build();
final HttpClientConnectionManager cm =
PoolingHttpClientConnectionManagerBuilder.create()
.setSSLSocketFactory(sslSocketFactory)
.build();
final HttpClient httpClient = HttpClients.custom()
.setConnectionManager(cm)
.evictExpiredConnections()
.build();
return builder
.requestFactory(() ->
new HttpComponentsClientHttpRequestFactory(httpClient))
.build();
}
}
The client credentials are taken from configuration settings under the client.ssl
key. Here is the @ConfigurationProperties
class used the RestTemplateBuilder
in the previous step.
@Configuration
@ConfigurationProperties("client.ssl")
public class ClientSSLProperties {
private String keyStore;
private String keyStorePassword;
private String trustStore;
private String trustStorePassword;
// GETTERS AND SETTERS ...
}
Here’s the configuration for the secure-caller
inside application.yml
file. The same as for the secure-callme
we expose the REST endpoint over HTTPS.
server.port: 8443
server.ssl:
enabled: true
key-store: ${CERT_PATH}/keystore.jks
key-store-password: ${PASSWORD}
trust-store: ${CERT_PATH}/truststore.jks
trust-store-password: ${PASSWORD}
client-auth: NEED
client.url: https://${HOST}:8443/callme
client.ssl:
key-store: ${CLIENT_CERT_PATH}/keystore.jks
key-store-password: ${PASSWORD}
trust-store: ${CLIENT_CERT_PATH}/truststore.jks
trust-store-password: ${PASSWORD}
The secure-caller
app calls GET /callme
exposed by the secure-callme
app using customized RestTemplate
.
@RestController
public class SecureCallerController {
RestTemplate restTemplate;
@Value("${client.url}")
String clientUrl;
public SecureCallerController(RestTemplate restTemplate) {
this.restTemplate = restTemplate;
}
@GetMapping("/caller")
public String call() {
return "I'm `secure-caller`! calling... " +
restTemplate.getForObject(clientUrl, String.class);
}
}
Generate and Renew Certificates on Kubernetes with Cert Manager
With cert-manager, we can automatically generate and renew certificates on Kubernetes. Of course, we could generate TLS/SSL certs using e.g. openssl
as well and then apply them on Kubernetes. However, Cert Manager simplifies that process. It allows us to declaratively define the rules for the certs generation process. Let’s see how it works. Firstly, we need the issuer object. We can create a global issuer for the whole as shown. It uses the simplest option – self-signed
.
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: ss-clusterissuer
spec:
selfSigned: {}
After that, we can generate certificates. Here’s the cert-manager Certificate
object for the secure-callme
app. There are some important things here. First of all, we can generate key stores together with a certificate and private key (1). The object refers to the ClusterIssuer
created in the previous step (2). The name of Kubernetes Service used during communication is secure-callme
, so the cert needs to have that name as CN. In order to enable certificate rotation we need to set validity time. The lowest possible value is 1 hour (4). So each time 5 minutes before expiration cert-manager will automatically renew a certificate (5). It won’t rotate the private key. In order to enable it we should set the parameter rotationPolicy
to Always
.
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: secure-callme-cert
spec:
keystores: # (1)
jks:
passwordSecretRef:
name: jks-password-secret
key: password
create: true
issuerRef: # (2)
name: ss-clusterissuer
group: cert-manager.io
kind: ClusterIssuer
privateKey:
algorithm: ECDSA
size: 256
dnsNames:
- secure-callme
secretName: secure-callme-cert
commonName: secure-callme # (3)
duration: 1h # (4)
renewBefore: 5m # (5)
The Certificate
object for secure-caller
is very similar. The only difference is in the CN
field. We will use the port-forward
option during the test, so I’ll set the domain name to localhost
(1).
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: secure-caller-cert
spec:
keystores:
jks:
passwordSecretRef:
name: jks-password-secret
key: password
create: true
issuerRef:
name: ss-clusterissuer
group: cert-manager.io
kind: ClusterIssuer
privateKey:
algorithm: ECDSA
size: 256
dnsNames:
- localhost
- secure-caller
secretName: secure-caller-cert
commonName: localhost # (1)
duration: 1h
renewBefore: 5m
After applying both manifests we can display a list of Certificates
. Each of them is related to the Secret
with the same name:
$ kubectl get certificate
NAME READY SECRET AGE
secure-caller-cert True secure-caller-cert 1m
secure-callme-cert True secure-callme-cert 1m
Here are the details of the secure-callme-cert
Secret
. It contains the key store and trust store in the JKS format. We will use both of them in the Spring Boot SSL configuration (server.ssl.trust-store
and server.ssl.key-store
properties). There is also a certificate (tls.crt
), a private key (tls.key
), and CA (ca.crt
).
$ kubectl describe secret secure-callme-cert
Name: secure-callme-cert
Namespace: default
Labels: <none>
Annotations: cert-manager.io/alt-names: secure-callme
cert-manager.io/certificate-name: secure-callme-cert
cert-manager.io/common-name: secure-callme
cert-manager.io/ip-sans:
cert-manager.io/issuer-group: cert-manager.io
cert-manager.io/issuer-kind: ClusterIssuer
cert-manager.io/issuer-name: ss-clusterissuer
cert-manager.io/uri-sans:
Type: kubernetes.io/tls
Data
====
ca.crt: 550 bytes
keystore.jks: 1029 bytes
tls.crt: 550 bytes
tls.key: 227 bytes
truststore.jks: 422 bytes
Deploy and Reload Apps on Kubernetes
Since we have already prepared all the required components and objects, we may proceed with the deployment of our apps. As I mentioned in the “Prerequisites” section we will use Skaffold for building and deploying apps on the local cluster. Let’s begin with the secure-callme
app.
First of all, we need to reload the app each time the secure-callme-cert
changes. It occurs once per hour when the cert-manager renews the TLS certificate. In order to enable the automatic restart of the pod with Stakater Reloader we need to annotate the Deployment with secret.reloader.stakater.com/reload
(1). The annotation should contain the name of Secret
, which triggers the app reload. Of course, we also need to mount key store and trust store files (3) and set the mount path for the Spring Boot app available under the CERT_PATH
env variable (2). We are mounting the whole secure-callme-cert
Secret
.
apiVersion: apps/v1
kind: Deployment
metadata:
name: secure-callme
annotations:
# (1)
secret.reloader.stakater.com/reload: "secure-callme-cert"
spec:
selector:
matchLabels:
app.kubernetes.io/name: secure-callme
template:
metadata:
labels:
app.kubernetes.io/name: secure-callme
spec:
containers:
- image: piomin/secure-callme
name: secure-callme
ports:
- containerPort: 8443
name: https
env:
- name: PASSWORD
valueFrom:
secretKeyRef:
key: password
name: jks-password-secret
- name: CERT_PATH # (2)
value: /opt/secret
volumeMounts:
- mountPath: /opt/secret # (3)
name: cert
volumes:
- name: cert
secret:
secretName: secure-callme-cert # (4)
The password to the key store files is available inside the jks-password-secret
:
kind: Secret
apiVersion: v1
metadata:
name: jks-password-secret
data:
password: MTIzNDU2
type: Opaque
There is also the Kubernetes Service
related to the app:
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/name: secure-callme
name: secure-callme
spec:
ports:
- name: https
port: 8443
targetPort: 8443
selector:
app.kubernetes.io/name: secure-callme
type: ClusterIP
Now, go to the secure-callme
directory and just run the following command:
$ skaffold run
The Deployment
manifest of the secure-caller
app is a little bit more complicated. The same as before we need to reload the app on Secret
change (1). However, this app uses two secrets. The first of them contains server certs (secure-caller-cert), while the second contains certs for communication with secure-callme
. Consequently, we are mounting two secrets (5) and we are setting the path with server key stores (2) and client key stores (3).
apiVersion: apps/v1
kind: Deployment
metadata:
name: secure-caller
# (1)
annotations:
secret.reloader.stakater.com/reload: "secure-caller-cert,secure-callme-cert"
spec:
selector:
matchLabels:
app.kubernetes.io/name: secure-caller
template:
metadata:
labels:
app.kubernetes.io/name: secure-caller
spec:
containers:
- image: piomin/secure-caller
name: secure-caller
ports:
- containerPort: 8443
name: https
env:
- name: PASSWORD
valueFrom:
secretKeyRef:
key: password
name: jks-password-secret
# (2)
- name: CERT_PATH
value: /opt/secret
# (3)
- name: CLIENT_CERT_PATH
value: /opt/client-secret
- name: HOST
value: secure-callme
volumeMounts:
- mountPath: /opt/secret
name: cert
- mountPath: /opt/client-secret
name: client-cert
volumes:
# (5)
- name: cert
secret:
secretName: secure-caller-cert
- name: client-cert
secret:
secretName: secure-callme-cert
Then, go to the secure-caller
directory and deploy the app. This time we enable port-forward
to easily test the app locally.
$ skaffold run --port-forward
Let’s display a final list of all running apps. We have cert-manager components, the Stakater reloader, and our two sample Spring Boot apps.
$ kubectl get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
my-cert-manager 1/1 1 1 1h
my-cert-manager-cainjector 1/1 1 1 1h
my-cert-manager-webhook 1/1 1 1 1h
my-reloader-reloader 1/1 1 1 1h
secure-caller 1/1 1 1 1h
secure-callme 1/1 1 1 1h
Testing Renew of Certificates on Kubernetes
The secure-caller
app is available on the 8443 port locally, while the secure-callme
app is available inside the cluster under the service name secure-callme
. Let’s make a test call. Firstly, we need to download certificates and private keys stored on Kubernetes:
$ kubectl get secret secure-caller-cert \
-o jsonpath \
--template '{.data.tls\.key}' | base64 --decode > tls.key
$ kubectl get secret secure-caller-cert \
-o jsonpath \
--template '{.data.tls\.crt}' | base64 --decode > tls.crt
$ kubectl get secret secure-caller-cert \
-o jsonpath \
--template '{.data.ca\.crt}' | base64 --decode > ca.crt
Now, we can call the GET /caller
endpoint using the following curl command. Under the hood, the secure-caller
calls the endpoint GET /callme
exposed by the secure-callme
also over HTTPS. If you did everything according to the instruction you should have the same result as below.
$ curl https://localhost:8443/caller \
--key tls.key \
--cert tls.crt \
--cacert ca.crt
I'm `secure-caller`! calling... I'm `secure-callme`!
Our certificate is valid for one hour.
Let’s see what happens after one hour. A new certificate has already been generated and both our apps have been reloaded. Now, if try to call the same endpoint as before using old certificates you should see the following error.
Now, if you repeat the first step in that section it should work properly. We just need to download the certs to make a test. The internal communication over SSL works automatically after a reload of apps.
Final Thoughts
Of course, there are some other ways for achieving the same result as in our exercise. For example, you can a service mesh tool like Istio and enable mutual TLS for internal communication. You will still need to handle the automatic renewal of certificates somehow in that scenario. Cert Manager may be replaced with some other tools like HashiCorp Vault, which provides features for generating SSL/TLS certificates. You can as well use Spring Cloud Kubernetes with Spring Boot for watching for changes in secrets and reloading them without restarting the app. However, the solution used to renew certificates on Kubernetes presented in that article is simple and will work for any type of app.