Monitor Kubernetes service certificates with a CertHalo agent

Services inside a Kubernetes cluster often talk to each other over TLS: internal APIs, admission webhooks, databases, message brokers and internal ingress controllers. Their certificates usually come from cert-manager or an internal CA, and most of them are only reachable inside the cluster. This guide runs a CertHalo agent there as a CronJob, so those certificates are graded and alerted on like your public domains.

For the general setup, including how to upload your internal root CA, see Monitor internal certificates with a CertHalo agent. Internal certificates are on CertHalo's Business plan.

What to monitor (and what not to)

Good candidates:

• Services and internal ingress that serve TLS: my-api.my-namespace.svc.cluster.local:8443.
• Admission and conversion webhooks: when their certificate expires, deployments start failing cluster-wide.
• Stateful services with TLS from the first byte: Redis, Kafka, RabbitMQ, LDAP over TLS.
• cert-manager issued certificates: cert-manager renews them, but renewals can fail silently (a broken issuer, a rate limit, a DNS-01 problem), and a certificate can renew without the pod reloading it. Watching what the service actually serves catches both.

Leave out service mesh workload certificates (Istio, Linkerd): the mesh rotates them every few hours and they're only presented to clients with mesh identities, so monitoring each one adds noise, not safety. Monitor the mesh's own root and intermediate CAs instead, by uploading them on the Developers page and watching the services that terminate TLS for clients.

Step 1: Store the key

Create an Agent key on CertHalo's Developers page, then store it in the cluster:

kubectl create namespace certhalo
kubectl -n certhalo create secret generic certhalo-agent --from-literal=api-key=chk_...

Step 2: List the services

Put the targets in a ConfigMap, using in-cluster DNS names, the same names your services use to reach each other:

apiVersion: v1
kind: ConfigMap
metadata:
  name: certhalo-agent-targets
  namespace: certhalo
data:
  targets.txt: |
    payments-api.payments.svc.cluster.local:8443
    webhook.cert-manager.svc.cluster.local:443
    kafka.messaging.svc.cluster.local:9093
    internal-ingress.ingress-nginx.svc.cluster.local:443

CertHalo checks that each certificate covers the name you list, so use the name clients use. If a certificate only covers the short form (payments-api.payments.svc), list that instead.

Step 3: Run the agent as a CronJob

apiVersion: batch/v1
kind: CronJob
metadata:
  name: certhalo-agent
  namespace: certhalo
spec:
  schedule: "0 */6 * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      backoffLimit: 1
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: agent
              image: ghcr.io/incubxperts/certhalo-agent:1
              args: ["--interval", "21600"]   # report once and exit; matches the schedule
              env:
                - name: CERTHALO_API_KEY
                  valueFrom: { secretKeyRef: { name: certhalo-agent, key: api-key } }
                - name: CERTHALO_AGENT_NAME
                  value: prod-cluster
              volumeMounts:
                - { name: targets, mountPath: /config, readOnly: true }
              securityContext: { runAsNonRoot: true, readOnlyRootFilesystem: true, allowPrivilegeEscalation: false }
          volumes:
            - name: targets
              configMap: { name: certhalo-agent-targets }

--interval tells CertHalo to expect a report every 6 hours, matching the schedule. If none arrives for 12 hours, for example because the CronJob was suspended or its image can't be pulled, you get a "no report" alert.

Give each cluster its own CERTHALO_AGENT_NAME so the dashboard shows where each certificate was seen.

Step 4: Allow the traffic

If the cluster uses network policies, the agent's pod needs:

• Egress to each service on the listed ports.
• Egress to certhalo.com on port 443, through your egress gateway or proxy if you have one (set HTTPS_PROXY in the container's env).
• DNS to resolve service names.

Step 5: Check it

Trigger a run without waiting for the schedule, and read its log:

kubectl -n certhalo create job --from=cronjob/certhalo-agent certhalo-agent-now
kubectl -n certhalo logs job/certhalo-agent-now

It prints how many hosts were accepted. The services then appear on the CertHalo dashboard with an Internal badge and the cluster's name. Any that were rejected are listed in the log with the reason; the troubleshooting table covers the usual ones.

Keeping the list current

Services come and go. Two simple habits keep the list honest:

• Keep the ConfigMap in the same repository as your manifests, so adding a service and monitoring it happen in one change.
• Review the dashboard's internal domains when you decommission something, and remove it there, or it will raise "can't reach host" alerts.