Multiple Node-Agent DaemonSets per Node Pool

Overview

In Kubernetes clusters with heterogeneous node pools (different CPU/memory sizes), it is often necessary to run the node-agent with different resource requests/limits and scheduling constraints per node pool. This ensures optimal resource usage and stability, as each node pool can have a node-agent DaemonSet tailored to its hardware profile.

The Kubescape Operator Helm chart supports this use case via the nodeAgent.multipleDaemonSets feature, allowing you to deploy multiple node-agent DaemonSets, each with its own configuration.

Why Use Multiple DaemonSets?

Resource Optimization: Assign appropriate CPU/memory requests and limits for each node pool, preventing over- or under-provisioning.
Scheduling Control: Use nodeSelector to target specific node pools, ensuring each DaemonSet runs only where intended.
Scalability: Easily add or modify configurations as your cluster grows or changes.

When to Use

Your cluster has multiple node pools with different hardware profiles (e.g., standard, high-memory, GPU).
You need to fine-tune the node-agent resource usage per node pool.
You want to ensure the node-agent only runs on specific nodes.

How to Enable and Configure

Enable the Feature

In your values.yaml (or via --set), enable the feature:

nodeAgent:
  multipleDaemonSets:
    enabled: true

Define Configurations

Add an entry under configurations for each node pool. Each entry can specify:

nodeSelector: to target the node pool
resources: requests and limits for CPU/memory

Example:

nodeAgent:
  multipleDaemonSets:
    enabled: true
    configurations:
      - nodeSelector:
          kubernetes.io/os: linux
          doks.digitalocean.com/node-pool: pool-1
        resources:
          requests:
            cpu: 300m
            memory: 128Mi
          limits:
            cpu: 400m
            memory: 512Mi
      - nodeSelector:
          kubernetes.io/os: linux
          doks.digitalocean.com/node-pool: pool-2
        resources:
          requests:
            cpu: 100m
            memory: 256Mi
          limits:
            cpu: 200m
            memory: 512Mi

Apply the Chart

Install or upgrade your Helm release as usual:

helm upgrade --install kubescape-operator ./charts/kubescape-operator -f values.yaml

How It Works

When nodeAgent.multipleDaemonSets.enabled is true, the chart will render a separate DaemonSet for each configuration in the configurations array.
Each DaemonSet will have its own nodeSelector and resource settings, and will only schedule pods on the matching nodes.

Notes

The original single DaemonSet is disabled when this feature is enabled.
All other Kubescape Operator components are unaffected and will be deployed as usual.
You can add as many configurations as needed for your node pools.

Troubleshooting

Ensure your node labels (used in nodeSelector) match those assigned to your node pools.
If a DaemonSet is not scheduling pods, check the nodeSelector and resource settings for typos or conflicts.