Installation Troubleshooting
1. Cluster not connecting
Possible solutions
The Helm command didn’t run
Make sure to copy the code in the cluster connection dialog and run it in your cluster.
Kollector did not start
The ARMO portal flags a cluster as connected only after the Kollector microservice creates a connection. Make sure that the Kollector is up and running:
kubectl -n kubescape get pods | grep kollector
The Kollector should be running:
NAME READY STATUS RESTARTS AGE
kollector-0 1/1 Running 0 1m38s
Kollector did not connect
The ARMO portal flags a cluster as connected only after the Kollector microservice creates a connection. Make sure that the Kollector managed to connect to the ARMO Portal. Run:
kubectl -n kubescape logs -f $(kubectl -n kubescape get pods | grep kollector | awk '{print $1}')
You should see the following log:
"msg":"connected successfully","URL":"wss://report.armo.cloud/k8s/cluster-reports?
2. Scanning results are not available
After running the Helm install and seeing that the cluster is connected, the UI does not forward you to the results page
Possible solutions
Refresh the page
Sometimes, the best solution is to simply refresh your webpage :)
The Kubescape or kubevuln pods are not running
You may find that one or more pods are not running after installation. Run the following:
kubectl -n kubescape get pods
Below you can see that only one pod is not running:
NAME READY STATUS RESTARTS AGE
gateway-5c8f486dcf-hsbpq 1/1 Running 0 5m38s
kollector-0 1/1 Running 0 5m38s
kubescape-744bc74f8c-z9drg 0/1 Pending 0 5m38s
kubevuln-7648c9c88f-6q7zd 0/1 Pending 0 5m38s
operator-5c86b8f97d-59nw5 1/1 Running 0 5m38s
Check the status of the pods that aren't running as follows:
kubectl -n kubescape describe pod <podname>
Below, the pod kubescape-744bc74f8c-z9drg has insufficient CPU in the Events section, at the bottom.
Name: kubescape-744bc74f8c-z9drg
Namespace: kubescape
Priority: 0
Node: cluster-8260/192.168.58.2
Start Time: Tue, 07 Mar 2023 12:33:33 +0200
Labels: app=kubescape
app.kubernetes.io/instance=kubescape
app.kubernetes.io/name=kubescape
helm.sh/chart=kubescape-cloud-operator-1.10.3
helm.sh/revision=2
pod-template-hash=5cff758d74
tier=ks-control-plane
Annotations: <none>
Status: Running
IP: 172.17.0.9
IPs:
IP: 172.17.0.9
Controlled By: ReplicaSet/kubescape-5cff758d74
Containers:
kubescape:
Container ID: docker://d6f8a3c613f2ea774e5f8458554b7d70a0b4694370ea6cb522ad5a96c777fdef
Image: quay.io/kubescape/kubescape:v2.2.4
Image ID: docker-pullable://quay.io/kubescape/kubescape@sha256:91d1ad9bf22757b972324a3909da05dbcd9bff9ed4544d04e85a47e7069b65eb
Port: 8080/TCP
Host Port: 0/TCP
Command:
ksserver
State: Running
Started: Tue, 07 Mar 2023 12:33:52 +0200
Ready: True
Restart Count: 0
Limits:
cpu: 600m
memory: 800Mi
Requests:
cpu: 250m
memory: 400Mi
Mounts:
/home/ks/.kubescape/config.json from kubescape-config-volume (rw,path="config.json")
/home/ks/.kubescape/host-scanner.yaml from host-scanner-definition (rw,path="host-scanner-yaml")
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-4fh8f (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kubescape-config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: kubescape-config
Optional: false
host-scanner-definition:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: host-scanner-definition
Optional: false
kube-api-access-4fh8f:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 6m1s default-scheduler 0/1 nodes are available: 1 Insufficient cpu. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
Warning FailedScheduling 52s default-scheduler 0/1 nodes are available: 1 Insufficient cpu. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
Normal NotTriggerScaleUp 1s (x6 over 5m2s) cluster-autoscaler pod didn't trigger scale-up:
Solution
The cluster had a single node assigned. Increasing the node count addressed the problem and the remaining Kubescape pods automatically started once the additional nodes were available.
3.The Kubescape pod restarted
You may find that the Kubescape pod restarted, this means Kubescape probably did not complete the scan. Run the following command:
kubectl -n kubescape get pods | grep kubescape
You should see the following output:
NAME READY STATUS RESTARTS AGE
kubescape-744bc74f8c-z9drg 1/1 Running 1 5m38s
Possible solutions
Typically the restart is caused due to memory issues.
Solution 1
We recommend increasing the memory request and CPU (by editing the Deployment):
kubectl -n kubescape edit deployment kubescape
After the kubescape pod is ready, restart the operator pod:
kubectl -n kubescape rollout restart deployment operator
Solution 2
Re-install the Helm chart and manually set the requested memory/CPU (add this to the helm upgrade --install command):
--set kubescape.resources.requests.memory=800Mi \
--set kubescape.resources.limits.memory=1500Mi \
--set kubescape.resources.requests.cpu=500m \
--set kubescape.resources.limits.cpu=1000m
Updated 5 months ago