Incident Engineering · Kubernetes
Kubernetes / Karpenter Incident Debugging
A troubleshooting case study showing how apparently separate Kubernetes scheduling, Karpenter identity, AWS IAM, and EBS storage constraints formed one failure chain.
Problem
Pods remained pending even though the cluster appeared to have spare capacity.
The investigation began with Kubernetes workloads that would not schedule as expected. Cluster CPU utilization alone did not explain the failure because Kubernetes scheduling decisions depend on resource requests, placement constraints, storage topology, and whether the node-provisioning controller can operate successfully.
Context
The symptom crossed multiple infrastructure layers.
The workload depended on Kubernetes scheduling, Karpenter for dynamic node provisioning, AWS IAM for controller authorization, and EBS volumes bound to specific Availability Zones. Each layer looked reasonable when examined independently, which made the failure chain less obvious.
Architecture
Kubernetes scheduling connected directly to AWS infrastructure.
Pending Pod
↓
Kubernetes Scheduler
↓
Karpenter Controller
↓ AssumeRoleWithWebIdentity
AWS IAM Role
↓
EC2 Node Provisioning
↓
EBS / Availability Zone Constraints
Investigation
Debugging followed the dependency chain instead of focusing on CPU utilization.
The first step was to inspect pod scheduling conditions and compare actual node capacity with Kubernetes resource requests. This separated physical utilization from the scheduler's view of allocatable capacity.
The investigation then moved into Karpenter. Controller logs exposed a failure duringAssumeRoleWithWebIdentity, which shifted the investigation from Kubernetes into AWS IAM.
The Karpenter Kubernetes service account identity did not match the namespace expected by the IAM role trust relationship. The controller and IAM configuration were individually plausible, but the identity values had drifted apart.
After aligning the IAM trust relationship with the service account actually used by the Karpenter controller, the controller could assume its role and resume instance-type discovery and provisioning activity.
Storage placement was also examined. EBS volumes are tied to an Availability Zone, so node placement must remain compatible with the volume topology required by the workload.
Security
IAM trust was treated as part of the application runtime path.
Workload identity depends on an exact relationship between the Kubernetes service account and the AWS IAM trust policy. The fix was not to broaden permissions unnecessarily, but to correct the specific identity relationship required by the controller.
Observability
Scheduler events and controller logs were essential evidence.
Kubernetes events, pod status, Karpenter controller logs, resource requests, node information, and AWS-side identity behavior provided the evidence needed to move from symptom to root cause.
Results
The controller returned to a healthy provisioning path.
Once Kubernetes identity and AWS IAM trust were aligned, Karpenter could operate normally again. The investigation also established EBS Availability Zone affinity as an important constraint to consider when diagnosing similar scheduling failures.
Lessons
Low utilization does not mean Kubernetes has usable scheduling capacity.
Resource requests, not only live CPU usage, influence scheduling decisions.
Kubernetes workload identity and AWS IAM trust must be debugged as one system.
Persistent storage topology can constrain where replacement nodes are useful.
Following the dependency chain is often faster than treating each infrastructure layer as an isolated subsystem.
Related article
Read the published technical walkthrough →Technologies