Go in Kubernetes: Less isn't always more 📉

Published on
·
Time to read
1 min read ☕
Post category
shortsgokubernetes

Typically, we correlate a service's request-handling capacity with its allocated resources (CPU/Memory). While this relationship generally holds, our recent experiments revealed some nuanced behaviors when running Go services in Kubernetes: setting CPU requests as low as possible doesn't always translate to better efficiency.

Typically, we correlate a service's request-handling capacity with its allocated resources (CPU/Memory). While this relationship generally holds, our recent experiments revealed some nuanced behaviors when running Go services in Kubernetes: setting CPU requests as low as possible doesn't always translate to better efficiency.

We tested our Go service's performance under burst requests between two resource configurations. Both configurations had CPU requests set but no CPU limits define:

SpecsCPUMemory
High 📈1 Core512 MB
Low 📉100 Milicore (0.1 Core)128 MB

Result

Figure 1

Spot the vertical line of CPU and Memory panel. The left-hand side represents the high-spec run, while the right-hand side represents the low spec run. Despite similar requests per minute, the CPU utilization patterns weren't as expected. The higher resource allocation handled the load smoothly, while the lower allocation resulted in cascading errors – particularly around Redis connections (timeouts, pool exhaustion).

The root cause is combination of CPU throttling and CPU steal. When we drastically reduce CPU requests, it disrupts reserved CPU-intensive operations like Redis pool management. This also manifests in garbage collection anomalies, with GC pauses spiking to 9 seconds as shown in figure below.

Figure 2

A common suggestion is to use uber-go/automaxprocs to automatically handle CPU allocation and define CPU limit in kubernetes manifest. But there's a catch: automaxprocs is most effective when CPU limits are ≥1000m. Since GOMAXPROCS can't be set below 1, services limiting <1000m CPU will still face throttling issues.

If your services don't need a full CPU core (1000m), this is where the art of resource allocation comes in. Find the sweet spot between resource allocation and utilization!