Sitemap

Image Management on Kubernetes Node

3 min readOct 28, 2024

This article will detail how Kubernetes manages images on nodes.

Pulling Images

Zoom image will be displayed

Kubelet interacts with the CRI components (such as containerd, cri-o) via the gRPC protocol. When creating a new Pod, kubelet calls the gRPC ImageService.PullImage method, and the CRI component downloads the image to the node. The organization and management of images on disk are handled by the CRI component, with differences existing among various CRI components.

The default configuration file for Containerd is located at /etc/containerd/config.toml, where the configuration item root=/var/lib/containerd/ specifies the storage path (this is a root directory that will be subdivided into subdirectories).

The storage-related configuration file for Cri-o is /etc/containers/storage.conf, where the configuration item root=/var/lib/containers/storage specifies the storage path. Additionally, there is a configuration item imagestore that can separately specify the image storage location.

Image GC

Zoom image will be displayed

Kubelet manages image deletion through garbage collection policies, with relevant configuration items including:

  • imageMinimumGCAge: Default value is 2 minutes, indicating the minimum lifespan of an image to prevent deletion of too recently created images during garbage collection.
  • imageMaximumGCAge: Default value is 0 seconds (indicating this feature is disabled), representing the maximum lifespan of an image (this feature was introduced in v1.30). If the time since the last use of the image exceeds this value, the image will be deleted.
  • imageGCHighThresholdPercent: Default value is 85; when disk usage is greater than or equal to 85%, garbage collection of images is triggered.
  • imageGCLowThresholdPercent: Default value is 80; during image garbage collection, once disk usage drops to 80% or less, image deletion stops.

Kubelet continuously loops through the following GC process for images every 5 minutes (this value is hard-coded):

  1. imagesInEvictionOrder: kubelet sends three gRPC requests (ImageService.ListImages, RuntimeService.ListPodSandbox, RuntimeService.ListContainers) to the CRI to retrieve usage information for all images on the node, sorting them by the last usage time.
  2. freeOldImages: If the time interval since the last use of an image exceeds imageMaximumGCAge, kubelet sends an ImageService.RemoveImage request to the CRI to delete those images.
  3. ImageFsStats: kubelet sends an ImageService.ImageFsInfo request to the CRI (cri-o depends on cadvisor) to retrieve disk usage information.
  4. freeSpace: When disk usage reaches or exceeds the imageGCHighThresholdPercent high threshold, deletion of images is triggered. Images are deleted in order of the longest unused time, one by one, with each deletion checking if the disk usage falls below the imageGCLowThresholdPercent low threshold. Once it drops below this threshold, image deletion stops.

Conclusion

On the node, images are managed by CRI components (such as containerd, cri-o). Kubelet performs image garbage collection every 5 minutes. If imageMaximumGCAge is configured, images meeting the conditions will be deleted regardless of whether disk usage has reached the high threshold. Afterward, image deletion will be triggered based on the configurations of imageGCHighThresholdPercent and imageGCLowThresholdPercent, with the actual deletion actions executed by the CRI components, while kubelet is only responsible for initiating gRPC calls.

Rifewang
Rifewang

Written by Rifewang

Software Engineer & Architect

No responses yet

Write a response