7 Storage Bottlenecks We See in Almost Every Kubernetes Environment
The majority of Datafy customers use Kubernetes, and this has given us an opportunity to learn about customer environments and use-cases. As we approach Kubecon later this month, we wanted to highlight 7 common storage bottlenecks we have observed often in such environments.
PVCs that look dynamic but sit on fixed EBS constraints
The whole intent of using Persistent Volumes (PVs) is to give Kubernetes pods a state, but a flexible one at that, as required by the specific pod. However, since managing per-pod requirements is not scaleable, we often see uniformity of allocating capacity and performance for PVs that far exceeds the requirements of the individual pod. Right-sizing per pod manually is not a viable solution.
gp3 volumes hitting bandwidth ceilings before IOPS limits
EBS gp3 volumes offer two performance knobs, one for IOPS (IOs per second) and one for bandwidth (MiB/s). If using 16,000 IOPS, at 16KB IO size, the required bandwidth is 256 MiB/s. However the default gp3 setting is 125MiB/s. Customers who neglect increasing the bandwidth will not be able to utilize the full extent of the IOPS they are paying for, which is a shame. Remember to always increase the bandwidth along with the IOPS.
io2 used as a default, not a requirement
Recently EBS expanded the limits of gp3 to 80,000 IOPS, 2GiB/s bandwidth and 64TiB volume capacity. Except for very strong latency consistency and durability requirements, gp3 should always be the default volume type, as it’s significantly cheaper than io2. See <blog on gp3/io2>.
Manual volume resizing during incidents
Running out of disk space is a common scenario. In EC2, it was relatively easy to automate detection for running out of disk space and automatic growth of EBS disks when that happens. But doing the same for Kubernetes PVs is far more complicated. Lacking automation, resizing those PVs during incidents is a slow remedy that relies on manual intervention, especially in large clusters, and is therefore error-prone.
LVM added as a workaround that becomes permanent
Linux LVM (Logical Volume Manager) is sometimes used as a workaround for EBS limitations, for example to achieve higher IOPS than the gp3 limitation. These workarounds have become baked into the environment and hard to get rid off. Recently EBS increased gp3 limitations for IOPS, bandwidth and capacity which makes using LVM unnecessary for most use-cases.
Snapshot sprawl with no lifecycle discipline
Before Kubernetes, EBS DLM or AWS Backup were often used for snapshot lifecycle management. However taking snapshots of Kubernetes PVs often falls out of the coverage of those tools. Taking snapshots of PVs natively via Kubernetes tools is easy to do, but those snapshots are often forgotten without proper lifecycle management, leading to snapshot sprawl.
Clusters scaling compute while storage stays static
There are many Kubernetes automatic scaling solutions for compute, like Karpenter. But what happens to the storage when compute scales up or down? EBS doesn’t support shrinking volumes so cannot scale down, and as mentioned above, growing EBS volumes that are PVs automatically is also complicated. In many situations, the storage remains static as the compute auto-scales up and down.