✳flâneur — a map of the web's best reading

k8s-1m Overview

bchess.github.io · 9,350 words · saved by 4 readers

Several years ago at OpenAI I helped author Scaling Kubernetes to 7500 Nodes which remains one of the CNCF’s most popular blog posts. Alibaba made a post about running Kubernetes clusters with 10K nodes. Google made a post about 15K nodes with Bayer Crop Science. Fast forward to today, GKE supports running some clusters up to 65K nodes, and AWS recently announced support for clusters up to 100K nodes. In online forums and in my own conversations with peers, I’ve encountered a lot of debate about how big a Kubernetes cluster can get. What tends to be lacking from these discussions is hard data and evidence-backed justifications. I’ve worked with engineers reluctant to push things beyond what they’ve seen before because they’re fearful or uncertain of what may go wrong. Or when something does go wrong, the response is to scale down the cluster rather than understand and address the bottleneck. The spirit of the k8s-1m project is to identify the hard blockers to scalability. What are the

k8s-1m Overview k8s-1m Overview This is an effort to create a fully functional Kubernetes cluster with 1 million active nodes. Table of Contents Why? Components Networking Pod IPs IPv4-only external service dependencies Network Policies Network flow needs (# of TCP connections) Managing state kube-apiservers vs etcd Meeting the QPS needs for a 1M node cluster etcd is too slow Reduce durability and eliminate replicas Reduce the interface Txn-Put mem_etcd: custom in-memory etcd Watch() Watches per node Update() Caching and locking Garbage collection Scheduler Basic design: shard on nodes The pai

Explore this link on the map →

saved by

related reading