2026-09-23 16:50:31
Models getting bigger and GPU partitioning getting more important are two things that can be true at the same time.
This September, at the GPUStack ecosystem conference in Beijing, I found myself chatting with their CTO, Sheng Liang, over the conference dinner. He asked me one question: in the era of large models, is GPU virtualization and partitioning still that important?
He asked it politely, but for someone who works on the HAMi community and its commercialization, it stung. Everyone understands the subtext: models no longer fit on a single card, and you people are still figuring out how to slice one card into pieces. Isn’t that doing aerodynamics research on a wagon wheel?
I gave a polite non-answer that evening, but the question kept bugging me, and I realized it deserved a proper written response. As it happens, in mid-September TypeSafe AI released Jev, a model that cannot chat, and that made the question much more interesting. This post is my formal answer.
First, what Jev is. TypeSafe calls it a System One Model: you send an unstructured state plus a set of typed questions, and the output is not free text but three predefined types, Choice (pick one from a list), Score (rate on a scale), and Noul (yes or no), each with a probability distribution and confidence.
Why do I say it “cannot chat”? Consider how we use a large language model (LLM) today to decide “is this ticket about billing”. Asking a big model to do this is like hiring a Pulitzer winner to fill in a multiple-choice sheet: they will absolutely write you a heartfelt short essay first, and then you scrape the answer out of it with a regex. To get them to say “yes”, we first taught them to write everything, then strapped them to a chair with JSON mode, and then wrote a parser to guard against improvisation.
Jev deletes that entire pipeline. It does no autoregressive generation; it returns typed results that code can consume directly. Multiple questions can go into a single call, evaluated in parallel against the same state. The official docs claim that “adding questions barely changes the response time”, because each question is evaluated independently and cannot pollute the others’ context.
An analogy: a large LLM is like a senior consultant who reads materials and writes reports. Jev is like the real-time approval node inside a company. It will not write your report, but it can process huge volumes of “approve / reject”, “A/B/C”, “risk 0 to 10” judgments per second. You would not hire a novelist to run your access control system, yet that is exactly the architecture many companies run today.
Two buckets of cold water, as usual. First, “Jev cannot hallucinate” needs de-noising: type safety guarantees the output stays within the predefined schema, not that the business judgment is correct. A perfectly valid high_risk can still be a mistake. Second, the “193.6x faster, 444.6x cheaper” numbers on the homepage come from workflows designed by TypeSafe’s own team, and the company itself admits they sit at the high end of real-world gains. They indicate how much headroom the new paradigm has, not that “Jev is 200x faster than GPT”. One more easily misread number: 250,000 tokens/s is an API rate limit, not measured single-GPU throughput. API throughput, model throughput, and GPU throughput have never been the same thing.
Reading introductions gets you nowhere; run it to get the feel. console.typesafe.ai is open for registration. Create an API key, then:
pip install typesafe-sdk
export TYPESAFE_API_KEY=your_keyThe example in the official docs is literally a support ticket classifier, three questions in one call:
from typesafe_sdk import Choice, Noul, Score, TypeSafeClient
with TypeSafeClient() as client:
response = client.system_one(
state={"document": "I was charged twice. Please fix this ASAP."},
questions={
"billing": Noul(instructions="Is this ticket about billing?"),
"tone": Choice(
instructions="What is the customer's tone?",
criteria={"calm": None, "frustrated": None, "angry": None},
),
"urgency": Score(
instructions="How urgent is this ticket?",
criteria=["can wait", "this week", "today"],
),
},
)
print(response.nouls["billing"].noul) # yes/no decision
print(response.choices["tone"].choice) # selected option
print(response.scores["urgency"].score) # rating levelNotice the shape of this API: no prompt, no temperature, no system message. You submit questions and options; you get back types and probabilities. If you have written LLM apps for a while, it feels like something is missing at first glance. Think again, and you realize what is missing was never supposed to be there.
The ecosystem is assembling faster than I expected. Beyond the official Python and TypeScript SDKs, the Rust community has typesafe_ai_rs, Pydantic’s docs already show a TypeSafeModel integration, and someone has built an MCP server for Jev. If you would rather not register directly, OpenRouter, Cloudflare Workers AI, and Vercel AI Gateway can all reach it. Chinese WeChat tech columns are filling up with Jev explainers too. The heat is real.
More interesting: you can already reproduce this approach with open source. Bespoke Labs’ Nimble is one example, a LoRA adapter trained on Qwen3.5-9B, about 165 MiB, Apache 2.0. Its method is blunt: score the allowed answer tokens directly, and generate no free-form text at all during inference.
from inference import NimbleModel
model = NimbleModel("nimble-model")
result = model.score(
context="The store accepts returns within 30 days. "
"This item was bought 12 days ago.",
schema={
"eligible": {
"type": "boolean",
"description": "Is this item within the store return window?",
}
},
)
print(result["fields"]["eligible"]["probabilities"])There are constraints: at most 26 choices per field, inputs over 2,048 tokens are rejected rather than truncated, and you need a CUDA GPU. But its existence proves a point: the so-called System One paradigm is not black magic. Take a general model, constrain its output space, score candidate answers directly, and you get something 70 or 80 percent of the way there for the cost of a LoRA. Which in turn makes Jev’s pricing worth studying: $0.042 per million input tokens, output free, with the homepage proudly noting it is 238x cheaper on input than a certain frontier model. Competitive pressure will likely arrive faster than expected.
Now back to architecture. The real value of this class of models, I think, is not replacing a 70B model with a 3B one, but decomposing the intelligence workload that one big model used to monopolize. An enterprise agent may contain dozens of judgments: intent recognition, permission classification, document relevance, content risk, tool routing, whether to retry, whether to hand off to a human. These do not need frontier-level capability on every single call.
The more sensible division of labor: code owns deterministic logic, specialized decision models own high-frequency judgments, small models own local language tasks, and big models handle only genuinely hard reasoning. NVIDIA Research’s position paper on SLMs and agentic AI says the same thing: most calls inside an agent are repetitive and specialized, and a heterogeneous model system is more economical than “call the same big model for everything”.
Expand the earlier ticket example: code does authentication and field validation first, then asks a decision model three questions in one call. Is it billing? What tone? How urgent? With enough confidence, route and execute directly; refunds and database writes stay in code. Only low-confidence or open-ended cases escalate to a big model.
What I like about this architecture is precisely that it is not “more AI”. It hands system control back to software. Follow this direction and applications grow a distinct judgment plane: a layer of low-latency, confidence-carrying decision models sitting between code and big models. The API Gateway decides where traffic goes; the judgment plane decides who should think about this request. Jev is an early implementation of this direction, and traditional classifiers, rerankers, SLM routers, and Nimble above all belong to the same layer.
Now the direct answer: as models get bigger, does GPU partitioning still matter?
Once you accept that “one application runs a dozen models at once”, the GPU layer’s problem changes. A frontier model may need 8x H200, while the embedding model next to it needs a few GB of memory; rerankers, small vision models, guardrails, and notebooks each consume a sliver of GPU. If you still use Kubernetes’ traditional whole-card semantics:
resources:
limits:
nvidia.com/gpu: 1then that is a commuter who calls a ride-share and gets a 49-seat bus every morning, with boarding forbidden for anyone else. Small models waste; many models fragment.
“Models are getting bigger, so GPU partitioning is obsolete” confuses two opposite directions on the resource axis.
Model parallelism solves one workload using many cards. Tensor Parallel, Pipeline Parallel, and Ray distributed inference all belong here: one card is not enough, use multiple cards on the same node; still not enough, go cross-node.
GPU partitioning solves many workloads sharing one card. HAMi (a CNCF Incubating project) turns the GPU from an integer device into a resource schedulable by memory, compute cores, and device share, pins the exact card at scheduling time, auto-matches MIG templates by compute and memory requirements, and enforces real resource constraints at the CUDA layer via HAMi-core rather than just numbers in the scheduler’s ledger. At the end of the day, split versus join is not just a question of direction. How finely you can split is the real engineering divide.
Here is the fun fact: the industry has already answered half of Sheng’s question for him. Platforms that aggregate models, the ones stitching big models across many cards, are shipping GPU partitioning and flexible slicing in the same product. Joining cards together with one hand while slicing them apart with the other is not a split personality; it is reality: demand exists in both directions at once, and picking a side is the mistake.
Looking further out, heterogeneity only gets worse. MoE decouples total parameters from active ones: Qwen3-30B-A3B has about 30B total parameters but activates only about 3B per token. 4-bit quantization and distillation keep pushing the real per-request cost down. Work like DistServe and Mooncake even splits prefill, decode, and KV cache across separate resource domains. Scheduling units will get finer, not coarser.
That said, do not treat GPU utilization as the only KPI. Cramming four inference instances into one card, taking nvidia-smi from 30% to 90%, does not automatically improve total cost of ownership; if P99 latency degrades and the failure domain widens, that utilization is worth little. HAMi fits GPU pools with obvious fragmentation, multiple tenants, and a high share of dev and test. For services with strict latency SLOs and cards saturated long-term, exclusive ownership or hardware isolation like MIG is the safer bet.
Jev’s significance is not that “small models won”, but that not every kind of intelligence needs to generate language. When one decision model or one LoRA adapter can handle half the checkbox work in an application, applications shift from “one big model does everything” to a dozen small models, each doing its job.
Big models keeping growing will not eliminate small models or GPU virtualization; heterogeneous model architectures mean the same cluster simultaneously faces “model too big” and “task too small” resource problems.
So the core capability of next-generation AI Infra is not merely slicing GPUs, nor merely stacking them, but freely splitting and composing compute resources when the workload demands it. In one sentence: split the small and fragmented, join the large and saturated, isolate the high-SLO, share the low-utilization.
So here is my answer: yes, and more than ever.
2026-09-15 15:55:15
Standing in the Shanghai venue in 2026, I found myself thinking about the days eight years ago when Dan Kohn and I organized the first KubeCon China.
In September 2026, KubeCon + CloudNativeCon returned to mainland China once again, back in Shanghai. The full name of the event has grown much longer than before: this time it also included the OpenInfra Summit and PyTorch Conference, with agent and MCP-focused events running alongside.
The longer name is not a trivial detail. It is almost a microcosm of the technological shifts of the past few years: cloud native is no longer the only protagonist on stage, AI has pushed its way into the spotlight, and Kubernetes, OpenStack, and various cloud native projects are repositioning themselves within the AI technology stack.
For me, this return carried a more personal layer of meaning.
The first KubeCon China was in 2018, also in Shanghai. Dan Kohn came to China back then to discuss and plan the event with us. Kubernetes was still expanding rapidly in those years; many people were hearing the term “Cloud Native” for the first time, and our hottest topics were containers, microservices, service mesh, and whether Kubernetes would become the data center operating system of the future.
More than 2,500 people attended that conference, a record for a first international KubeCon at the time. I still remember clearly how Dan looked on stage, in the venue, and at community events.
Dan later passed away from illness. Today, the KubeCon scholarship program is named after him. Seeing the Dan Kohn Scholarship again in Shanghai, I could not simply treat it as one more program name on the conference website. For me, it connects to a group of real people and to the earliest days of China’s cloud native community.
Eight years on, Kubernetes no longer needs to prove that it can run in production. It has long been part of the infrastructure. Yet the center of the technology world is shifting as well. The question of those years was “how do applications run on Kubernetes”; today the question has become “how does AI run more efficiently and more reliably on top of heterogeneous computing power”.
This time I was mainly responsible for the HAMi booth. Besides meeting many old friends I had not seen in years, one immediate impression was that there were more international attendees than I expected.
At the HAMi booth in particular, my sense was that more than half of the visitors came from overseas. Of course, this is just one person’s observation at a single booth and does not represent the overall attendee mix of the conference. But it still says something: when it comes to GPU scheduling, sharing, and heterogeneous computing management, Chinese projects are drawing global developers who actively want to learn more.
The Todea team from Korea wrote a very detailed recap after the event, summarizing the shared theme of the conference as “recovering capacity from the same hardware”. That phrasing matches what I felt at the booth. The questions people asked about HAMi were no longer just “can Kubernetes recognize GPUs”, but how to partition GPUs, how to improve utilization, how to run training and inference in the same cluster, and how to manage different accelerators such as NVIDIA, Ascend, Cambricon, and Hygon.
This means the GPU has gone from being an add-on device to a first-class resource that the Kubernetes control plane must understand.
The Todea recap also documented several production cases. China Merchants Bank put training and inference on a shared Kubernetes foundation, raising average accelerator compute utilization from 35% to over 60%; INTSIG improved GPU utilization and reduced costs through GPU partitioning, bin-packing scheduling, affinity rules, and elastic scaling; and the case from Viettel in Vietnam was even more direct: under the same workloads and service levels, utilization per card rose from 13% to 59%.
These numbers come from each team’s presentations at the conference; the scenarios and measurement criteria are not identical, so they cannot be compared side by side directly. But together they point to one shift: the primary question of AI infrastructure is moving from “do we have GPUs” to “how much useful work are our existing GPUs actually doing”.
If you only listen to social media, you might think cloud native in China has cooled off compared with a few years ago. Startup pitch decks have changed their labels, marketing budgets have flowed to large models and agents, and many people who used to work on cloud native have started studying inference, GPUs, vector databases, and AI platforms.
I am one of them. Over the past few years I have gradually shifted my focus to AI infrastructure, working on adapting GPUs to Kubernetes. Only after making that turn did I realize that this was not leaving cloud native; it was following cloud native deeper into computing resources.
The “State of Cloud Native Development in China” report released by the CNCF and SlashData during the conference painted a different picture. As of Q1 2026, China has about 1.75 million cloud native developers, of whom roughly 400,000 also work on AI development. The share of cloud native among Chinese backend developers has grown from 30% two years ago to 48%.
Another statistic feels distinctly Chinese: 48% of China’s industrial IoT developers are cloud native developers, above the global average of 42%. Manufacturing, telecommunications, energy, and hardware are the industries where Chinese developers are more concentrated; these systems often need to run for long periods and strongly favor private, controlled infrastructure. Cloud native in China has never simply meant public cloud. It increasingly lives in on-premises data centers, dedicated clouds, edge nodes, and industry platforms.
So I would rather put it this way: cloud native in China has not disappeared; it has sunk beneath the surface. It has gone from a novel concept to the default engineering approach for many systems, and internal developer platforms have hidden it even further. According to the report, 88% of backend developers worldwide already work in some form of standardized DevOps or platform environment. A developer may use a Kubernetes-based platform every day without ever needing to write a Dockerfile, configure container networking, or operate a cluster.
When infrastructure becomes invisible, its voice in public discourse grows quieter, while its responsibility in production systems only grows heavier.
This conference left me feeling that the CNCF’s presence in China has not simply grown stronger or weaker; its nature has changed.
KubeCon China 2018 had a very clear center: Kubernetes and the CNCF. At that time China was the world’s third-largest source of contributions to CNCF projects, and Chinese companies joining the foundation, donating projects, and becoming top contributors were the biggest news of the event.
In 2026, China remains the world’s second-largest source of contributors to CNCF projects. Code contributions have not disappeared, and Kubernetes has not lost its place in production. But this event needed to share a single stage with OpenInfra and PyTorch to fully express today’s technical demands. Infrastructure, container orchestration, model frameworks, and agent systems have been chained together into one story that no single foundation can tell alone.
This raises a question: now that cloud native has become the default foundation, can the CNCF keep defining the questions that the next generation of developers cares about most?
In China, this question is especially practical. Technology decisions at Chinese companies are increasingly shaped by heterogeneous chips, data compliance, private deployment, and local supply chains. Community members still contribute code, but the day-to-day connection between the foundation and local developers is harder to maintain than the contribution numbers in project repositories. A conference can bring one concentrated reunion; real presence comes from year-round local content, user case studies, maintainer growth, and cross-language collaboration.
If the CNCF only emphasizes that “Kubernetes still matters”, that is of course true, but it is no longer enough. What matters more is explaining clearly how Kubernetes takes on AI’s new workloads, and how Chinese developers and domestic hardware can genuinely enter global standards and upstream communities.
At the end of 2025, the Linux Foundation established the Agentic AI Foundation (AAIF), with Anthropic’s Model Context Protocol (MCP), Block’s goose, and OpenAI’s AGENTS.md among its first projects. In less than a year, a new open source narrative has taken shape around agent protocols, tool calling, context management, and collaboration standards.
The impact of the AAIF on the CNCF is not that agents will replace Kubernetes. What it is really competing for is developer attention, vendor budgets, and the entry point to standard-setting.
Ten years ago, a developer who wanted to help build the next generation of infrastructure most likely started with containers, Kubernetes, and cloud native. Today, a young developer is more likely to start with LLM APIs, MCP, agent frameworks, and AI coding tools. The first interface they care about is no longer the Kubernetes API, but how models call tools, how agents exchange context, and how to control permissions and costs.
That is also why this edition of the conference placed agent and MCP-focused events in the same week as KubeCon. Agentic AI is forming its own community center, and its boundary with cloud native has not yet stabilized.
I do not believe the CNCF and the AAIF will end up in a zero-sum competition. The AAIF sits closer to the protocol layer and the developer entry point for agents, while the CNCF is better at governing long-running distributed systems. Once an agent truly enters production, it still runs into identity, isolation, networking, policy, observability, resilience, and multi-tenancy problems, which are precisely the problems the cloud native community has solved again and again over the past decade.
But cooperation will not happen by itself. Who defines the runtime unit of an agent, who manages state, who standardizes identity and telemetry data, and who owns the security boundary? These questions will all shape how future projects and foundations divide the work.
We used to say that Kubernetes is a general-purpose orchestration platform. Agentic AI will force us to re-examine the words “general purpose”.
Traditional web service workloads are relatively stable: a Pod starts and keeps handling requests, and CPU and memory can roughly describe its resource needs. An agent, by contrast, may start suddenly, call several external tools, wait on model inference, save its context, and then go dormant or terminate. It resembles both a short-lived task and a stateful long-runner; it needs sandbox isolation, yet it must also access external systems on the user’s behalf.
This puts several new kinds of pressure on Kubernetes.
The first is the resource model. Training, inference, and agent workloads can no longer be described by “how many GPUs” alone. Video memory, compute ratios, interconnect topology, model caches, and domestic heterogeneous accelerators all need to enter scheduling decisions. HAMi and Dynamic Resource Allocation (DRA) address the layer of this pipeline closest to the device.
The second is the workload model. A single LLM inference replica may no longer be one Pod but a group of Pods coordinating across nodes. Prefill/decode disaggregation, gang scheduling, LeaderWorkerSet, and multi-cluster scheduling all show that the original Deployment abstraction is no longer sufficient to express AI systems.
The third is security and governance. An agent does not just respond to requests; it takes actions. It needs its own identity, fine-grained authorization, network egress control, tool-call auditing, and recoverable execution records. Simply stuffing each agent into a Pod does not automatically solve these problems, but Kubernetes’ ServiceAccounts, policy engines, sandbox runtimes, and observability stack provide a foundation that can keep evolving.
The CNCF and SlashData report also shows that AI developers follow a different cloud native maturity path. They first build data pipelines with Kubernetes, microservices, event-driven architecture, and observability tooling; then rely on immutable infrastructure to make training reproducible; and only after moving into production inference do they need service mesh, chaos engineering, and multi-cluster management more.
This ordering shows that AI is not just another new workload on top of cloud native. It is reordering the priorities of cloud native technology.
Technology trends can be summarized from reports and agendas, but the truly irreplaceable part of a conference remains the people.
This time I met many old friends I had not seen in years. Some still work on Kubernetes; some have moved on to gateways, large models, agents, or GPU chips; others have changed roles and companies several times over. We talked about past projects, and about the new problems each of us faces today.
I have come to believe more and more that a community is not a group of people who forever discuss the same technology. A real community is one where, after the technology cycle turns, those same people are still willing to come back with new questions.
In 2018, we believed Kubernetes would transform infrastructure. In 2026, we are judging whether Kubernetes can support AI’s transformation of software. The questions have changed, and so have the people, but the way of building trust through open source and collaborating across companies has not.
When Dan pushed to bring KubeCon to China, its greatest significance was never just hosting a conference. He helped build a bridge between Chinese developers and the global open source community. Today, that bridge needs to carry more than cloud native; it also needs to connect chips, models, inference systems, and agent protocols.
If I had to sum up KubeCon China 2026 in one sentence, I would say: cloud native has not left the stage; it is losing the center of the stage while becoming the foundation the entire stage stands on.
China’s cloud native community still has scale, production practice, and sustained upstream contributions; its participants have simply dispersed into AI infrastructure, heterogeneous computing, platform engineering, and agent systems. The challenge for the CNCF is to reconnect these new directions. The impact of the AAIF reminds everyone that the entry point for the next generation of open standards may have already shifted from container orchestration to the protocols between agents and tools.
For Kubernetes, this is both pressure and a second chance. It does not need to become an agent framework, nor own the entire AI technology stack. What it needs to do is turn GPUs, networking, identity, isolation, state, and observability into a reliable production foundation, so that AI workloads can run at scale the way microservices do today.
For me personally, this return to Shanghai felt like recalibrating my coordinates. Looking back, I can still vividly picture Dan planning the first conference with us; looking forward, I have already devoted more of my energy to combining GPUs with Kubernetes. The world changes fast, but as long as the community is still willing to reunite, discuss, and build together, the road from the past into the future is still there.
2026-08-28 15:38:12
GPU scheduling is not just “who gets which card”—it is answering three layered questions in sequence: which node, which card, and where the card sits inside the machine. This post walks from that three-layer framework to the HAMi v2.10 policy chain, then settles the most commonly misread policy semantic: mutex does not ask for exclusion from other mutex Pods; it asks for a zero-tenant card. The verdict comes from a decisive experiment anyone can reproduce on a laptop.
An eight-GPU H100 server sells for more than a million yuan, and the GPUs account for the bulk of that cost. You buy it for compute, but the card has three properties a CPU does not: it is too expensive to leave idle; its granularity is too coarse—an entire card carries 80 GB of memory while an inference replica may need only two or three; and its performance is sensitive to neighbors—co-located tenants contend with each other for compute, memory bandwidth, and cache. Together, these three define the fundamental problem GPU scheduling must solve: place this expensive card, at the right granularity, in the right position, for the right workload, at the right time.
The Kubernetes default scheduler, meanwhile, was designed for stateless microservices. Its entire knowledge of a GPU is one line in the resource declaration:
resources:
limits:
nvidia.com/gpu: 1Through its eyes, a GPU is an indivisible integer count. That creates three blind spots:
The first two blind spots live inside the card, and GPU-sharing projects like HAMi exist precisely for them: slice the card by memory and compute, and let multiple Pods co-locate. The third blind spot is subtler and the most counterintuitive. Manan Paliwal’s article Why Kubernetes Is Slowing Down Your GPUs describes exactly this scenario: an H100 cluster that looks perfectly healthy, yet distributed training runs 30% to 40% slower than expected, and adding more cards barely helps. The cause is not CUDA, PyTorch, or NCCL—the scheduler placed the GPU and its RDMA NIC on different NUMA nodes, so every packet has to cross the inter-socket CPU link before reaching the network. Or consider a four-card machine where GPU 0 and GPU 1 hang under PCIe switch A and GPU 2 and GPU 3 under switch B: a workload requests two GPUs, and the scheduler is entirely capable of returning GPU 1 plus GPU 2—the count is right, the placement is wrong, and communication now crosses switches and the CPU interconnect. The most stinging line in that article sums it up: nothing crashes, no job fails, performance quietly evaporates.
Turn the blind spots into a question list, and a single GPU scheduling decision is really three layered questions answered in sequence:
| Dimension | Question to answer | Consequence of getting it wrong or skipping it |
|---|---|---|
| Node | Which node to place on | Poor bin-packing leaves a pile of fragmented nodes; poor spreading wastes whole cards |
| Card | Which card, co-located or exclusive | Sharing denied where it fits is waste; exclusivity denied where it matters is performance interference |
| Topology | Physical position of the card within the machine | Cross-NUMA, cross-PCIe traffic silently bleeds performance |
The default scheduler answers only the first layer, crudely, with “are total resources sufficient?"—layers two and three are left entirely to luck. The Kubernetes ecosystem’s answers are layered: Kueue does hardware-aware whole-job placement at the queueing layer, kubelet’s Topology Manager aligns NUMA inside the node, and the NVIDIA Network Operator with Multus exposes RDMA networks directly to workloads. HAMi’s angle is different: it moves layers two and three into the scheduler itself, expressed as a language of Pod annotations.
Concretely, the v2.10 scheduling model looks like this:
hami.io/node-scheduler-policy for the node level (a single value, binpack or spread), and hami.io/gpu-scheduler-policy for the card level (comma-separated policy chains as of v2.10);Mapped against the table above: binpack and spread answer “how to pack”; numa and topology-aware answer “is the position right”; mutex answers “may it have neighbors.” A chain like mutex,binpack,numa translates to: give me a card nobody has used, pack as tightly as possible, and ideally keep it on the same NUMA node.
On this map, mutex is the only policy carrying exclusive semantics—and the newest addition in v2.10. The precise meaning of its “exclusivity” happens to be the murkiest part of both the documentation and community understanding, and that is precisely the question to settle next.
Verdict first, answering the most direct question: does hami.io/gpu-scheduler-policy: "mutex" make Pods carrying the annotation mutually exclusive with one another? No. Its actual behavior is:
HAMi v2.10 brought the mutex policy and composable policy chains (such as mutex,binpack) to the hami.io/gpu-scheduler-policy annotation. Two readings of the mutex semantics have long coexisted in the community:
| Reading | Meaning |
|---|---|
| Reading A: zero tenants | A mutex Pod may only be scheduled onto a GPU card with no workload at all, whether or not the existing workloads carry mutex |
| Reading B: mutex-only exclusion | A mutex Pod is exclusive only against other mutex Pods and may share a card with ordinary Pods |
Most test reports (including one internal LWS test of ours) cannot distinguish between the two readings, because they set gpuSchedulerPolicy: mutex as a global default. When every Pod is mutex, the two semantics predict identical behavior: with any Pod already on a card, later mutex Pods get rejected. To separate A from B, you must design a mixed scenario: fill the card with ordinary Pods only, then submit a mutex Pod.
The verdict has already been stated: HAMi implements Reading A. Four layers of evidence support it: the original request in issue #2009 says “no existing users”; the implementation in PR #2011 checks dev.Used > 0 (a count that covers every Pod on the card); observations from our real-GPU experiment on a four-card T4 node on GKE; and the local mock experiment in this post. This article documents the full reproduction of the fourth layer of evidence—it needs no GPU at all and runs in about twenty minutes on a laptop.
To understand why this policy appeared, start from the fundamental tension of GPU sharing.
The value of GPU virtualization projects like HAMi lies in slicing: cut an 80 GB card into shares by memory and compute, co-locate multiple inference replicas on it, and utilization goes up while cost comes down. But slicing has an inherent price: co-located tenants share SMs, share memory bandwidth, and pollute each other’s L2 cache. Throughput-oriented batch jobs don’t care; two classes of workloads find it lethal: latency-sensitive online inference (uncontrollable tail latency), and benchmarking or training jobs that need a stable baseline (irreproducible results).
Before mutex, users who wanted a card to themselves had only two unsatisfactory options:
mutex fills the gap in between: a scheduler-layer, Pod-granularity, zero-hardware-dependency soft-exclusivity switch. It introduces no new isolation technology; it simply tells the scheduler “give me a card with zero tenants right now.” Combined with HAMi’s existing soft slicing, a cluster can for the first time serve both classes at once: shared slices amortize cost at peak, while mutex claims an idle card exclusively for overnight benchmarks or latency-sensitive services, released the moment the job finishes.
Its arrival in v2.10 is not isolated either. The same roadmap (#1889) advanced policy chains (#2010) in parallel, because real clusters stack their demands: pack tightly to free whole cards, keep NUMA affinity, and still hold a few exclusive cards for critical workloads. The chained expression mutex,binpack,numa is the complete form of those demands. Another typical case is replica-group scheduling with LeaderWorkerSet: GPU Pods within the same replica group should not share physical cards, and marking the group’s Pods as mutex satisfies that naturally.
There is one more design trade-off worth savoring: mutex exclusivity is one-way and binds only its own placement moment. Implementation-wise it merely reuses the scheduler’s existing Used counter (any Pod allocation on a card increments it), introducing no runtime locks or reservations. The choice keeps the implementation minimal and the semantics unambiguous, at the cost of not constraining later workloads: ordinary Pods scheduled afterward can still join the card. For end-to-end exclusivity, request a whole card’s resources or pin the card with use-gpuuuid. Where to draw the line of “guarantee” in a scheduling system is always a trade-off; HAMi draws it at placement time and leaves stronger constraints for users to express explicitly.
The evaluation model of the policy chain was given in the previous section; here we zoom into the exact behavior of the mutex filter inside the scheduler. It stands guard at the entrance of the candidate card set with a single criterion: whether the card’s Used count is 0. That count covers every Pod on the card, mutex annotation or not.
The “one-way exclusivity” is clearest on a timeline. The same Used > 0 check executes only when the mutex Pod itself is being scheduled:
The three experiments below map to these two diagrams: Experiment 1 verifies that the filter condition is “any tenant” rather than “mutex tenants only”; Experiment 2 verifies the happy path of the first diagram; Experiment 3 verifies the one-way property of the second.
The overall idea: bring up a single-node cluster with kind, install the HAMi scheduler (control plane only), then use HAMi’s official mock-device-plugin to register two fake Tesla T4s on the node. The HAMi scheduler runs its policy computation on them exactly as it would on real cards.
The experiment uses exactly the same build as HAMi’s official Lab 14 (the GKE experiment): the Helm chart comes from HAMi source commit 45b3d46769b44cfc1445728dfcb8e524939afba1 (master HEAD on 2026-08-17, i.e. the v2.10.0 release candidate), and the image is the per-commit tag HAMi CI published for that commit, projecthami/hami:45b3d46. Do not take the shortcut of latest: it is a moving tag, and by the time of writing it had already drifted to a newer master commit (the version embedded in the image changed from 45b3d46 to 949f78e), which would distort the reproduction.
kind nodes pulling images themselves is at the mercy of the network; pull on the host first, then load:
kind create cluster --name hami-mutex2 --image kindest/node:v1.36.1
# Pre-pull on the host (retry a few times on a flaky network)
docker pull projecthami/hami:45b3d46
docker pull projecthami/mock-device-plugin:latest
docker pull \
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.36.1
docker pull liangjw/kube-webhook-certgen:v1.1.1
kind load docker-image projecthami/hami:45b3d46 --name hami-mutex2
kind load docker-image projecthami/mock-device-plugin:latest --name hami-mutex2
kind load docker-image \
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.36.1 \
--name hami-mutex2
kind load docker-image liangjw/kube-webhook-certgen:v1.1.1 --name hami-mutex2The kube-scheduler image is the sidecar used by the scheduler embedded in the HAMi chart—its tag must match the cluster version; certgen is the image for the webhook certificate Job during installation.
# Fetch the same chart source as Lab 14
curl -fsSL https://codeload.github.com/Project-HAMi/HAMi/tar.gz/45b3d46769b44cfc1445728dfcb8e524939afba1 \
-o hami-src.tar.gz
tar xzf hami-src.tar.gz
helm install hami HAMi-45b3d46769b44cfc1445728dfcb8e524939afba1/charts/hami \
-n kube-system \
--set global.imageTag=45b3d46 \
--set devicePlugin.enabled=false \
--set mockDevicePlugin.enabled=true \
--set mockDevicePlugin.image.tag=latest \
--set mockDevicePlugin.image.pullPolicy=IfNotPresentThree parameters matter:
devicePlugin.enabled=false: no real GPUs here, so HAMi’s own device plugin is disabled;mockDevicePlugin.enabled=true: the mock plugin takes over, registering virtual memory/compute extended resources on the node;mockDevicePlugin.image.tag=latest: latest is mandatory. Version 1.0.1 cannot parse the new vnpus config format in the current chart and crashes outright with cannot unmarshal !!map into []ascend.VNPUConfig—a pit I fell into personally.The mock plugin is designed as a three-piece set: a device-config block (already shipped in the chart), a count extended resource on the node (the health gate—only needs to be greater than 0), and the hami.io/node-nvidia-register annotation (describing the fake cards). The latter two must be provided by hand:
NODE=hami-mutex2-control-plane
# Health gate: nvidia.com/gpu = 2 cards x 10 slices
kubectl patch node $NODE --subresource=status --type=json -p '[
{"op": "add", "path": "/status/capacity/nvidia.com~1gpu", "value": "20"}
]'
# Two fake T4s: GPU-MOCK-A and GPU-MOCK-B, 15360 MiB each
kubectl annotate node $NODE --overwrite \
'hami.io/node-nvidia-register=[
{"id":"GPU-MOCK-A","count":10,"devmem":15360,"devcore":100,
"type":"NVIDIA-Tesla-T4","health":true,"numa":0,"mode":"hami-core"},
{"id":"GPU-MOCK-B","index":1,"count":10,"devmem":15360,"devcore":100,
"type":"NVIDIA-Tesla-T4","health":true,"numa":0,"mode":"hami-core"}
]'Wait about 30 seconds, then confirm the mock plugin finished registering resources:
kubectl get node $NODE -o jsonpath='{.status.allocatable}' | python3 -c "
import json, sys
alloc = json.load(sys.stdin)
nvidia = {k: v for k, v in alloc.items() if 'nvidia' in k}
print(json.dumps(nvidia, indent=2))
"{
"nvidia.com/gpu": "20",
"nvidia.com/gpucores": "200",
"nvidia.com/gpumem": "30720",
"nvidia.com/gpumem-percentage": "200"
}Both cards are ready. All test Pods use the same template: request 1 vGPU and 1000 MiB of memory. The environment preloads every image, but note that imagePullPolicy: IfNotPresent is still worth writing explicitly, to avoid depending on the registry at runtime:
cat > plain-pod.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: plain-N
spec:
restartPolicy: Never
containers:
- name: app
image: docker.io/projecthami/hami:45b3d46
imagePullPolicy: IfNotPresent
command: ["sh", "-c", "sleep 3600"]
resources:
limits:
nvidia.com/gpu: 1
nvidia.com/gpumem: 1000
EOFThe single tool for observing placement is this command—the HAMi scheduler writes the chosen card onto a Pod annotation:
kubectl get pods -o custom-columns=\
'POD:.metadata.name,'\
'CARD:.metadata.annotations.hami\.io/vgpu-devices-allocated'Step one: deploy two plain Pods with no policy annotations at all (sed renames them):
sed 's/plain-N/plain-1/' plain-pod.yaml | kubectl apply -f -
sed 's/plain-N/plain-2/' plain-pod.yaml | kubectl apply -f -
kubectl wait --for=condition=Ready pod/plain-1 pod/plain-2 --timeout=3mPOD CARD
plain-1 GPU-MOCK-B,NVIDIA,1000,0:;
plain-2 GPU-MOCK-A,NVIDIA,1000,0:;The default spread policy puts the two plain Pods on separate cards. At this moment the cluster contains not a single mutex Pod. Submit one:
cat > mutex-pod.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: mutex-1
annotations:
hami.io/gpu-scheduler-policy: "mutex"
spec:
restartPolicy: Never
containers:
- name: app
image: docker.io/projecthami/hami:45b3d46
imagePullPolicy: IfNotPresent
command: ["sh", "-c", "sleep 3600"]
resources:
limits:
nvidia.com/gpu: 1
nvidia.com/gpumem: 1000
EOF
kubectl apply -f mutex-pod.yaml
sleep 20
kubectl get pod mutex-1
kubectl describe pod mutex-1 | sed -n '/Events:/,$p' | tail -3NAME READY STATUS RESTARTS AGE
mutex-1 0/1 Pending 0 20s
Warning FailedScheduling 20s hami-scheduler
0/1 nodes are available: 1 2/2 ExclusiveDeviceAllocateConflict.
no new claims to deallocate,
preemption: 0/1 nodes are available: 1
No preemption victims found for incoming pod.2/2 ExclusiveDeviceAllocateConflict: both cards rejected. If Reading B (mutex-only exclusion) held, there would be no mutex Pod on either card right now, and this Pod should schedule immediately. It did not.
Step two: delete plain-1 to free GPU-MOCK-B, and watch where the mutex Pod goes:
kubectl delete pod plain-1 --force --grace-period=0
kubectl wait --for=condition=PodScheduled pod/mutex-1 --timeout=2mPOD CARD
mutex-1 GPU-MOCK-B,NVIDIA,1000,0:;
plain-2 GPU-MOCK-A,NVIDIA,1000,0:;The mutex Pod landed on the card that had been completely emptied, while plain-2 was still running on the other card. Zero-tenant semantics confirmed.
Clear all Pods, then submit three mutex Pods in a row:
kubectl delete pod --all --force --grace-period=0
kubectl apply -f mutex-pod.yaml
sed 's/name: mutex-1/name: mutex-2/' mutex-pod.yaml | kubectl apply -f -
sed 's/name: mutex-1/name: mutex-3/' mutex-pod.yaml | kubectl apply -f -POD PHASE CARD
mutex-1 Running GPU-MOCK-B,NVIDIA,1000,0:;
mutex-2 Pending GPU-MOCK-A,NVIDIA,1000,0:;
mutex-3 Pending <none>The first two mutex Pods take the two cards; the third is rejected with 2/2 ExclusiveDeviceAllocateConflict. Both readings predict the same outcome here, so this serves as the control group. As a side note, mutex-2 briefly emitted a node lock contention event before succeeding on retry—a known HAMi node-lock contention behavior that does not affect semantics.
One last question: once a mutex Pod has taken a card, is that card exclusive from then on? With mutex-1 running on GPU-MOCK-B, submit a plain sharing Pod annotated binpack:
POD CARD
binpack-1 GPU-MOCK-B,NVIDIA,1000,0:;
mutex-1 GPU-MOCK-B,NVIDIA,1000,0:;
mutex-2 GPU-MOCK-A,NVIDIA,1000,0:;The binpack Pod landed on the very same card the mutex Pod was using. Exclusivity applies only at the mutex Pod’s own placement moment and is one-way: it demands an empty target card at placement time, but does not reserve the card afterward.
All three experiments agree fully with the code, the original requirement, and the observations on real GPUs. HAMi’s mutex semantics fit in two sentences:
Practical advice for users: if your goal is “this workload owns the whole card until it exits,” mutex only gets you halfway. Either request the card’s full resources (saturate both memory and compute), or pin the card with nvidia.com/use-gpuuuid; both are also covered in the official HAMi blog.
Looking back at the motivation and the mechanism, the value of mutex is not any isolation technology per se, but that it adds the “I want to live alone” option to the soft-slicing world, hooked into the policy chain through the lightest possible implementation—a filter. Placed back on the three-dimension map from the beginning of this post, mutex occupies just one cell, “card-level exclusivity”; but precisely because policies compose, that cell can stack with packing preference and NUMA affinity in the same chain, each evolving on its own. The methodology for judging its behavior matters just as much: when documentation wording diverges, construct a minimal scenario where the two readings predict different outcomes, reproduce it locally with mock devices, and the verdict arrives within minutes. The same method can extend to verifying the ranking behavior of binpack,numa chains, the fallback rules of combined policies, and more—extensions welcome.
kubectl delete pod --all --force --grace-period=0
helm uninstall hami -n kube-system
kind delete cluster --name hami-mutex2This post started from the question “why do GPU scheduling at all,” broke out the three scheduling dimensions—node, card, topology—and showed how HAMi v2.10 covers them with a policy chain of filters plus sort keys. It then zoomed into mutex, the only policy with exclusive semantics: using kind plus mock-device-plugin, we built a GPU-less local environment and settled its semantic dispute through three experiments—it demands a zero-tenant target card (not merely exclusion from other mutex Pods), and its exclusivity is one-way, binding only at placement time. The experiments use the same v2.10.0 release-candidate build as official Lab 14; every command and output comes from actual runs and is fully reproducible. The post also records three real-world pitfalls: mock plugin 1.0.1 cannot parse the new config format and requires the latest tag, the latest image should be explicitly IfNotPresent, and preloading images onto the kind node is essential.
2026-08-15 19:56:19
For GPU sharing to become a governable infrastructure capability, what has been missing was never partitioning—it was a verifiable correspondence between scheduling decisions and runtime isolation.
Since the beginning of this year, my writing has followed a single thread: how GPUs evolve from exclusive, scarce hardware into a governable infrastructure capability. That thread has passed through several dimensions. Early in the year, when discussing open GPU scheduling, the focus was on DRA, CDI, and the structural questions of standards and lock-in avoidance. Later, HAMi v2.9 showed Kubernetes becoming the GPU control plane, bringing the debate over resource models and control planes to the surface. Then came two deep dives: one toward observability, from GPU to token in eight layers; one toward measurement, with GPU utilization “failing” us and the notion of Productive GPU-Hours. Last month, in the article written after HAMi joined CNCF Incubating, the topic was consensus: when there is more than one technical route, how do a community and its ecosystem make the choice?
Put these dimensions together and a clear structure emerges: scheduling decides how resources are divided, observability makes them visible, measurement makes them accountable, and consensus decides who has the final say. But one dimension has remained hanging: enforcement. Once resources are allocated, what stops a workload from exceeding its share at runtime? Governance without enforcement is merely advice. This is precisely why multi-tenant users dare not share GPUs.
The adoption of HAMi-core by the NVIDIA KAI Scheduler lands squarely on this gap. This article discusses how the enforcement link is being closed, and what that means for the entire thread of compute governance.
To understand GPU governance, first decompose the GPU into a layered stack from the Kubernetes perspective. A GPU is not a single resource; it spans at least five layers: scheduling and allocation, Kubernetes device resources, container device injection, node accelerator software, and physical hardware. The essence of governance is to give every layer a clear, single owner. This layering is not tied to any one vendor: swap NVIDIA for Ascend or AMD and the structure holds—only the concrete components in each layer change.
The five-layer GPU software stack, from scheduling and allocation down to physical hardware:
Taking the most representative ecosystem, NVIDIA, as an example: the GPU Operator covers the middle three layers, turning a GPU-equipped node into a standardized container runtime node; Ascend’s NPU suite and AMD’s GPU Operator play the same role in their respective ecosystems. But this layer does not decide which queue a job should enter or whether gang scheduling applies, nor does it enforce memory caps at runtime—the same holds for NVIDIA’s CUDA layer as for Ascend’s CANN layer. Mixing layers together in discussion is the most common conceptual mistake in GPU governance.
Within this stack, the point where GPU sharing has long been stuck is clear: the scheduling layer can allocate, the device resource layer can declare, but nothing enforced those allocations at runtime. A container declares how much memory it needs, the scheduler accounts for it accordingly, yet the runtime cannot stop the container from blowing past its declaration and consuming the entire card’s memory.
The result is that multi-tenant users dare not truly share GPUs, and expensive accelerators can only be consumed exclusively. This is not a defect of any particular scheduler; it is a missing “isolated execution” across the entire chain, along with the observability to match. GPU sharing has therefore long stopped at “allocatable” without reaching “governable”.
In practice, the most common confusion is treating GPU Operator, KAI Scheduler, and HAMi as three comparable products. In reality their responsibilities barely overlap—each owns a different layer of the stack. The following uses the NVIDIA ecosystem as the example, but the same ownership structure holds for Ascend, AMD, and other vendors.
| System | Layer owned | Role in GPU sharing |
|---|---|---|
| GPU Operator | Node GPU software lifecycle (Driver, Container Toolkit, CDI, DCGM) | Makes the node a usable NVIDIA container runtime node |
| KAI Scheduler | Scheduling policy, queues, fairness, and shared accounting | Decides how GPUs are allocated and shared |
| HAMi Core | Runtime memory and compute isolation | Turns “cooperative accounting” into “runtime enforcement” |
There is a governance principle here that is easy to overlook: a node must have one and only one owner of its device resources. The real risk is not a missing component, but two systems each believing they own the same layer. For example, if the NVIDIA Device Plugin and the HAMi Device Plugin both register nvidia.com/gpu on the same node, they conflict outright. Once ownership is clearly drawn, these three systems are not mutually exclusive—instead they form an AI infrastructure stack with clean boundaries.
HAMi-core entering the NVIDIA KAI Scheduler closes exactly the missing link. There is a technical fact often overlooked: KAI Scheduler’s GPU sharing does only cooperative accounting by default—the official documentation states plainly that it does not enforce memory limits, nor does it isolate the memory usage of different processes. What HAMi Core does is turn that “cooperative accounting” into runtime enforcement.
The scheduler makes allocation decisions; HAMi Core intercepts CUDA calls at runtime to enforce memory limits—for the first time, the two can be reconciled. I introduced this integration in the HAMi community here; I later verified it on GKE—allocations within quota run normally, requests beyond quota are rejected outright, and the reproduction steps are documented as a complete lab. Sharing thus shifts from a trust-based gentlemen’s agreement to a verifiable contract.
The four-layer verification of GPU sharing: after scheduling, device allocation, and visibility are all correct, isolation enforcement is the final layer:
A common pitfall: seeing the container display only its partitioned memory via nvidia-smi (npu-smi on Ascend) and concluding that isolation works. Visibility is not enforcement. What is genuinely worth doing is a negative test: have the workload actively request more memory than its quota, and observe whether the request is rejected. Only an allocation failure (CUDA OOM) proves that isolation actually takes effect at runtime.
This is also why I place this layer in cluster acceptance criteria, rather than stopping at a one-off visibility check. Correct scheduling, correct device allocation, and correct visibility do not add up to correct enforcement—and enforcement is precisely the layer HAMi Core completes.
What carries more structural significance is NVIDIA’s choice: KAI Scheduler (which originated as Run:ai and now belongs to NVIDIA) did not build isolation in-house, but adopted HAMi-core directly. The significance is not that an open source project won an endorsement—it is that a path has been confirmed: GPU isolation and governance can exist as an independent software layer, without depending entirely on hardware virtualization such as MIG or SR-IOV.
For heterogeneous compute governance, this point is especially critical. Hardware virtualization capabilities vary wildly across vendors—some support fine-grained partitioning, others barely at all; a unified software isolation layer is the prerequisite for cross-vendor governance to exist at all. Over a longer horizon, the resource plane (Device Plugin, DRA) is gradually separating from the injection plane (CDI), and devices are evolving from a simple integer extended resource toward a model with attributes, declarations, and dynamic allocation.
The resource plane and injection plane are gradually separating, as devices evolve toward an attributable, declarative, dynamically allocatable model:
This direction is the continuation of the “GPU control plane” I proposed when discussing HAMi v2.9.
“Governable” does not stop at isolation. For a resource to be governed, it must also be observable, operable, and manageable uniformly across heterogeneous environments. This is precisely what HAMi 2.10, scheduled for release on August 21, focuses on completing:
This main thread itself also enters 2.10: through the standalone KAI Resource Isolator, the division of labor between scheduling and isolation is being formalized as a product. It should be noted that the relevant isolation capabilities are still maturing—actual enforcement of memory limits and directory permissions for non-root containers still have PRs pending before release—so the current stage is better described as “rapidly maturing” rather than “fully usable”.
Taken together, the emphasis of 2.10 is pushing GPU sharing from “usable” toward “observable, operable, and uniformly governable across heterogeneous clusters”—consistent with the governance closed loop discussed earlier.
Back to the starting point of this thread: this year, from open scheduling to the control plane, observability, efficiency measurement, and community consensus, every dimension of compute governance has been discussed—except enforcement, which remained hanging. HAMi-core entering the NVIDIA KAI Scheduler closes exactly that link: scheduling decisions and runtime isolation can, for the first time, be verifiably reconciled. Viewed as a layered stack, governance means clear ownership for every layer: GPU Operator owns the node software lifecycle, KAI owns scheduling and shared accounting, and HAMi Core owns runtime enforcement. And software isolation being adopted by a mainstream scheduler means GPU governance is shifting from reliance on hardware virtualization toward a software layer reusable across vendors. Isolation is only the starting point of governance; observability and heterogeneous unification come next, and HAMi 2.10 is advancing along exactly that direction. The road from scarce hardware to governable infrastructure capability is becoming concrete.
2026-07-08 21:51:44
After HAMi became a CNCF Incubating project, I want to talk about something overlooked: AI is shifting the scarce resource of open source communities from code to consensus.
On July 2, 2026, HAMi officially became a CNCF Incubating project (see the announcement). For an open source project, this means more than recognition of its technical capability; it means that community governance, ecosystem building, and real-world adoption have all entered a new phase.
But if you only read HAMi’s growth as “a GPU virtualization project succeeded,” you might miss the more important shift.
My time building the HAMi community has left me with one increasingly strong feeling: AI is changing how open source communities produce. As AI coding drives the cost of producing code lower and lower, the core of competition in open source will no longer be who wrote the most code, but who can build stronger technical consensus, attract more contributors, and form a sustainable ecosystem network.
This article is my attempt to lay out that argument clearly.
Let’s start with the data. Here is where HAMi stands today:
| Metric | Value |
|---|---|
| GitHub Stars | 3,700+ |
| Contributors | Nearly 500, from 27 countries |
| Participating organizations | Multiple, and growing |
| Release cadence | Once every three months |
I’m not listing these numbers to show off “growth metrics.” I’m making a different point: an open source project is shifting from “software maintained by a team” into “a technical community of people gathered around a shared goal.”
That distinction matters. Software can be forked, rewritten, or generated overnight by AI. But a community with a shared goal, trust, and rhythm cannot be forked. That is the irreplaceable asset of an open source project.
From a governance-maturity perspective, HAMi has passed through three milestones:
Note the middle stretch: from entering Sandbox in August 2024 to reaching Incubating in July 2026, roughly two years. During those two years the code certainly grew, but what actually convinced the CNCF Technical Oversight Committee (TOC) was the diversification of the community, the formalization of governance, and real production adoption. None of that is something you produce by writing code.
HAMi was first open-sourced in 2021. Back then, most developers did not see AI coding the way we do today. Whether an open source project survived depended on developers genuinely investing their time, discussing problems in issues, submitting code through pull requests, and building trust through code review.
Today, the environment has changed.
In a recent HAMi community livestream (Mastering HAMi DRA, Yang Shouren, HAMi Community Livestream Episode 2), someone asked the maintainers a question: “How much of HAMi’s code now comes from AI assistance?”
The answer from Yang Shouren stuck with me: about half of the code in the HAMi community today is already AI-assisted.
Half. And that share is still rising.
This immediately raises a sharp question: if AI can write more and more code, what is the value of an open source community? Could one person plus a few AI agents just fork a “new HAMi”?
My answer is no, because what AI lowers is the cost of producing code, not the cost of building technical consensus.
Let me sharpen that point.
An AI agent can already do a lot today: write code, fix bugs, add tests, generate docs, produce migration scripts. These capabilities are getting stronger fast. But there are a few things in a community that AI cannot replace today, and in my judgment will not replace soon.
First, deciding which problems are worth solving.
Take HAMi. Why does GPU sharing matter? Not because “slicing cards finely” is a cool technique. It matters because once AI infrastructure scales, reality looks like this: GPU costs are enormous, heterogeneous hardware keeps multiplying, and Kubernetes’ native resource model is no longer enough.
The community has to first agree that “this problem is worth investing in” before anyone writes any code. That agreement is a human-to-human matter, supported by real scenarios, real costs, and real pain. AI can solve a problem you have already defined, but “which problem is worth defining” is decided by community consensus.
Second, choosing a technical path.
In GPU virtualization there are many paths: MIG, MPS, time-slicing, vGPU, DRA. Each has trade-offs, and choosing wrong can cost you two years of detours.
Code can be generated, but architectural choice is fundamentally a value judgment. HAMi’s decision on Ascend 910C to move from hardware SR-IOV to userspace HAMi-core was not about someone writing a better piece of code; it was about the maintainers holding to a judgment that “hardware partitioning is too coarse, software partitioning is more flexible.” That kind of judgment is ground out through repeated discussion, failure, and validation in the community, not prompted out.
Third, trust.
Users don’t choose HAMi because of “how much AI-generated code is in this repo.” They care about: who maintains it? Who reviews it? Are there real production cases? Does the community respond when something breaks?
Each of these is a relationship between people, a product of community organization, not a product of code quality.
Put these three together, and the production model of open source communities in the AI era is shifting:
In the past, code came first and the community sedimented out of the code; in the future, consensus comes first, AI rapidly turns consensus into code, and code flows back to test the consensus. The center of gravity of the scarce resource moves from “code” on the left to “consensus” on the right.
Since AI has become a new category of contributor, a community’s governance rules have to keep up.
My advice is: don’t treat AI merely as a tool, treat it as a new type of contributor. It used to be “developers write code, humans review”; in the future it will be “humans set intent, AI generates code, the community reviews, and shared knowledge is distilled.” There is an extra layer in the middle, and an extra layer of governance complexity.
HAMi is already responding to this. Its CONTRIBUTING.md is explicit:
If you are using any kind of AI assistance to contribute to HAMi, it must be disclosed in the pull request.
In other words, if you use AI to help with a contribution, you must declare it in the PR. But the community also knows that a norm without a gate is not enough (see the discussion in Issue #1998), and there is already ongoing discussion about how to give that norm real enforcement.
This is actually a problem every AI-era open source project will run into. I’d break it into a few questions:
Whoever figures out and operationalizes these rules first will keep contribution quality stable in the AI era. HAMi’s exploration here is worth a look for every open source project.
Back to the promotion itself.
I’d lean against treating it as an “honor.” What CNCF Incubating really validates is not code quality, but whether a project has the capacity to become infrastructure. It examines a whole package: technical maturity, community governance, production adoption, and ecosystem building.
HAMi’s case, in one sentence, is not “a Chinese team built a GPU project.” It is this:
An AI Infrastructure community, jointly shaped by developers from around the world, is taking shape.
The first is a product story; the second is an ecosystem story. The Incubating recognition from the CNCF is recognizing the latter, because competition over infrastructure is never competition between individual products; it is competition between ecosystem networks.
The open source competition of the next decade will not be just a competition of code, but a competition of communities.
Once AI gives everyone near-infinite capacity to produce code, the truly scarce capabilities will be three: finding the right problem, building technical consensus, and organizing developers worldwide to solve a problem together.
HAMi’s path to CNCF Incubating is just one snapshot of how open source communities are evolving in the AI era. Code will keep getting cheaper, and consensus will keep getting more expensive. Whoever understands this inversion will be the one who can build open source communities with real depth in the AI era.
jimmysong) or follow HAMi on GitHub to join the community focused on GPU virtualization and heterogeneous compute scheduling. Let’s talk about open source governance in the AI era.
2026-06-24 22:30:00
HAMi used to save cards in the cluster. Now it decides how good a desktop AI workstation feels.
A few days ago an old friend came by to chat. We used to run the cloud-native community scene together back home, so we go way back. She recently joined a company called Olares, and somewhere in the conversation she dropped this: their project had integrated HAMi.
I run the HAMi community day to day, so whenever I hear someone using it for something, I want to take a closer look. I went and dug through the Olares repo and website, and my first reaction was, huh, this is actually interesting.
Isn’t this exactly the kind of local AI workstation I’d been eyeing forever but never pulled the trigger on? I wrote in My Personal AI Stack that for someone like me who mainly works with my head, subscribing to models beats maintaining a high-end GPU. But how far can “one machine running the full AI stack” really go, and who has actually built it, I’ve wanted to see with my own eyes.
Olares isn’t a “NAS with a GPU bolted on,” and it isn’t a “mini PC with Ollama installed.” It’s more like a personal cloud built on Kubernetes: local models, apps, identity, remote access, dev environment, storage, even GPU governance, all packed into one machine as a desktop cloud OS.
HAMi isn’t filler here. It’s the layer that turns “one card” into “a resource pool you can share, isolate, and schedule.”
There’s no shortage of things calling themselves AI workstations, but most are just beefier mini PCs. Olares is different. It’s genuinely designed as a cloud.
The company positions this machine outright as a 24/7 personal AI cloud, not a PC you sit in front of. That single framing explains almost everything that follows: why it has to use Kubernetes, why it cares so much about network traversal and remote access, why GPU governance suddenly matters here.
Two other details tell you a lot: it supports Thunderbolt 5 external eGPUs, and two machines can cluster up. In other words, it was never a sealed box. It starts as a single node and grows upward.
Architecturally, what Olares does is compress an entire cloud runtime into a single machine: auth and authorization, app lifecycle, tunnels and traversal, secrets, observability middleware, not a layer skipped.
This isn’t “a few AI apps preinstalled.” It’s a complete cloud runtime stuffed into a desktop device.
The diagram below is my own re-drawn layering based on its public docs, with the HAMi layer pulled out explicitly because it’s the point of this whole piece.
One thing jumped out while I was reading: Olares’s docs haven’t kept up with its own product.
Its 2025 architecture page still says nvshare, noting that GPUs are limited to one card per node. But flip through the release notes and 1.12 already integrates the HAMi scheduler, with exclusive, time-slicing, and memory-slicing modes all there; 1.12.2 adds multi-GPU; 1.12.5 supports DGX Spark outright and rolls automatic scheduling across all three modes.
The product is outrunning its docs. That alone tells you something: it’s shifting from a “personal cloud OS” toward a “local AI cloud OS.” That’s also why I wanted to write a whole piece on it.
HAMi is a CNCF Sandbox project, positioned as a heterogeneous AI compute virtualization middleware. On the path there’s a Webhook, a scheduler, a Device Plugin, and HAMi-core handling in-container resource control. I summed it up in one line in Kubernetes Is Becoming the GPU Control Plane of the AI Era: it turns GPU slicing from “a hardware capability” into “a control-plane capability.”
On Olares, that’s the real thing behind the GPU mode toggle in settings. On the surface it’s a UI option; underneath, it’s a policy switch on the resource plane.
The most critical piece is almost certainly HAMi-core, which in one sentence: intercepts CUDA calls inside the container and does memory virtualization, compute throttling, and utilization monitoring. It doesn’t slice with hardware. It manages with software.
The diagram below puts that injection path and the three GPU modes side by side.
The evidence lines up too. In Olares’s GitHub issues you can see HAMi’s libvgpu.so crashing under WSL2, the discussion mentions it getting injected into every process via /etc/ld.so.preload, and the Olares team itself says this implementation “drew heavy inspiration” from the HAMi project.
One thing I should be clear about, though: whether Olares uses upstream HAMi-core directly or maintains a forked version, there’s no single official answer in public. I won’t fill that in for them.
The three modes make sense in order. Full-card exclusive is for heavy loads; time-slicing lets lightweight services take turns, and Olares even swaps inactive models into memory first, with roughly 5% switching overhead; memory-slicing cuts the memory into fixed quotas so multiple apps run together. On something like DGX Spark, where CPU and GPU share memory, it defaults to memory-slicing because there’s no traditional memory paging in and out to begin with.
HAMi used to live mostly in big clusters: multi-tenant, inference serving, mixed training-and-inference, heterogeneous cards. NIO running it for sharing across 80 nodes and 600 cards is the textbook example. That line is well-trodden.
What feels new about Olares is that it moves the same problem onto a single machine.
What you actually run at once on Olares is never just one Ollama: a local model, a chat UI, a research agent, an image or video pipeline, plus a pile of OCR, speech-to-text, and automation tools. In a scenario like that, without a GPU scheduling layer, the GPU always degenerates into “whoever starts first grabs it.”
So for HAMi this matters. It goes from “a tool that saves cards for platform engineers” to “something that directly decides whether an ordinary user has a good time.” In the cluster it manages utilization and cost; on Olares it manages concurrent experience, model switching, and whether this machine is actually pleasant to use. HAMi doesn’t only talk to platform engineers anymore.
My sense is, edge AI won’t settle at “a stronger local model box.” It’ll grow into a full stack of “control plane plus model plane plus application plane.”
NVIDIA pushing DGX Spark and RTX Spark onto the desktop has already moved the story from “run models locally” to “run agents locally.” Olares’s CUDA-plus-x86 is one path, DGX Spark’s unified memory is another, and below that there’s the DIY mini-PC-plus-eGPU route you assemble yourself. Interestingly, Olares lists eGPU and dual-node as supported paths itself, so the line between appliance and DIY isn’t that sharp.
But my guess is, the one that breaks out won’t be the one with the most ferocious specs. It’ll be the one that fuses resource governance, dev experience, app ecosystem, security, and remote access into one closed loop. If desktop AI workstations really become a category, what people compete on isn’t GPU and memory, it’s the control plane.
I went ahead and added Olares to the AI Native Landscape I maintain, so I can keep watching how it grows.
In one line: Olares is a desktop AI OS with Kubernetes at its core, and HAMi is the layer that turns its single-machine GPU into a shareable, isolatable, schedulable resource pool.
From cluster to desktop, HAMi’s story is expanding from “saving cards” to “making edge AI usable.” If desktop AI workstations become a real category, what people compete on is the control plane, not single-card performance.
