2026-11-09 18:00:00
We no longer ask “can Kubernetes run AI?” We ask: once AI is running, who manages it, and who is accountable for it?
My calendar has been packed with conferences lately. At the end of October I co-ran a HAMi GPU sharing workshop at ContainerDays Singapore 2026. For the upcoming COSCon'26 and KCD Hangzhou 2026, I reviewed part of the proposals. And tomorrow, November 10, KubeCon + CloudNativeCon North America 2026 opens in Salt Lake City.
Over the past decade I have attended many cloud native conferences and watched Kubernetes, service mesh, observability, GitOps, and platform engineering go from new ideas to the mainstream. But reading these agendas side by side this year, one feeling was stronger than ever: the cloud native stack we know is being handed a very different kind of work.
This post puts the four conferences together and shares what I see changing.
Start with the figure. Using the public agendas available on November 9, I roughly counted each conference’s focus by session title keywords.
ContainerDays Singapore brought the topic close to developers. Of 64 sessions, 22 had titles about agents and tooling: sandboxes for coding agents, fine-grained authorization for MCP tools, and how teams run evals, all on the same agenda.
KubeCon North America was the most direct. CNCF added a new AI Inference + Agentic track; 45 of the 174 sessions across the 11 topic tracks sit in it, about 26%. That doesn’t count DRA-based GPU sharing in the Platform Engineering track or AI-assisted troubleshooting in the Observability track.
COSCon'26 had the widest scope. Beyond agents, model distribution, and hardware isolation, it covers youth open source, RISC-V, and open source law. AI is not only reshaping infrastructure; it is seeping into open source collaboration, education, and community governance.
KCD Hangzhou was the most focused. Its three tracks, Agent Infra, AI Infra & LLM Serving, and AI Observability & Reliability, map almost exactly to the core questions of AI native infrastructure today: how agents run, how inference stays efficient, and how the system stays reliable. 22 of its 25 sessions had titles matching AI infrastructure keywords.
These numbers only describe agenda structure. They can’t rank conferences or measure year-over-year growth, since each event is positioned and categorized differently. But one thing is clear: AI has moved from being one application area at cloud native conferences to being the thread that organizes the agenda.
AI/ML already showed up at KubeCon in 2023, and in 2024 and 2025 we talked a lot about running training and inference on Kubernetes. In 2026 the questions go much deeper: which instance should an inference request go to? How do we split GPUs and memory? How do we reuse the KV cache? What isolation does an agent need when it executes code? Where does a task resume after an interruption?
Kubernetes’ original abstractions (Pod, Deployment, Service, HPA) were built around stateless microservices: replica counts, resource requests, service discovery, failure recovery. AI workloads bring different requirements. Scheduling must consider device topology, memory capacity, sharing, and isolation, not just CPU and memory. In inference serving, prefill and decode have different compute profiles, KV cache placement affects routing, and model load time directly determines cold-start latency. High GPU utilization does not mean throughput, latency, and cost are optimal.
That is why DRA, HAMi, Kueue, Volcano, KServe, llm-d, and vLLM are worth following. They live at different layers: some manage devices, some orchestrate workloads, some manage inference requests. They are not substitutes for each other; together they are exploring the new control boundaries of AI infrastructure. In the CNCF 2025 annual survey, 66% of the responding organizations that host AI workloads use Kubernetes for some or all of their inference. Reusing the cloud native foundation is already a fact.
Put simply, we used to schedule containers and compute; now we also have to understand the execution semantics of models, requests, caches, and tasks. Kubernetes is not being replaced. The infrastructure on top of it is being redesigned.
If LLM serving changes how resources are used, agents may change even more.
Imagine you are on call and a service suddenly slows down. You ask an agent to read the logs, diff recent changes, and suggest a fix. If it misses a deploy from twenty minutes ago, its conclusion goes off track. That context normally lives scattered across the release system, the service catalog, and team chat. One SRE talk at KubeCon North America says in its abstract that missing context was the first wall they hit when bringing AI into troubleshooting.
Go one step further and let it run the rollback, and the question is no longer “is the answer right?” Whose identity does it use? Which environments can it roll back? Will a timeout retry run the rollback twice? If its suggestion contradicts the on-call engineer, who decides?
This reminds me of early cloud native. When we first put apps in containers, we thought running was enough. Then we discovered we also needed service discovery, configuration, identity, and observability. Agents are going through something similar, except their behavior is more dynamic and has direct side effects on external systems. That is why agent sandboxes, checkpoint/restore, tool authorization, and task traces keep showing up on the Singapore and Hangzhou agendas.
I believe an increasingly important field will emerge over the next few years: Agentic Runtime, runtime infrastructure for executing agent tasks. It needs clear boundaries across the task lifecycle, environment isolation, permission control, state recovery, and observability. That said, I don’t think we should rebuild everything. Containers, sandboxes, identity, network policy, and controllers remain highly reusable. The key is telling apart the problems that already have answers from those that genuinely need new abstractions.
Observability is in the same position. A trace tells you which tools an agent called and where it failed, but not necessarily whether the task met the user’s goal. Healthy infrastructure and a successful task are no longer the same thing. Reliability will need SRE, observability, model evaluation, and task auditing working together.
The technology is changing, and so is how we work.
AI produces first drafts faster and faster, and many people feel they can do more at once. But if someone has to re-understand every change afterward, the team has only moved the queue from “waiting for someone to write” to “waiting for someone to review.” An agent can write tests for me, but I still need to know what they cover. It can produce a rollback command, but I still need to know which command affects users.
Open source maintainers feel this most concretely. The bar for opening a PR is dropping, but maintainers don’t get more hours. One PR with reproduction steps, a failing test, and a design rationale may be worth more than ten unverified generated PRs. We need to rethink how we measure a “useful contribution.”
The same goes for developer relations. At the Singapore workshop we had already designed the labs to run on a local kind cluster without real GPUs, yet some attendees still got stuck on dependency installs and OS differences. From the maintainer’s side the demo works; from the user’s side they may not have taken the first step. AI can generate more complete tutorials, but it can’t replace watching real developers use your project. I wrote up the details in What a GPU Workshop Reveals About the Developer Experience Gap in Cloud Native Infrastructure.
Looking at the four events together, different communities enter AI infrastructure from their own strengths. Singapore sits close to developer tooling and security. KubeCon North America extends Kubernetes through standards work such as DRA, the Gateway API Inference Extension, and the OpenTelemetry GenAI semantic conventions. The Chinese community shows deep low-level work on heterogeneous GPU compute, inference optimization, and agent sandboxes, and is actively contributing to these projects.
I don’t want to rank regions based on a few agendas. The more important question is whether these efforts can flow into one open ecosystem. A key reason cloud native succeeded over the past decade is that developers worldwide collaborated openly around Kubernetes and CNCF. If every model platform demands its own runtime, tool protocol, and resource management stack, AI could repeat the fragmentation of early cloud computing.
First, how inference teams measure the compute they save. GPU utilization is easy to show; I would rather see latency, failed retries, and cost measured together on the same real workload.
Second, whether agents can move from read-only checks to narrowly scoped write operations while leaving records someone else can pick up.
Third, whether open source projects start tracking the cost of verification and maintenance, for example how long it really takes from receiving a change to completing a meaningful review.
The four conferences send the same signal: cloud native is not obsolete, but the workloads it carries have changed. The focus is shifting from “deploying AI on Kubernetes” to “running and managing AI”: device allocation, inference routing, agent permissions and recovery, and task-level observability and evaluation are each growing new problems.
For engineers, the change is not only architectural. Tasks can be handed to agents, but acceptance criteria, permission boundaries, and incident handling must move from personal experience into agreements the whole team can inspect. A year from now, the simplest test of whether an AI workflow has really entered a team may still be the handoff: when the person who owns it is on vacation, can someone else read the records, understand what happened, and know at which step to stop it?
2026-10-29 19:06:46
Haze on arrival, and a cracked “Garden City” illusion: a fitting metaphor for where cloud native finds itself.
This was my first time in Singapore.
Before the trip, my mental image of the city was clean, green, and efficient, maybe even too perfect. What greeted me on arrival was an overcast sky. Friends there told me the haze had hung around for days, and the legendary “Garden City” filter cracked a little.
Of course, that is not necessarily Singapore’s own doing. Air does not stop at national borders, and environmental impact is never isolated. Thinking about it later, it works as a metaphor: whether it is the natural environment, a commercial ecosystem, or an open source technology ecosystem, we all live inside interdependent systems where nobody can truly stay clean on their own.
That feeling only grew stronger at ContainerDays Singapore 2026: the cloud native community is going through something very similar.
ContainerDays took place at Suntec City. The venue infrastructure is mature, other events were running at the same time, and people from different countries and industries gathered there to meet, exchange ideas, and look for partnerships. Singapore really is a comfortable place to hold an international conference and do business.
I also caught up with old friends and met new ones, including my colleague Reza Jelveh, Ivan Porta, Saiyam Pathak, Jerry Chen from DaoCloud, Sagar Utekar, Raymond Chong from Singtel, and Dharmjit Singh, plus Yan Wang, who works in blockchain. I also ran into Yang Dongdong, a solutions architect at AWS who had transferred from Beijing to Singapore just a month earlier.
Talking with them gave me a more concrete feel for Singapore’s tech ecosystem. Many Chinese tech companies pick Singapore as their first stop when going global: an international business environment, a relatively mature legal and financial system, and a geographic position that connects to the Southeast Asian market. For engineers, the job market and the international working environment are attractive too.
From these limited conversations, though, my sense is that Singapore’s density of deep infrastructure engineering talent, especially in Platform Engineering, may not match its standing as a regional business hub. It reminded me of Hong Kong: strong commercial connectivity, but not necessarily the place where fundamental technology innovation originates. That is a personal observation from a short trip, not a verdict on the market.
Still, it raises a question worth asking: is a region’s technology ecosystem driven by business opportunity, or by its engineer community?
The two are not in conflict, but they produce different ecosystems. Business opportunities attract companies and capital; sustained technical innovation also needs long-accumulated engineering practice, talent mobility, and a developer community willing to share and collaborate. For Chinese open source companies going global, Singapore may be an excellent commercial starting point, but building real technical influence takes more than that.
My strongest impression from the conference was not any single dazzling new technology. It was that the whole cloud native community is searching for a new direction.
Over the past decade, containers, Kubernetes, service mesh, GitOps, and observability gradually matured, and cloud native went from a technical revolution to the default state of modern software infrastructure. That is a success, but success brings its own anxiety: once Kubernetes is just part of the infrastructure, once more and more capabilities are absorbed by cloud providers and managed services, how do the open source projects, startups, and developer communities built on top of those technologies keep proving their value?
AI looks like the obvious answer. Some people are building AI infrastructure, some work on GPU scheduling and utilization, some have moved to agent sandboxes and AI agent runtimes, and some are extending existing Platform Engineering systems to AI workloads.
I do not think this is simple hype-chasing. The deeper reason is that cloud native has solved much of the previous generation’s infrastructure problems, while AI is redefining the application itself.
Traditional cloud native applications usually have a well-defined execution model, lifecycle, and resource model. AI agents do not: they call tools dynamically, execute code, reach external systems, and may need isolated environments, persistent state, and elastic compute. Scheduling, security, isolation, observability, and resource management, the problems we know well, are coming back in new shapes.
Seen this way, cloud native is not obsolete. Much of what it built over the past decade may become the foundation of AI-native infrastructure. But one caution: slapping an AI label on an existing project is not a transformation. What matters is understanding what is genuinely different about the new workloads, and which assumptions in current architectures no longer hold.
Another observation from the floor: compared with pure low-level infrastructure topics, sessions that leaned toward AI agent applications and showed working results drew noticeably more attention.
That is easy to understand. For most developers, infrastructure is not the goal; what they can build with it is the real question. When cloud native was growing fastest, we discussed how to deploy applications, manage microservices, and improve delivery efficiency. Now developers care about building agents, letting agents execute tasks safely, and getting these systems into production.
That poses a challenge to the infrastructure community: if we keep telling our story purely in terms of technical components, it gets harder to connect with the broader developer population. The value of infrastructure has to be re-seen through the applications it enables.
This does not mean the underlying technology matters less; it means the narrative has to change. A GPU scheduling system cannot only talk about scheduling algorithms; it has to answer how it lowers the cost of AI workloads. A sandbox project cannot only explain its isolation mechanism; it has to explain why agents need it and how it affects security and developer experience. I suspect the most important connection point between the cloud native and AI communities over the next few years is right here.
Reza Jelveh and I also ran a hands-on HAMi workshop on GPU sharing and scheduling, done entirely on a local kind cluster with no real GPU required. Quite a few developers interested in GPU resource management showed up, and several participants finished all the labs, which was great to see. But the session also exposed gaps in our preparation.
The biggest problem was the environment. We did not ask participants to register and prepare their setups in advance, so we burned a lot of time installing dependencies on site. The install path was also not sufficiently validated across operating systems, especially Windows.
These problems look trivial, but they define the workshop experience. For the instructor, environment setup is a preliminary step; for the participant, it can decide whether they ever reach the actual technical topic. Next time we should collect participants’ system setups and backgrounds through a form beforehand, send clear preparation instructions, and validate the installation flow per OS, keeping on-site time for labs, discussion, and understanding the mechanics.
It also made me rethink a question in developer advocacy: are we demonstrating how powerful a project is, or helping people actually gain the ability to use it?
There is a long distance between the two. For open source infrastructure projects especially, the quality of documentation, installation experience, examples, and workshops often does more than a brilliant talk to decide whether a developer keeps using the project.
At international conferences, I have always valued the conversations. Talk content can be recovered from recordings, slides, and blog posts, but hallway chats, discussions that unfold over coffee breaks, and catching up with old friends are almost impossible to capture after the fact.
Talking with friends from different companies and countries, you feel how much the underlying concerns have in common: some are figuring out how to keep an open source project sustainable, some are searching for a new product position in the AI era, some are adapting to new markets and jobs, and some keep solving infrastructure problems that never trend but never go away in production.
These conversations remind me that a technology ecosystem is not decided by a trend chart. It is made of specific people, and their choices, collaborations, and long-term commitment. A healthy open source community should not only chase the next hot concept; it also needs room for the people willing to polish foundational technology over the long run.
Leaving Singapore, I thought again about the overcast sky I saw on arrival. A city can have efficient infrastructure and an excellent business environment and still not fully control the air around it. Technology ecosystems are the same: the cloud native community cannot stand outside the AI wave, and there is no reason to.
But the question is not only how to connect Kubernetes, GPU scheduling, or sandboxes to AI. It is how to keep building genuinely valuable infrastructure through this new round of change.
Over the past decade, cloud native changed how software is built and run. Over the next decade, AI agents may further change how software executes, and the relationship between people and software. These two shifts are not separate. For those of us who have long worked on cloud native and open source infrastructure, this is a moment to be excited and to stay clear-eyed at the same time.
Technology names will change, hot topics will rotate, communities will reorganize, but the questions that really matter, how to use resources more efficiently, how to build reliable systems, how to reduce complexity for developers, how to drive innovation through open collaboration, have not gone away. Instead of rushing to find cloud native a new label, we should think hard about the irreplaceable value it can offer the next generation of software.
That is probably the deepest impression ContainerDays Singapore left me with.
2026-10-23 18:28:11
From Kubernetes and GPUs to MCP and agent runtimes, rethinking infrastructure and the open-source ecosystem for the AI era.
While following the progress of the Agentic AI Foundation (AAIF), I noticed something interesting: many friends I have known from the cloud native community over the past decade are moving into AI.
Some former colleagues have joined OpenAI; others are building AI gateways, MCP servers, and agent frameworks. Developers who used to work on Kubernetes, DevOps, and platform engineering are shifting toward GPUs, model inference, and agent runtimes.
I have gone through the same transition myself. From Kubernetes, CNCF, and the Chinese cloud native community, to five years at Tetrate, and now a focus on HAMi, GPU resource governance, and AI native infrastructure, I have kept asking the same question: how do we make complex systems run more efficiently and reliably?
But faced with the endless stream of new concepts like AI Gateway, MCP Gateway, Agent Runtime, and Sandbox, I keep coming back to one question:
Do the cloud native foundations we spent more than a decade building really need to be reinvented for the AI era?
Looking back at how cloud native evolved, Docker, Kubernetes, and the open-source ecosystem around CNCF solved a long list of problems: application delivery, resource orchestration, elastic scaling, failure recovery, security isolation, and observability.
None of these capabilities become obsolete in the AI era.
Model inference services still need scheduling and autoscaling; AI platforms still need identity, security, and multi-tenant management; agent applications still need reliability and observability.
What actually changes is the characteristics of the workloads.
Traditional web services care mostly about request throughput and latency, while LLM inference also has to account for GPU memory, KV cache, time to first token (TTFT), and inter-token latency (ITL). Traditional applications follow relatively deterministic execution paths, while agents can plan tasks dynamically, call tools, and even generate code.
AI native needs new infrastructure abstractions, but that does not mean a decade of engineering experience should be thrown away.
In my view, AI native is not a replacement for cloud native. It is an infrastructure refactor centered on AI workloads.
Whether Kubernetes fits AI workloads has long been debated.
Supporters point to its mature declarative APIs, controllers, and resource orchestration; critics point out that it was never designed for GPUs, distributed training, or large-scale inference.
I think the real question is not whether to use Kubernetes, but which capabilities can be reused and which abstractions need to change.
Take HAMi, a project I am currently involved in. Traditional Kubernetes GPU management mostly allocates whole cards, which easily wastes resources for small inference and development tasks.
HAMi extends Kubernetes device management and scheduling to support GPU sharing, resource allocation, and isolation, rather than reimplementing a container orchestration system.
But GPU sharing is not only about utilization. You also have to distinguish resource isolation, fault isolation, and security isolation. A working memory quota does not guarantee that one failing task will not affect the others.
That is exactly the new challenge AI workloads pose to traditional resource management.
And higher GPU utilization alone does not mean lower business cost. If overcommitment drives up inference latency, fails agent tasks, or forces repeated retries, the final bill can be even higher.
The optimization target for AI infrastructure should move beyond GPU utilization toward cost, reliability, and service quality per successful task.
In September 2026, Agent Router, driven by Tetrate, joined AAIF. Before that, agentgateway from Solo.io had also joined the foundation.
Seeing this news hit home for me. Cloud native infrastructure companies I knew well are meeting again in the AI native space.
That is no surprise.
Model routing, load balancing, authentication, rate limiting, security policy, and observability have always been problems that API gateways and cloud native infrastructure are good at solving.
But agents bring new challenges.
A model call can be billed by token; an MCP tool call may involve user authorization; an agent may dynamically choose different tools mid-flight and even perform side-effecting operations.
So an AI gateway should reuse proven traffic-handling machinery like Envoy, but it must also understand the new semantics of models, tools, and agents.
The question is: does an enterprise need one unified AI gateway, or separate model gateways, MCP gateways, and agent collaboration gateways?
That is not a product naming question. It is a question of where infrastructure responsibilities should be drawn.
The evolution of MCP reminds me of the path cloud native infrastructure once took.
MCP lowers the cost of connecting AI applications to tools and data sources through a unified protocol. But once MCP servers enter production, problems such as scalability, authorization, load balancing, failure recovery, and auditing appear.
MCP moving toward a stateless protocol core in 2026 proves the point: the protocol can depend less on connection sessions, but business state, authorization context, and task lifecycle still need explicit management.
These are all familiar distributed systems problems.
Agent runtimes, however, do need some genuinely new abstractions.
A traditional Kubernetes Pod can host an agent process, but one agent task may span multiple model calls, execution environments, and tools, and may need to run for a long time, pause, and resume.
The lifecycle of a Pod is not the lifecycle of an agent task.
Likewise, containers and sandboxes provide execution isolation, but they do not by themselves solve delegated permissions, tool authorization, or approval for sensitive operations.
So my view is:
An agent runtime does not need to reinvent the container, but it does need a new task, state, and security model built on top of existing infrastructure.
At the same time, AI native security design cannot focus only on prompt injection. It also needs explicit resource isolation, fault isolation, and security isolation, so that even when an agent makes a mistake, the blast radius stays bounded.
Beyond the technology, the open-source ecosystem deserves even more attention.
Over the past decade, the value of CNCF was not just hosting Kubernetes. It pushed vendors to collaborate around open standards and gradually built a complete cloud native ecosystem.
Today, AAIF is trying to build a similar foundation for agentic AI. Projects around MCP, A2A (Agent2Agent), AI gateways, and developer tooling are exploring protocol interoperability and neutral governance.
But I do not think we can conclude yet whether AAIF will become the next CNCF.
It has to answer a deeper question first: with CNCF, LF AI & Data, and other foundations already in place, why does agentic AI need a new foundation? And how do we avoid duplicating work across foundations, protocols, and projects?
Membership and project counts are not sufficient proof of success. What matters is cross-vendor interoperability, production adoption, open governance, and whether developers can keep participating in technical decisions.
As AI makes code generation easier and easier, one shift is worth noting:
The scarcest value in open-source communities may be shifting from code itself to consensus, trust, and long-term collaboration.
That is why I am paying close attention to AAIF.
Over the past decade, I have met many developers through CNCF, the Chinese cloud native community, and Tetrate.
Some have joined AI companies like OpenAI; some are building agents, MCP, and AI gateways; and many former Kubernetes, platform engineering, and infrastructure developers are now working on inference services, GPUs, and AI platforms.
This is not just people chasing the next hot thing.
What matters more is that AI is generating a large number of engineering problems that look a lot like cloud native’s, yet with new characteristics.
People who know distributed systems need to learn how models and agents actually run, while AI developers need to learn reliability, security isolation, resource management, and production operations.
The most competitive engineers of the next era will probably not be those who know only cloud native or only AI, but those who understand where the two meet and can combine them.
I also hope this cross-domain exchange happens in open-source communities.
Back to the original question: do we really need to reinvent the wheel?
We do not need to reinvent infrastructure that has already been proven, but we must redefine the new abstractions that AI workloads demand.
Kubernetes does not need to be abandoned, but GPU and inference scheduling need to evolve. Gateway technology like Envoy is worth reusing, but MCP and agents need new governance. Containers and sandboxes still matter, but agent runtimes need their own task lifecycle and permission model. Traditional observability remains the foundation, but AI platforms need to track task success rate, end-to-end cost, and service quality.
AI native is neither a simple upgrade of cloud native nor a wholesale rebuild.
Ten years ago we built the cloud native ecosystem around Kubernetes and CNCF; ten years later we are debating a new technical architecture around AI native.
But I believe what actually moves technology forward is still open standards, engineering practice, and sustained collaboration among developers.
Do you see AI native as the natural evolution of cloud native, or a new paradigm that requires rebuilding infrastructure?
If you are building agents, MCP, AI gateways, model inference, or GPU platforms, I would love to hear about the problems you are hitting, especially the ones existing cloud native technology does not solve well.
These views are my own.
2026-09-23 16:50:31
Models getting bigger and GPU partitioning getting more important are two things that can be true at the same time.
This September, at the GPUStack ecosystem conference in Beijing, I found myself chatting with their CTO, Sheng Liang, over the conference dinner. He asked me one question: in the era of large models, is GPU virtualization and partitioning still that important?
He asked it politely, but for someone who works on the HAMi community and its commercialization, it stung. Everyone understands the subtext: models no longer fit on a single card, and you people are still figuring out how to slice one card into pieces. Isn’t that doing aerodynamics research on a wagon wheel?
I gave a polite non-answer that evening, but the question kept bugging me, and I realized it deserved a proper written response. As it happens, in mid-September TypeSafe AI released Jev, a model that cannot chat, and that made the question much more interesting. This post is my formal answer.
First, what Jev is. TypeSafe calls it a System One Model: you send an unstructured state plus a set of typed questions, and the output is not free text but three predefined types, Choice (pick one from a list), Score (rate on a scale), and Noul (yes or no), each with a probability distribution and confidence.
Why do I say it “cannot chat”? Consider how we use a large language model (LLM) today to decide “is this ticket about billing”. Asking a big model to do this is like hiring a Pulitzer winner to fill in a multiple-choice sheet: they will absolutely write you a heartfelt short essay first, and then you scrape the answer out of it with a regex. To get them to say “yes”, we first taught them to write everything, then strapped them to a chair with JSON mode, and then wrote a parser to guard against improvisation.
Jev deletes that entire pipeline. It does no autoregressive generation; it returns typed results that code can consume directly. Multiple questions can go into a single call, evaluated in parallel against the same state. The official docs claim that “adding questions barely changes the response time”, because each question is evaluated independently and cannot pollute the others’ context.
An analogy: a large LLM is like a senior consultant who reads materials and writes reports. Jev is like the real-time approval node inside a company. It will not write your report, but it can process huge volumes of “approve / reject”, “A/B/C”, “risk 0 to 10” judgments per second. You would not hire a novelist to run your access control system, yet that is exactly the architecture many companies run today.
Two buckets of cold water, as usual. First, “Jev cannot hallucinate” needs de-noising: type safety guarantees the output stays within the predefined schema, not that the business judgment is correct. A perfectly valid high_risk can still be a mistake. Second, the “193.6x faster, 444.6x cheaper” numbers on the homepage come from workflows designed by TypeSafe’s own team, and the company itself admits they sit at the high end of real-world gains. They indicate how much headroom the new paradigm has, not that “Jev is 200x faster than GPT”. One more easily misread number: 250,000 tokens/s is an API rate limit, not measured single-GPU throughput. API throughput, model throughput, and GPU throughput have never been the same thing.
Reading introductions gets you nowhere; run it to get the feel. console.typesafe.ai is open for registration. Create an API key, then:
pip install typesafe-sdk
export TYPESAFE_API_KEY=your_keyThe example in the official docs is literally a support ticket classifier, three questions in one call:
from typesafe_sdk import Choice, Noul, Score, TypeSafeClient
with TypeSafeClient() as client:
response = client.system_one(
state={"document": "I was charged twice. Please fix this ASAP."},
questions={
"billing": Noul(instructions="Is this ticket about billing?"),
"tone": Choice(
instructions="What is the customer's tone?",
criteria={"calm": None, "frustrated": None, "angry": None},
),
"urgency": Score(
instructions="How urgent is this ticket?",
criteria=["can wait", "this week", "today"],
),
},
)
print(response.nouls["billing"].noul) # yes/no decision
print(response.choices["tone"].choice) # selected option
print(response.scores["urgency"].score) # rating levelNotice the shape of this API: no prompt, no temperature, no system message. You submit questions and options; you get back types and probabilities. If you have written LLM apps for a while, it feels like something is missing at first glance. Think again, and you realize what is missing was never supposed to be there.
The ecosystem is assembling faster than I expected. Beyond the official Python and TypeScript SDKs, the Rust community has typesafe_ai_rs, Pydantic’s docs already show a TypeSafeModel integration, and someone has built an MCP server for Jev. If you would rather not register directly, OpenRouter, Cloudflare Workers AI, and Vercel AI Gateway can all reach it. Chinese WeChat tech columns are filling up with Jev explainers too. The heat is real.
More interesting: you can already reproduce this approach with open source. Bespoke Labs’ Nimble is one example, a LoRA adapter trained on Qwen3.5-9B, about 165 MiB, Apache 2.0. Its method is blunt: score the allowed answer tokens directly, and generate no free-form text at all during inference.
from inference import NimbleModel
model = NimbleModel("nimble-model")
result = model.score(
context="The store accepts returns within 30 days. "
"This item was bought 12 days ago.",
schema={
"eligible": {
"type": "boolean",
"description": "Is this item within the store return window?",
}
},
)
print(result["fields"]["eligible"]["probabilities"])There are constraints: at most 26 choices per field, inputs over 2,048 tokens are rejected rather than truncated, and you need a CUDA GPU. But its existence proves a point: the so-called System One paradigm is not black magic. Take a general model, constrain its output space, score candidate answers directly, and you get something 70 or 80 percent of the way there for the cost of a LoRA. Which in turn makes Jev’s pricing worth studying: $0.042 per million input tokens, output free, with the homepage proudly noting it is 238x cheaper on input than a certain frontier model. Competitive pressure will likely arrive faster than expected.
Now back to architecture. The real value of this class of models, I think, is not replacing a 70B model with a 3B one, but decomposing the intelligence workload that one big model used to monopolize. An enterprise agent may contain dozens of judgments: intent recognition, permission classification, document relevance, content risk, tool routing, whether to retry, whether to hand off to a human. These do not need frontier-level capability on every single call.
The more sensible division of labor: code owns deterministic logic, specialized decision models own high-frequency judgments, small models own local language tasks, and big models handle only genuinely hard reasoning. NVIDIA Research’s position paper on SLMs and agentic AI says the same thing: most calls inside an agent are repetitive and specialized, and a heterogeneous model system is more economical than “call the same big model for everything”.
Expand the earlier ticket example: code does authentication and field validation first, then asks a decision model three questions in one call. Is it billing? What tone? How urgent? With enough confidence, route and execute directly; refunds and database writes stay in code. Only low-confidence or open-ended cases escalate to a big model.
What I like about this architecture is precisely that it is not “more AI”. It hands system control back to software. Follow this direction and applications grow a distinct judgment plane: a layer of low-latency, confidence-carrying decision models sitting between code and big models. The API Gateway decides where traffic goes; the judgment plane decides who should think about this request. Jev is an early implementation of this direction, and traditional classifiers, rerankers, SLM routers, and Nimble above all belong to the same layer.
Now the direct answer: as models get bigger, does GPU partitioning still matter?
Once you accept that “one application runs a dozen models at once”, the GPU layer’s problem changes. A frontier model may need 8x H200, while the embedding model next to it needs a few GB of memory; rerankers, small vision models, guardrails, and notebooks each consume a sliver of GPU. If you still use Kubernetes’ traditional whole-card semantics:
resources:
limits:
nvidia.com/gpu: 1then that is a commuter who calls a ride-share and gets a 49-seat bus every morning, with boarding forbidden for anyone else. Small models waste; many models fragment.
“Models are getting bigger, so GPU partitioning is obsolete” confuses two opposite directions on the resource axis.
Model parallelism solves one workload using many cards. Tensor Parallel, Pipeline Parallel, and Ray distributed inference all belong here: one card is not enough, use multiple cards on the same node; still not enough, go cross-node.
GPU partitioning solves many workloads sharing one card. HAMi (a CNCF Incubating project) turns the GPU from an integer device into a resource schedulable by memory, compute cores, and device share, pins the exact card at scheduling time, auto-matches MIG templates by compute and memory requirements, and enforces real resource constraints at the CUDA layer via HAMi-core rather than just numbers in the scheduler’s ledger. At the end of the day, split versus join is not just a question of direction. How finely you can split is the real engineering divide.
Here is the fun fact: the industry has already answered half of Sheng’s question for him. Platforms that aggregate models, the ones stitching big models across many cards, are shipping GPU partitioning and flexible slicing in the same product. Joining cards together with one hand while slicing them apart with the other is not a split personality; it is reality: demand exists in both directions at once, and picking a side is the mistake.
Looking further out, heterogeneity only gets worse. MoE decouples total parameters from active ones: Qwen3-30B-A3B has about 30B total parameters but activates only about 3B per token. 4-bit quantization and distillation keep pushing the real per-request cost down. Work like DistServe and Mooncake even splits prefill, decode, and KV cache across separate resource domains. Scheduling units will get finer, not coarser.
That said, do not treat GPU utilization as the only KPI. Cramming four inference instances into one card, taking nvidia-smi from 30% to 90%, does not automatically improve total cost of ownership; if P99 latency degrades and the failure domain widens, that utilization is worth little. HAMi fits GPU pools with obvious fragmentation, multiple tenants, and a high share of dev and test. For services with strict latency SLOs and cards saturated long-term, exclusive ownership or hardware isolation like MIG is the safer bet.
Jev’s significance is not that “small models won”, but that not every kind of intelligence needs to generate language. When one decision model or one LoRA adapter can handle half the checkbox work in an application, applications shift from “one big model does everything” to a dozen small models, each doing its job.
Big models keeping growing will not eliminate small models or GPU virtualization; heterogeneous model architectures mean the same cluster simultaneously faces “model too big” and “task too small” resource problems.
So the core capability of next-generation AI Infra is not merely slicing GPUs, nor merely stacking them, but freely splitting and composing compute resources when the workload demands it. In one sentence: split the small and fragmented, join the large and saturated, isolate the high-SLO, share the low-utilization.
So here is my answer: yes, and more than ever.
2026-09-15 15:55:15
Standing in the Shanghai venue in 2026, I found myself thinking about the days eight years ago when Dan Kohn and I organized the first KubeCon China.
In September 2026, KubeCon + CloudNativeCon returned to mainland China once again, back in Shanghai. The full name of the event has grown much longer than before: this time it also included the OpenInfra Summit and PyTorch Conference, with agent and MCP-focused events running alongside.
The longer name is not a trivial detail. It is almost a microcosm of the technological shifts of the past few years: cloud native is no longer the only protagonist on stage, AI has pushed its way into the spotlight, and Kubernetes, OpenStack, and various cloud native projects are repositioning themselves within the AI technology stack.
For me, this return carried a more personal layer of meaning.
The first KubeCon China was in 2018, also in Shanghai. Dan Kohn came to China back then to discuss and plan the event with us. Kubernetes was still expanding rapidly in those years; many people were hearing the term “Cloud Native” for the first time, and our hottest topics were containers, microservices, service mesh, and whether Kubernetes would become the data center operating system of the future.
More than 2,500 people attended that conference, a record for a first international KubeCon at the time. I still remember clearly how Dan looked on stage, in the venue, and at community events.
Dan later passed away from illness. Today, the KubeCon scholarship program is named after him. Seeing the Dan Kohn Scholarship again in Shanghai, I could not simply treat it as one more program name on the conference website. For me, it connects to a group of real people and to the earliest days of China’s cloud native community.
Eight years on, Kubernetes no longer needs to prove that it can run in production. It has long been part of the infrastructure. Yet the center of the technology world is shifting as well. The question of those years was “how do applications run on Kubernetes”; today the question has become “how does AI run more efficiently and more reliably on top of heterogeneous computing power”.
This time I was mainly responsible for the HAMi booth. Besides meeting many old friends I had not seen in years, one immediate impression was that there were more international attendees than I expected.
At the HAMi booth in particular, my sense was that more than half of the visitors came from overseas. Of course, this is just one person’s observation at a single booth and does not represent the overall attendee mix of the conference. But it still says something: when it comes to GPU scheduling, sharing, and heterogeneous computing management, Chinese projects are drawing global developers who actively want to learn more.
The Todea team from Korea wrote a very detailed recap after the event, summarizing the shared theme of the conference as “recovering capacity from the same hardware”. That phrasing matches what I felt at the booth. The questions people asked about HAMi were no longer just “can Kubernetes recognize GPUs”, but how to partition GPUs, how to improve utilization, how to run training and inference in the same cluster, and how to manage different accelerators such as NVIDIA, Ascend, Cambricon, and Hygon.
This means the GPU has gone from being an add-on device to a first-class resource that the Kubernetes control plane must understand.
The Todea recap also documented several production cases. China Merchants Bank put training and inference on a shared Kubernetes foundation, raising average accelerator compute utilization from 35% to over 60%; INTSIG improved GPU utilization and reduced costs through GPU partitioning, bin-packing scheduling, affinity rules, and elastic scaling; and the case from Viettel in Vietnam was even more direct: under the same workloads and service levels, utilization per card rose from 13% to 59%.
These numbers come from each team’s presentations at the conference; the scenarios and measurement criteria are not identical, so they cannot be compared side by side directly. But together they point to one shift: the primary question of AI infrastructure is moving from “do we have GPUs” to “how much useful work are our existing GPUs actually doing”.
If you only listen to social media, you might think cloud native in China has cooled off compared with a few years ago. Startup pitch decks have changed their labels, marketing budgets have flowed to large models and agents, and many people who used to work on cloud native have started studying inference, GPUs, vector databases, and AI platforms.
I am one of them. Over the past few years I have gradually shifted my focus to AI infrastructure, working on adapting GPUs to Kubernetes. Only after making that turn did I realize that this was not leaving cloud native; it was following cloud native deeper into computing resources.
The “State of Cloud Native Development in China” report released by the CNCF and SlashData during the conference painted a different picture. As of Q1 2026, China has about 1.75 million cloud native developers, of whom roughly 400,000 also work on AI development. The share of cloud native among Chinese backend developers has grown from 30% two years ago to 48%.
Another statistic feels distinctly Chinese: 48% of China’s industrial IoT developers are cloud native developers, above the global average of 42%. Manufacturing, telecommunications, energy, and hardware are the industries where Chinese developers are more concentrated; these systems often need to run for long periods and strongly favor private, controlled infrastructure. Cloud native in China has never simply meant public cloud. It increasingly lives in on-premises data centers, dedicated clouds, edge nodes, and industry platforms.
So I would rather put it this way: cloud native in China has not disappeared; it has sunk beneath the surface. It has gone from a novel concept to the default engineering approach for many systems, and internal developer platforms have hidden it even further. According to the report, 88% of backend developers worldwide already work in some form of standardized DevOps or platform environment. A developer may use a Kubernetes-based platform every day without ever needing to write a Dockerfile, configure container networking, or operate a cluster.
When infrastructure becomes invisible, its voice in public discourse grows quieter, while its responsibility in production systems only grows heavier.
This conference left me feeling that the CNCF’s presence in China has not simply grown stronger or weaker; its nature has changed.
KubeCon China 2018 had a very clear center: Kubernetes and the CNCF. At that time China was the world’s third-largest source of contributions to CNCF projects, and Chinese companies joining the foundation, donating projects, and becoming top contributors were the biggest news of the event.
In 2026, China remains the world’s second-largest source of contributors to CNCF projects. Code contributions have not disappeared, and Kubernetes has not lost its place in production. But this event needed to share a single stage with OpenInfra and PyTorch to fully express today’s technical demands. Infrastructure, container orchestration, model frameworks, and agent systems have been chained together into one story that no single foundation can tell alone.
This raises a question: now that cloud native has become the default foundation, can the CNCF keep defining the questions that the next generation of developers cares about most?
In China, this question is especially practical. Technology decisions at Chinese companies are increasingly shaped by heterogeneous chips, data compliance, private deployment, and local supply chains. Community members still contribute code, but the day-to-day connection between the foundation and local developers is harder to maintain than the contribution numbers in project repositories. A conference can bring one concentrated reunion; real presence comes from year-round local content, user case studies, maintainer growth, and cross-language collaboration.
If the CNCF only emphasizes that “Kubernetes still matters”, that is of course true, but it is no longer enough. What matters more is explaining clearly how Kubernetes takes on AI’s new workloads, and how Chinese developers and domestic hardware can genuinely enter global standards and upstream communities.
At the end of 2025, the Linux Foundation established the Agentic AI Foundation (AAIF), with Anthropic’s Model Context Protocol (MCP), Block’s goose, and OpenAI’s AGENTS.md among its first projects. In less than a year, a new open source narrative has taken shape around agent protocols, tool calling, context management, and collaboration standards.
The impact of the AAIF on the CNCF is not that agents will replace Kubernetes. What it is really competing for is developer attention, vendor budgets, and the entry point to standard-setting.
Ten years ago, a developer who wanted to help build the next generation of infrastructure most likely started with containers, Kubernetes, and cloud native. Today, a young developer is more likely to start with LLM APIs, MCP, agent frameworks, and AI coding tools. The first interface they care about is no longer the Kubernetes API, but how models call tools, how agents exchange context, and how to control permissions and costs.
That is also why this edition of the conference placed agent and MCP-focused events in the same week as KubeCon. Agentic AI is forming its own community center, and its boundary with cloud native has not yet stabilized.
I do not believe the CNCF and the AAIF will end up in a zero-sum competition. The AAIF sits closer to the protocol layer and the developer entry point for agents, while the CNCF is better at governing long-running distributed systems. Once an agent truly enters production, it still runs into identity, isolation, networking, policy, observability, resilience, and multi-tenancy problems, which are precisely the problems the cloud native community has solved again and again over the past decade.
But cooperation will not happen by itself. Who defines the runtime unit of an agent, who manages state, who standardizes identity and telemetry data, and who owns the security boundary? These questions will all shape how future projects and foundations divide the work.
We used to say that Kubernetes is a general-purpose orchestration platform. Agentic AI will force us to re-examine the words “general purpose”.
Traditional web service workloads are relatively stable: a Pod starts and keeps handling requests, and CPU and memory can roughly describe its resource needs. An agent, by contrast, may start suddenly, call several external tools, wait on model inference, save its context, and then go dormant or terminate. It resembles both a short-lived task and a stateful long-runner; it needs sandbox isolation, yet it must also access external systems on the user’s behalf.
This puts several new kinds of pressure on Kubernetes.
The first is the resource model. Training, inference, and agent workloads can no longer be described by “how many GPUs” alone. Video memory, compute ratios, interconnect topology, model caches, and domestic heterogeneous accelerators all need to enter scheduling decisions. HAMi and Dynamic Resource Allocation (DRA) address the layer of this pipeline closest to the device.
The second is the workload model. A single LLM inference replica may no longer be one Pod but a group of Pods coordinating across nodes. Prefill/decode disaggregation, gang scheduling, LeaderWorkerSet, and multi-cluster scheduling all show that the original Deployment abstraction is no longer sufficient to express AI systems.
The third is security and governance. An agent does not just respond to requests; it takes actions. It needs its own identity, fine-grained authorization, network egress control, tool-call auditing, and recoverable execution records. Simply stuffing each agent into a Pod does not automatically solve these problems, but Kubernetes’ ServiceAccounts, policy engines, sandbox runtimes, and observability stack provide a foundation that can keep evolving.
The CNCF and SlashData report also shows that AI developers follow a different cloud native maturity path. They first build data pipelines with Kubernetes, microservices, event-driven architecture, and observability tooling; then rely on immutable infrastructure to make training reproducible; and only after moving into production inference do they need service mesh, chaos engineering, and multi-cluster management more.
This ordering shows that AI is not just another new workload on top of cloud native. It is reordering the priorities of cloud native technology.
Technology trends can be summarized from reports and agendas, but the truly irreplaceable part of a conference remains the people.
This time I met many old friends I had not seen in years. Some still work on Kubernetes; some have moved on to gateways, large models, agents, or GPU chips; others have changed roles and companies several times over. We talked about past projects, and about the new problems each of us faces today.
I have come to believe more and more that a community is not a group of people who forever discuss the same technology. A real community is one where, after the technology cycle turns, those same people are still willing to come back with new questions.
In 2018, we believed Kubernetes would transform infrastructure. In 2026, we are judging whether Kubernetes can support AI’s transformation of software. The questions have changed, and so have the people, but the way of building trust through open source and collaborating across companies has not.
When Dan pushed to bring KubeCon to China, its greatest significance was never just hosting a conference. He helped build a bridge between Chinese developers and the global open source community. Today, that bridge needs to carry more than cloud native; it also needs to connect chips, models, inference systems, and agent protocols.
If I had to sum up KubeCon China 2026 in one sentence, I would say: cloud native has not left the stage; it is losing the center of the stage while becoming the foundation the entire stage stands on.
China’s cloud native community still has scale, production practice, and sustained upstream contributions; its participants have simply dispersed into AI infrastructure, heterogeneous computing, platform engineering, and agent systems. The challenge for the CNCF is to reconnect these new directions. The impact of the AAIF reminds everyone that the entry point for the next generation of open standards may have already shifted from container orchestration to the protocols between agents and tools.
For Kubernetes, this is both pressure and a second chance. It does not need to become an agent framework, nor own the entire AI technology stack. What it needs to do is turn GPUs, networking, identity, isolation, state, and observability into a reliable production foundation, so that AI workloads can run at scale the way microservices do today.
For me personally, this return to Shanghai felt like recalibrating my coordinates. Looking back, I can still vividly picture Dan planning the first conference with us; looking forward, I have already devoted more of my energy to combining GPUs with Kubernetes. The world changes fast, but as long as the community is still willing to reunite, discuss, and build together, the road from the past into the future is still there.
2026-08-28 15:38:12
GPU scheduling is not just “who gets which card”—it is answering three layered questions in sequence: which node, which card, and where the card sits inside the machine. This post walks from that three-layer framework to the HAMi v2.10 policy chain, then settles the most commonly misread policy semantic: mutex does not ask for exclusion from other mutex Pods; it asks for a zero-tenant card. The verdict comes from a decisive experiment anyone can reproduce on a laptop.
An eight-GPU H100 server sells for more than a million yuan, and the GPUs account for the bulk of that cost. You buy it for compute, but the card has three properties a CPU does not: it is too expensive to leave idle; its granularity is too coarse—an entire card carries 80 GB of memory while an inference replica may need only two or three; and its performance is sensitive to neighbors—co-located tenants contend with each other for compute, memory bandwidth, and cache. Together, these three define the fundamental problem GPU scheduling must solve: place this expensive card, at the right granularity, in the right position, for the right workload, at the right time.
The Kubernetes default scheduler, meanwhile, was designed for stateless microservices. Its entire knowledge of a GPU is one line in the resource declaration:
resources:
limits:
nvidia.com/gpu: 1Through its eyes, a GPU is an indivisible integer count. That creates three blind spots:
The first two blind spots live inside the card, and GPU-sharing projects like HAMi exist precisely for them: slice the card by memory and compute, and let multiple Pods co-locate. The third blind spot is subtler and the most counterintuitive. Manan Paliwal’s article Why Kubernetes Is Slowing Down Your GPUs describes exactly this scenario: an H100 cluster that looks perfectly healthy, yet distributed training runs 30% to 40% slower than expected, and adding more cards barely helps. The cause is not CUDA, PyTorch, or NCCL—the scheduler placed the GPU and its RDMA NIC on different NUMA nodes, so every packet has to cross the inter-socket CPU link before reaching the network. Or consider a four-card machine where GPU 0 and GPU 1 hang under PCIe switch A and GPU 2 and GPU 3 under switch B: a workload requests two GPUs, and the scheduler is entirely capable of returning GPU 1 plus GPU 2—the count is right, the placement is wrong, and communication now crosses switches and the CPU interconnect. The most stinging line in that article sums it up: nothing crashes, no job fails, performance quietly evaporates.
Turn the blind spots into a question list, and a single GPU scheduling decision is really three layered questions answered in sequence:
| Dimension | Question to answer | Consequence of getting it wrong or skipping it |
|---|---|---|
| Node | Which node to place on | Poor bin-packing leaves a pile of fragmented nodes; poor spreading wastes whole cards |
| Card | Which card, co-located or exclusive | Sharing denied where it fits is waste; exclusivity denied where it matters is performance interference |
| Topology | Physical position of the card within the machine | Cross-NUMA, cross-PCIe traffic silently bleeds performance |
The default scheduler answers only the first layer, crudely, with “are total resources sufficient?"—layers two and three are left entirely to luck. The Kubernetes ecosystem’s answers are layered: Kueue does hardware-aware whole-job placement at the queueing layer, kubelet’s Topology Manager aligns NUMA inside the node, and the NVIDIA Network Operator with Multus exposes RDMA networks directly to workloads. HAMi’s angle is different: it moves layers two and three into the scheduler itself, expressed as a language of Pod annotations.
Concretely, the v2.10 scheduling model looks like this:
hami.io/node-scheduler-policy for the node level (a single value, binpack or spread), and hami.io/gpu-scheduler-policy for the card level (comma-separated policy chains as of v2.10);Mapped against the table above: binpack and spread answer “how to pack”; numa and topology-aware answer “is the position right”; mutex answers “may it have neighbors.” A chain like mutex,binpack,numa translates to: give me a card nobody has used, pack as tightly as possible, and ideally keep it on the same NUMA node.
On this map, mutex is the only policy carrying exclusive semantics—and the newest addition in v2.10. The precise meaning of its “exclusivity” happens to be the murkiest part of both the documentation and community understanding, and that is precisely the question to settle next.
Verdict first, answering the most direct question: does hami.io/gpu-scheduler-policy: "mutex" make Pods carrying the annotation mutually exclusive with one another? No. Its actual behavior is:
HAMi v2.10 brought the mutex policy and composable policy chains (such as mutex,binpack) to the hami.io/gpu-scheduler-policy annotation. Two readings of the mutex semantics have long coexisted in the community:
| Reading | Meaning |
|---|---|
| Reading A: zero tenants | A mutex Pod may only be scheduled onto a GPU card with no workload at all, whether or not the existing workloads carry mutex |
| Reading B: mutex-only exclusion | A mutex Pod is exclusive only against other mutex Pods and may share a card with ordinary Pods |
Most test reports (including one internal LWS test of ours) cannot distinguish between the two readings, because they set gpuSchedulerPolicy: mutex as a global default. When every Pod is mutex, the two semantics predict identical behavior: with any Pod already on a card, later mutex Pods get rejected. To separate A from B, you must design a mixed scenario: fill the card with ordinary Pods only, then submit a mutex Pod.
The verdict has already been stated: HAMi implements Reading A. Four layers of evidence support it: the original request in issue #2009 says “no existing users”; the implementation in PR #2011 checks dev.Used > 0 (a count that covers every Pod on the card); observations from our real-GPU experiment on a four-card T4 node on GKE; and the local mock experiment in this post. This article documents the full reproduction of the fourth layer of evidence—it needs no GPU at all and runs in about twenty minutes on a laptop.
To understand why this policy appeared, start from the fundamental tension of GPU sharing.
The value of GPU virtualization projects like HAMi lies in slicing: cut an 80 GB card into shares by memory and compute, co-locate multiple inference replicas on it, and utilization goes up while cost comes down. But slicing has an inherent price: co-located tenants share SMs, share memory bandwidth, and pollute each other’s L2 cache. Throughput-oriented batch jobs don’t care; two classes of workloads find it lethal: latency-sensitive online inference (uncontrollable tail latency), and benchmarking or training jobs that need a stable baseline (irreproducible results).
Before mutex, users who wanted a card to themselves had only two unsatisfactory options:
mutex fills the gap in between: a scheduler-layer, Pod-granularity, zero-hardware-dependency soft-exclusivity switch. It introduces no new isolation technology; it simply tells the scheduler “give me a card with zero tenants right now.” Combined with HAMi’s existing soft slicing, a cluster can for the first time serve both classes at once: shared slices amortize cost at peak, while mutex claims an idle card exclusively for overnight benchmarks or latency-sensitive services, released the moment the job finishes.
Its arrival in v2.10 is not isolated either. The same roadmap (#1889) advanced policy chains (#2010) in parallel, because real clusters stack their demands: pack tightly to free whole cards, keep NUMA affinity, and still hold a few exclusive cards for critical workloads. The chained expression mutex,binpack,numa is the complete form of those demands. Another typical case is replica-group scheduling with LeaderWorkerSet: GPU Pods within the same replica group should not share physical cards, and marking the group’s Pods as mutex satisfies that naturally.
There is one more design trade-off worth savoring: mutex exclusivity is one-way and binds only its own placement moment. Implementation-wise it merely reuses the scheduler’s existing Used counter (any Pod allocation on a card increments it), introducing no runtime locks or reservations. The choice keeps the implementation minimal and the semantics unambiguous, at the cost of not constraining later workloads: ordinary Pods scheduled afterward can still join the card. For end-to-end exclusivity, request a whole card’s resources or pin the card with use-gpuuuid. Where to draw the line of “guarantee” in a scheduling system is always a trade-off; HAMi draws it at placement time and leaves stronger constraints for users to express explicitly.
The evaluation model of the policy chain was given in the previous section; here we zoom into the exact behavior of the mutex filter inside the scheduler. It stands guard at the entrance of the candidate card set with a single criterion: whether the card’s Used count is 0. That count covers every Pod on the card, mutex annotation or not.
The “one-way exclusivity” is clearest on a timeline. The same Used > 0 check executes only when the mutex Pod itself is being scheduled:
The three experiments below map to these two diagrams: Experiment 1 verifies that the filter condition is “any tenant” rather than “mutex tenants only”; Experiment 2 verifies the happy path of the first diagram; Experiment 3 verifies the one-way property of the second.
The overall idea: bring up a single-node cluster with kind, install the HAMi scheduler (control plane only), then use HAMi’s official mock-device-plugin to register two fake Tesla T4s on the node. The HAMi scheduler runs its policy computation on them exactly as it would on real cards.
The experiment uses exactly the same build as HAMi’s official Lab 14 (the GKE experiment): the Helm chart comes from HAMi source commit 45b3d46769b44cfc1445728dfcb8e524939afba1 (master HEAD on 2026-08-17, i.e. the v2.10.0 release candidate), and the image is the per-commit tag HAMi CI published for that commit, projecthami/hami:45b3d46. Do not take the shortcut of latest: it is a moving tag, and by the time of writing it had already drifted to a newer master commit (the version embedded in the image changed from 45b3d46 to 949f78e), which would distort the reproduction.
kind nodes pulling images themselves is at the mercy of the network; pull on the host first, then load:
kind create cluster --name hami-mutex2 --image kindest/node:v1.36.1
# Pre-pull on the host (retry a few times on a flaky network)
docker pull projecthami/hami:45b3d46
docker pull projecthami/mock-device-plugin:latest
docker pull \
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.36.1
docker pull liangjw/kube-webhook-certgen:v1.1.1
kind load docker-image projecthami/hami:45b3d46 --name hami-mutex2
kind load docker-image projecthami/mock-device-plugin:latest --name hami-mutex2
kind load docker-image \
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.36.1 \
--name hami-mutex2
kind load docker-image liangjw/kube-webhook-certgen:v1.1.1 --name hami-mutex2The kube-scheduler image is the sidecar used by the scheduler embedded in the HAMi chart—its tag must match the cluster version; certgen is the image for the webhook certificate Job during installation.
# Fetch the same chart source as Lab 14
curl -fsSL https://codeload.github.com/Project-HAMi/HAMi/tar.gz/45b3d46769b44cfc1445728dfcb8e524939afba1 \
-o hami-src.tar.gz
tar xzf hami-src.tar.gz
helm install hami HAMi-45b3d46769b44cfc1445728dfcb8e524939afba1/charts/hami \
-n kube-system \
--set global.imageTag=45b3d46 \
--set devicePlugin.enabled=false \
--set mockDevicePlugin.enabled=true \
--set mockDevicePlugin.image.tag=latest \
--set mockDevicePlugin.image.pullPolicy=IfNotPresentThree parameters matter:
devicePlugin.enabled=false: no real GPUs here, so HAMi’s own device plugin is disabled;mockDevicePlugin.enabled=true: the mock plugin takes over, registering virtual memory/compute extended resources on the node;mockDevicePlugin.image.tag=latest: latest is mandatory. Version 1.0.1 cannot parse the new vnpus config format in the current chart and crashes outright with cannot unmarshal !!map into []ascend.VNPUConfig—a pit I fell into personally.The mock plugin is designed as a three-piece set: a device-config block (already shipped in the chart), a count extended resource on the node (the health gate—only needs to be greater than 0), and the hami.io/node-nvidia-register annotation (describing the fake cards). The latter two must be provided by hand:
NODE=hami-mutex2-control-plane
# Health gate: nvidia.com/gpu = 2 cards x 10 slices
kubectl patch node $NODE --subresource=status --type=json -p '[
{"op": "add", "path": "/status/capacity/nvidia.com~1gpu", "value": "20"}
]'
# Two fake T4s: GPU-MOCK-A and GPU-MOCK-B, 15360 MiB each
kubectl annotate node $NODE --overwrite \
'hami.io/node-nvidia-register=[
{"id":"GPU-MOCK-A","count":10,"devmem":15360,"devcore":100,
"type":"NVIDIA-Tesla-T4","health":true,"numa":0,"mode":"hami-core"},
{"id":"GPU-MOCK-B","index":1,"count":10,"devmem":15360,"devcore":100,
"type":"NVIDIA-Tesla-T4","health":true,"numa":0,"mode":"hami-core"}
]'Wait about 30 seconds, then confirm the mock plugin finished registering resources:
kubectl get node $NODE -o jsonpath='{.status.allocatable}' | python3 -c "
import json, sys
alloc = json.load(sys.stdin)
nvidia = {k: v for k, v in alloc.items() if 'nvidia' in k}
print(json.dumps(nvidia, indent=2))
"{
"nvidia.com/gpu": "20",
"nvidia.com/gpucores": "200",
"nvidia.com/gpumem": "30720",
"nvidia.com/gpumem-percentage": "200"
}Both cards are ready. All test Pods use the same template: request 1 vGPU and 1000 MiB of memory. The environment preloads every image, but note that imagePullPolicy: IfNotPresent is still worth writing explicitly, to avoid depending on the registry at runtime:
cat > plain-pod.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: plain-N
spec:
restartPolicy: Never
containers:
- name: app
image: docker.io/projecthami/hami:45b3d46
imagePullPolicy: IfNotPresent
command: ["sh", "-c", "sleep 3600"]
resources:
limits:
nvidia.com/gpu: 1
nvidia.com/gpumem: 1000
EOFThe single tool for observing placement is this command—the HAMi scheduler writes the chosen card onto a Pod annotation:
kubectl get pods -o custom-columns=\
'POD:.metadata.name,'\
'CARD:.metadata.annotations.hami\.io/vgpu-devices-allocated'Step one: deploy two plain Pods with no policy annotations at all (sed renames them):
sed 's/plain-N/plain-1/' plain-pod.yaml | kubectl apply -f -
sed 's/plain-N/plain-2/' plain-pod.yaml | kubectl apply -f -
kubectl wait --for=condition=Ready pod/plain-1 pod/plain-2 --timeout=3mPOD CARD
plain-1 GPU-MOCK-B,NVIDIA,1000,0:;
plain-2 GPU-MOCK-A,NVIDIA,1000,0:;The default spread policy puts the two plain Pods on separate cards. At this moment the cluster contains not a single mutex Pod. Submit one:
cat > mutex-pod.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: mutex-1
annotations:
hami.io/gpu-scheduler-policy: "mutex"
spec:
restartPolicy: Never
containers:
- name: app
image: docker.io/projecthami/hami:45b3d46
imagePullPolicy: IfNotPresent
command: ["sh", "-c", "sleep 3600"]
resources:
limits:
nvidia.com/gpu: 1
nvidia.com/gpumem: 1000
EOF
kubectl apply -f mutex-pod.yaml
sleep 20
kubectl get pod mutex-1
kubectl describe pod mutex-1 | sed -n '/Events:/,$p' | tail -3NAME READY STATUS RESTARTS AGE
mutex-1 0/1 Pending 0 20s
Warning FailedScheduling 20s hami-scheduler
0/1 nodes are available: 1 2/2 ExclusiveDeviceAllocateConflict.
no new claims to deallocate,
preemption: 0/1 nodes are available: 1
No preemption victims found for incoming pod.2/2 ExclusiveDeviceAllocateConflict: both cards rejected. If Reading B (mutex-only exclusion) held, there would be no mutex Pod on either card right now, and this Pod should schedule immediately. It did not.
Step two: delete plain-1 to free GPU-MOCK-B, and watch where the mutex Pod goes:
kubectl delete pod plain-1 --force --grace-period=0
kubectl wait --for=condition=PodScheduled pod/mutex-1 --timeout=2mPOD CARD
mutex-1 GPU-MOCK-B,NVIDIA,1000,0:;
plain-2 GPU-MOCK-A,NVIDIA,1000,0:;The mutex Pod landed on the card that had been completely emptied, while plain-2 was still running on the other card. Zero-tenant semantics confirmed.
Clear all Pods, then submit three mutex Pods in a row:
kubectl delete pod --all --force --grace-period=0
kubectl apply -f mutex-pod.yaml
sed 's/name: mutex-1/name: mutex-2/' mutex-pod.yaml | kubectl apply -f -
sed 's/name: mutex-1/name: mutex-3/' mutex-pod.yaml | kubectl apply -f -POD PHASE CARD
mutex-1 Running GPU-MOCK-B,NVIDIA,1000,0:;
mutex-2 Pending GPU-MOCK-A,NVIDIA,1000,0:;
mutex-3 Pending <none>The first two mutex Pods take the two cards; the third is rejected with 2/2 ExclusiveDeviceAllocateConflict. Both readings predict the same outcome here, so this serves as the control group. As a side note, mutex-2 briefly emitted a node lock contention event before succeeding on retry—a known HAMi node-lock contention behavior that does not affect semantics.
One last question: once a mutex Pod has taken a card, is that card exclusive from then on? With mutex-1 running on GPU-MOCK-B, submit a plain sharing Pod annotated binpack:
POD CARD
binpack-1 GPU-MOCK-B,NVIDIA,1000,0:;
mutex-1 GPU-MOCK-B,NVIDIA,1000,0:;
mutex-2 GPU-MOCK-A,NVIDIA,1000,0:;The binpack Pod landed on the very same card the mutex Pod was using. Exclusivity applies only at the mutex Pod’s own placement moment and is one-way: it demands an empty target card at placement time, but does not reserve the card afterward.
All three experiments agree fully with the code, the original requirement, and the observations on real GPUs. HAMi’s mutex semantics fit in two sentences:
Practical advice for users: if your goal is “this workload owns the whole card until it exits,” mutex only gets you halfway. Either request the card’s full resources (saturate both memory and compute), or pin the card with nvidia.com/use-gpuuuid; both are also covered in the official HAMi blog.
Looking back at the motivation and the mechanism, the value of mutex is not any isolation technology per se, but that it adds the “I want to live alone” option to the soft-slicing world, hooked into the policy chain through the lightest possible implementation—a filter. Placed back on the three-dimension map from the beginning of this post, mutex occupies just one cell, “card-level exclusivity”; but precisely because policies compose, that cell can stack with packing preference and NUMA affinity in the same chain, each evolving on its own. The methodology for judging its behavior matters just as much: when documentation wording diverges, construct a minimal scenario where the two readings predict different outcomes, reproduce it locally with mock devices, and the verdict arrives within minutes. The same method can extend to verifying the ranking behavior of binpack,numa chains, the fallback rules of combined policies, and more—extensions welcome.
kubectl delete pod --all --force --grace-period=0
helm uninstall hami -n kube-system
kind delete cluster --name hami-mutex2This post started from the question “why do GPU scheduling at all,” broke out the three scheduling dimensions—node, card, topology—and showed how HAMi v2.10 covers them with a policy chain of filters plus sort keys. It then zoomed into mutex, the only policy with exclusive semantics: using kind plus mock-device-plugin, we built a GPU-less local environment and settled its semantic dispute through three experiments—it demands a zero-tenant target card (not merely exclusion from other mutex Pods), and its exclusivity is one-way, binding only at placement time. The experiments use the same v2.10.0 release-candidate build as official Lab 14; every command and output comes from actual runs and is fully reproducible. The post also records three real-world pitfalls: mock plugin 1.0.1 cannot parse the new config format and requires the latest tag, the latest image should be explicitly IfNotPresent, and preloading images onto the kind node is essential.
