2026-08-13 23:30:27
In a service-based architecture, a single product screen showing a user profile, that user’s five most recent orders, the delivery status of each order, and a short list of recommendations requires data from four separate services. Each service stores only its own data, so none of the four services returns the full set on its own. In other words, the caller issues four separate calls, and merges the resulting four responses in the structure needed for the user interface. This merging step is API composition, and it exists in every system where data is split across more than one service.
The code that performs this merging can run in several places. It can run inside the mobile application, on a server in the datacenter, at a CDN edge location, or inside one of the four services. Putting a server between the mobile application and the four services adds a network hop, which sounds like it should cost extra time. However, it usually reduces total load time instead, because a round trip between a phone and a server on a weak mobile connection can take a few hundred milliseconds, while a round trip between two services inside the same datacenter takes a fraction of a millisecond. To put it simply, trading four expensive round trips for one expensive round trip plus four cheap ones is often a large net saving.
However, latency is only the first tradeoff. The place where the merge operation runs also determines what happens when one of the four services is unavailable, how much of the response can be cached, and which team has to approve a change before the screen ships.
In this article, we are going to dive deep into the area of the API composition problem and the patterns associated with it. Here’s what we will cover:
The API composition problem
Client-side composition
Over-fetching and under-fetching
Composition, aggregation, and orchestration
API gateways
Backends for Frontends
GraphQL as a composition layer
Edge composition
Availability and caching tradeoffs
Versioning across multiple frontends
Ownership of the composition layer
2026-08-12 23:30:02
GLM-5.2 is Z.ai’s flagship long-horizon coding model, featuring a usable 1M-token context window that reliably maintains API contracts and engineering intent where others fail. Stop managing GPU clusters. With Crusoe Serverless Fine-Tuning, you customize GLM-5.2 on your proprietary data in a tenant-isolated environment—no data sharing, no infrastructure overhead. When your job finishes, deploy to Self-Serve Deployments in one click or download raw .safetensors weights for full portability. Every run is reproducible, auditable, and fully yours. No lock-in. No guesswork. Experience production-grade, customized performance without the DevOps burden.
AI models have solved the writing code part of software development to a great extent. Today, a capable model can produce a working function, a full component, or a small application from a plain-language description. It can do so in seconds for a fraction of the cost.
This change has shifted the economics of every developer platform. As the generation of new code becomes cheaper and widely available, it stops being the differentiating factor for a platform. This is the reason GitHub, Vercel, and Replit are trying to rebuild themselves around solving other hard problems in the software development process.
To understand what the three companies are doing, we trace one unit of work through every platform while asking the same questions:
Where does the AI actually run the code it writes?
How does each platform verify that the code works?
How does the finished product reach production, and who is permitted to ship it?
Here’s what we will cover:
Why does cheap code generation move the hard engineering problem to another place, and how does that impact the platform’s overall value?
GitHub’s orchestration bet, built on ephemeral cloud environments and a control layer that routes work across competing agents.
Vercel’s production bet, built on isolated microVM sandboxes and a billing model matched to how agents actually run
Replit’s verification angle, built on a self-testing loop that drives a real browser to catch code that only looks like it works
How the MCP standard lets any agent reach any tool, and why all three companies now support it
How Stripe turned payments into something an agent can set up from inside a coding tool, and the credential design that keeps it safe
Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.
A language model can now turn a description into working code. This means a developer can write a sentence and receive a function, a page, or a small application that actually runs right out of the gate.
This capability used to be the scarce and valuable part of a developer tool. Today it is widely available and close to free, which leaves a platform that offers only generation with little to charge for.
So what matters now is not raw code generation, but aspects of software development that come up after the code is available. Three questions carry most of the weight now:
Where does the agent run the code it produces, and how is that environment kept safe?
How does the platform confirm that the generated code actually works?
How does the result reach production, and who is allowed to ship it?
GitHub, Vercel, and Replit answer these questions in different ways.
GitHub puts its effort into coordination
Vercel into the path to production
Replit into verification
GitHub gives us the clearest place to start, because it deals with the pull request workflow most developers already use.
AI is in your engineering workflow. While the token spend shows it, the throughput doesn’t. The human is very much still in the loop, and that’s a context problem.
Join live on Aug 19 (FREE) to learn:
The 4 metrics to measure where AI gains leak out before production.
The 8 stages of context maturity, the specific walls capping your metrics, and a free tool to pinpoint where your team is
Why more MCPs and bigger context windows aren’t enough, and what it takes to get real value from your agents.
GitHub made a specific choice about where the value sits.
Rather than building its own model and competing on raw generation, it built a control layer that coordinates many agents and keeps their work governed, all inside the pull request workflow that developers use every day.
The mechanics start with where the code runs.
GitHub’s coding agent operates in its own ephemeral development environment, which is a temporary workspace that exists only for that task. It is powered by GitHub Actions, the same automation system that runs tests and builds on the platform [1][2]. In practice, you assign a task to the agent the way you would open a ticket. The agent reads through the repository, edits files, runs the tests and linters (tools that check code for problems), and opens a draft pull request for a person to review [1]. Each cloud run happens in a fully isolated, single-use Linux environment hosted by GitHub. Every new task starts from a clean workspace [4]. The review step stays human, which keeps the existing quality gate intact.
See the diagram below:
A single agent sits above the coordination layer. GitHub calls it Agent HQ. It introduces a mission control view that lets a developer assign, steer, and approve work across a fleet of agents from GitHub and VS Code [3]. The agents available inside a paid Copilot subscription include ones from Anthropic, OpenAI, Google, Cognition, and xAI [3].
Governance is treated as version-controlled configuration. Teams define custom agents through AGENTS.md files that carry rules such as a preferred logger or a required testing style, and a control plane gives administrators security policies, audit logging, and model-access controls in one place [3].
See the diagram below:
For developers who are already accustomed to handling issues and pull requests, this design adds agents to a pretty familiar workflow. Reviewing a colleague’s branch is not so different from reviewing an agent’s branch.
Routing to other companies’ models is a deliberate decision. GitHub is positioning the workflow, the execution environment, and the governance layer as the durable product, and treating the underlying model as a swappable component.
Let us now look at how Vercel developed its architecture around what happens after the code exists. This is somewhere between a working demo and production-ready software.
Vercel starts from a different premise where code generation is assumed, and the design deals with carrying the generated code into production. This is the stage where most enterprise software work actually happens, because it involves existing applications rather than fresh prototypes.
The rebuilt version of v0, Vercel’s generation product, runs on a sandbox. It is an isolated space for executing code that imports a real GitHub repository and automatically pulls in the project’s environment variables and configuration. Every prompt produces code that fits the actual application and lives in the repository itself [5].
A Git panel handles the workflow around it. You create a branch for each chat, open a pull request against the main branch, and deploy when it merges. This means a product manager or a designer can ship through the same review process an engineer uses [5].
Vercel’s rationale about this approach is that AI-assisted building is already happening inside companies, and it has produced real failures. Incidents have been reported, such as credentials pasted into prompts, private data reaching the public internet, and deleted databases, often with the audit trail left empty [5]. Therefore, wrapping code generation in real deployment controls is the right response.
See the diagram below:
Underneath the workflow is the execution layer. Every sandbox runs inside a Firecracker microVM, a lightweight virtual machine that isolates untrusted code. The reason for this isolation is that the code an AI wrote is code you have yet to review. Therefore, running it needs a boundary strong enough to contain mistakes. A microVM provides that strong boundary.
The billing model depends on how agents actually run. Vercel’s Fluid compute lets several requests share one running instance, with one using the processor while another waits on input or output. It charges for active processor time while treating wait time as free [6][7]. Agentic workloads spend much of their time waiting on a model to respond, so this pricing matches the real work being done.
This approach deals with a frustration many developers face in their careers. The thing that worked in a demo behaves differently in production. However, Vercel’s design tries to close that gap by making the preview a real deployment from the start.
Nevertheless, strong isolation carries a cost per unit of compute. There is an open question about whether the heaviest workloads eventually move to cheaper execution options elsewhere. Vercel’s wager is that a single, convenient platform keeps them in place.
To summarize, GitHub owns the workflow, and Vercel owns the route to production. Both of them still depend on one assumption that Replit chose to attack directly. Let us look at that now.
Replit concentrated its work on whether autonomously generated code genuinely works using a verification loop built into the agent itself.
Replit’s Agent 3 runs what the company calls a reflection loop. The agent generates code, runs it, tests the result, and repairs failures, repeating that cycle until the tests pass [8]. This loop is reliable because of how the testing is done. Replit built a REPL-based verification system that runs code immediately and pairs that execution with a real browser it drives automatically, so it can click buttons, submit forms, and check data the way a user would [9].
The specific problem this approach targets has a memorable name inside Replit: the Potemkin interface. It is basically a feature that looks complete on screen yet fails the moment it is used [9]. Catching that class of error is what allows the agent to run on its own for more than 200 minutes at a stretch, a large increase over the roughly 20 minutes of its predecessor [8][9].
The verification runs as its own process.
A testing subagent follows a simple cycle of taking an action, observing the result, and repeating. When it finishes, it returns a summary to the main agent describing what works and what broke [9]. This multi-hundred-step testing costs a median of roughly twenty cents per session and runs several times faster and more cheaply than relying on general-purpose computer-use models [9].
See the diagram below:
Anyone who has shipped a feature that looked finished and later broke understands the problem here. Replit is trying to automate the check that catches exactly that, so that the automation can be trusted.
A verification loop reduces the risk substantially. Some failures still surface only under conditions a single test session might miss, which keeps this a pretty hard problem even with a capable tester in place.
Every architecture we have looked at assumes its agent can reach tools and data that live outside the model, and doing that cleanly requires a common method. That method is the Model Context Protocol, usually shortened to MCP.
Before a standard like this existed, connecting several AI applications to several external tools meant writing a separate custom integration for each pairing. In this approach, the number of integrations grew quickly as both sides multiplied. Anthropic introduced MCP to replace those fragmented, one-off connections with a single protocol, so each application and each tool implements the standard once and then works with everything else without additional changes [10].
A host, which is the AI application such as an IDE or a chat client, creates one or more clients, and each client connects to a server that exposes some capability [11]. A server offers three kinds of capability:
Tools, which the model can call to take an action, such as creating a record or running a query.
Resources, which supply context data the model can read, such as a file or a database schema.
Prompts, which provide reusable instruction templates.
The whole exchange runs over a defined message format across either a local or a remote connection. The effect is that a tool provider builds one MCP server and every compliant agent can use it [11].
Replit was among the earliest developer tools to integrate MCP [10], GitHub added an MCP registry to VS Code where a server can be enabled with a single click [3], and Stripe runs an official MCP server for its payment operations [12].
Going deeper, MCP also standardizes how agents reach existing APIs rather than replacing those APIs. A single shared entry point also concentrates security in one place. That makes careful control over which servers an agent may use an important part of any real deployment.
Each of these architectures and approaches has trade-offs:
GitHub gains breadth and governance by routing to many vendors’ models. But the cost is that it owns the surface rather than the intelligence underneath. Whether a coordination and governance layer stays valuable as models and agents keep changing is an open question.
Vercel gains strong isolation by running generated code inside microVMs, and that isolation carries a cost per unit of compute. There is a question about whether the heaviest workloads eventually move to cheaper execution elsewhere
Replit gains long stretches of autonomy through its verification loop, and the more work an agent does on its own, the more weight rests on that verification being right. The Potemkin problem (code that looks complete yet fails when used) stays difficult even with a capable tester, because some failures appear only in situations a test session might miss.
MCP gains a clean, reusable way to connect agents and tools, and a shared standard also concentrates risk into a common entry point. When many agents reach many tools through one protocol, controlling which servers an agent may use, and their permission levels, becomes a central problem rather than a detail.
Understanding these costs is what separates picking one of these tools from understanding why it was built the way it was.
The pattern across all three companies is the same. Code generation became cheap, so the value moved into the engineering that surrounds it. Each company placed its bet on a different piece of that surrounding work.
GitHub bet on orchestration, building a control layer that runs and governs many agents inside the pull request workflow developers already use.
Vercel bet on production, wrapping generated code in real deployments and running it inside isolated microVMs built for untrusted code.
Lastly, Replit bet on verification, driving a real browser in a self-testing loop so an agent can work on its own for hours and still be checked.
Underneath all three, MCP provides the common protocol that lets any agent reach any tool, which is why every one of these platforms now supports it.
References:
2026-08-11 23:30:42
Free masterclass: Learn practical strategies for predictable low-latency writes at scale
Free Masterclass: Optimizing Write-Intensive Database Performance
Database writes – at scale – are one of the hardest problems in distributed systems. This masterclass will teach you how to understand and avoid latency spikes in real-time, write-heavy database workloads. Our panel of experts will share a practical framework for diagnosing write bottlenecks and knowing which strategies to apply in different scenarios.
After this free 2-hour masterclass designed for developers, engineers, architects, and database practitioners, you will know how to:
Identify which factors (database internals, database configurations, data modeling…) are impacting your write performance
Avoid mistakes that have caused 40x write amplification in production
Sustain low P99 latencies even during sustained growth and volatile spikes
All attendees will get the complete Database Performance at Scale book by the masterclass instructor Felipe Mendes.
How does a website charge a visitor who arrives anonymously, skips every advertisement, and leaves within a second?
For most of the web’s history, the question rarely came up, because the visitor was a person whose attention a site could sell through an ad or a subscription. If the website had good content, the website owner could count on multiple such visits by the same user. But now the visitor being a person is not always true. More than half of the traffic online now comes from software that acts on a person’s behalf, requests a page, and leaves without engaging with any ads or subscriptions [2].
Cloudflare seeks to change this.
As you might be aware, Cloudflare sits between a large share of the world’s websites and everything requesting them. It works as a reverse proxy that each request passes through before it reaches the origin server [1]. This lets Cloudflare read a request and act on it early. Over the past year, the company has used this position to sort automated traffic by what it does, to verify the identity behind a request, and, most recently, to collect payment for a request through an open protocol named x402 [1].
In this article, we will go through Cloudflare’s solution in the following five steps:
The web’s usual way of earning value leans on human attention, and agent traffic changes it fundamentally.
A look at Cloudflare’s initial solution around blocking automated traffic and then charging for each crawl.
Settling identity, permission, and payment inside a single request.
A look at how x402 exchange completes this settlement.
The x402 exchange completes this settlement through a short back-and-forth over ordinary HTTP
The costs and open questions of this approach
Disclaimer: This post is based on publicly shared details from Cloudflare. References at the end. Please comment if you notice any inaccuracies.
For most of its history, websites made money after a request rather than during it. A browser asked for a page, and the server returned it at no charge. The value arrived later, once a person saw an advertisement, bought a subscription, or came back for another visit [1]. The request itself stayed free, and this setup funded a large part of the Internet.
A growth in agent traffic is changing this dramatically. An agent is software that acts on a person’s behalf, which in practice means it requests a page or a data feed once, takes what it needs, and finishes in a single pass. It moves past advertisements, operates outside any subscription, and completes its task before a site has a chance to earn from that attention [1]. Each of the three old settlement points depended on a person staying long enough to be counted, so software traffic leaves those points idle.
The strain here is that this kind of traffic is now fast becoming the majority. More than half of the requests reaching websites come from software rather than people [2]. This means that request volume climbs while revenue stays flat.
To make things clear, parts of the Internet were already charging by usage before any of this. Cloud services and APIs have been sold by the call and by the hour for years, though only to a buyer the seller already knew, who signed up and received an API key [1]. Charging an anonymous caller a fraction of a cent for a single request stayed impractical, because collecting such a small payment once cost more than the payment returned [1].
If the value used to settle downstream of the request, the natural question is which party is positioned to move that settlement back onto the request. The answer starts with where Cloudflare sits.
A reverse proxy is a server that stands in front of other servers and receives requests on their behalf. Since Cloudflare operates as a reverse proxy for a large portion of the web, a request headed for one of those sites reaches Cloudflare’s network first and passes through it on the way to the origin [1]. The origin is the site’s own server, the machine that ultimately holds the page or runs the API.
On a side note, the position of a proxy is more general-purpose than caching. Caching stores a copy of a response so it can be served quickly the next time, and it is just one of the many useful things a proxy can do. From the same middle position, a request can also be read, classified, checked, and acted on before it continues to the origin. This broader capability is quite consequential at Cloudflare’s scale. [2].
However, acting on a request early depends on first knowing what the request is for. This is a classification problem. Let us see how Cloudflare handles this.
Before any rule can apply to an agent, the traffic has to be sorted by what it does. Cloudflare’s taxonomy groups automated traffic by behavior rather than by the single label “AI”. Three behaviors impact the key policy decisions [4].
Search covers behavior that builds an index of a site so an engine can answer questions about it later. This behavior is responsible for sending referral visitors back.
An agent covers behavior that acts in real time on a person’s behalf, usually with a human waiting for the result. Think of it like an assistant fetching a page during a conversation.
Training covers behavior that takes content to train or fine-tune a model, where the content is absorbed into the model rather than pointing a visitor back.
These three appear identical in a raw request log, yet they carry very different consequences for a site’s business.
A single crawler can also perform more than one of these behaviors, but this separation lets them record all of them[4]. A crawler that both builds a search index and gathers training data is tracked as doing both, which lets a site owner reason about the full set of things that the crawler does on their pages.
Cloudflare classifies further behaviors as well, including checkout actions and data collection, and it also lets a site express what a bot may store and reshare after accessing a page [4].
With a way to see the proxy position and a way to classify what arrives, let us now look at Cloudflare’s first attempts at implementing a policy.
Cloudflare’s first solution was a simple one. A single control lets a site block automated AI traffic outright. This protected the content and left the earning model untouched. Blocking answers the question of access, yet it does not rebuild the revenue that the old bargain provided.
The second solution added a middle path.
Pay Per Crawl lets a website allow a crawler, block it, or charge it a flat per-request price, with Cloudflare handling the billing as the merchant of record [5]. A site owner could keep a crawler out, let it through, or attach a price to its access, all from one setting. This turned the binary of block-or-allow into three options and gave content owners a way to earn from crawler access at network scale [5].
A year on, Cloudflare made an adjustment to its own model. The company argued that a crawl is a weak measure of value, because a single page might be crawled once and then cited in thousands of AI answers, or crawled repeatedly and cited in none [3]. It backed the argument with a figure from its own network, that more than half of the crawl traffic from well-behaved bots goes to re-fetching pages that have stayed the same since the last visit [3]. Counting crawls, then, counts something that only loosely tracks the value delivered.
So the unit of payment began to move from the crawl toward the use, an approach Cloudflare describes as Pay Per Use. It is candidly framed as an experiment at this point [3]. This is because while pricing the outcome aligns payment with value more closely than pricing the fetch, it is also harder to measure.
The design goal for recent changes is to resolve identity, permission, and payment inside a single request, at the edge, before the origin responds.
Three main concerns are considered over here:
Identity: The traditional identifier, the User-Agent string, can be set to any value by the caller, so it offers weak assurance. Cloudflare’s answer is Web Bot Auth, an authentication method that uses cryptographic signatures in HTTP messages to verify that a request comes from a particular automated source [7]. In practice, the operator signs its request with a private key and publishes the matching public key at a known location, and Cloudflare validates the signature at the edge [8]. A valid signature stands in for a reliable identity, which replaces a guess with a verifiable claim.
Permission: This is expressed through the behavior classification already covered and through the preferences a site sets about how its content may be used [4].
Payment: This comes last and is attached to the request through x402. We will cover this in detail in the next section.
All three concerns resolve at the edge, so the origin receives a request only once identity, permission, and payment have been settled [1]. The metering and settlement are taken away from the website’s own servers. What stays with the site owner is the part that matters to them, which is their rules, their prices, and their revenue [1].
To summarize:
Identity answers who is making the request, through a signed and verified claim.
Permission answers whether this behavior is allowed on these pages, through the classification and content preferences.
Payment answers whether the caller has paid the stated price through the x402 exchange.
These pieces sit at different stages of maturity. Identity verification through Web Bot Auth is available today at the edge [7], while the Monetization Gateway that brings the payment piece together is open as a waitlist rather than a shipped product [1].
The key takeaway is that when one component sits in the middle of a flow, the concerns that are common across every request are collected at that point. Authentication, authorization, and billing consolidating at a gateway is the same pattern that appears in service meshes and middleware.
The x402 protocol makes it possible to pay over HTTP. It takes its name from a status code that has been part of the HTTP standard for a long time [1]. The code is 402, and it means Payment Required. Sites behind Cloudflare already send more than a billion of these responses on an average day, which shows how often a machine requests something priced and receives a message that a payment is due [6].
See the diagram below that shows the overall setup:
The exchange runs through a short sequence that can be compared to a small state machine [1].
A client requests a resource that sits behind a price.
Rather than returning the resource, the server responds with 402 and a small payload stating the price, the accepted asset, and where to pay.
The client re-sends the same request with proof of payment attached.
A facilitator verifies the payment, and the server returns the resource.
Two properties make this suitable for machine traffic. The payment amounts can be very small because the protocol adds almost no overhead to the request. And the payment itself serves as the credential, so a buyer with no prior relationship can access the content by showing the proof of payment [1]. This property is the one that matches an anonymous agent passing through once, since it removes the signup step that per-seat licensing and API keys always required.
The negotiation process is handled inside ordinary requests and responses, with a redirect to a checkout page absent and a separate payment API absent [1]. Nothing was added at the protocol level, since the 402 code has been part of HTTP for decades. What changed is that a settlement angle now exists that makes collecting a fraction of a cent practical.
Resolving identity, permission, and payment at the edge means one proxy performs those functions for a large share of the web at once [1]. Cloudflare presents that responsibility as an advantage for settling everything inside one request. But it can also be seen as a potential risk when so much runs through a single provider.
Cloudflare also states several limits plainly:
Trust that travels with a request may reach only the traffic that can afford to be identifiable, and small or privacy-sensitive sources of traffic need other building blocks, such as private rate limiting [4].
Usage-based payment helps a site that already has demand, and does little for a small site whose real difficulty is discoverability rather than monetization, which leaves that site weighing visibility against giving content away [4].
Collecting a payment through this exchange depends on callers built to recognize and honor the 402 response. Therefore, the revenue depends on adoption within the ecosystem. [6].
Pricing an outcome rather than a crawl aligns payment with value, but it is also harder to measure and verify. This is the reason Cloudflare frames the shift to Pay-Per-Use as an experiment [3].
Lastly, Cloudflare argues that crawlers combining several purposes under one identity reduce transparency for a site. This is because the site cannot tell why it is being accessed [4]. While the argument is sound based on technical merits, it also aligns with Cloudflare’s commercial interest in separated, verifiable traffic.
None of these costs undoes the shift. They simply mark the current trade-offs that website owners should consider before adopting it.
The web is moving value settlement from after the request to inside it. For most of the web’s history, a request was served free, and value was settled later through human attention. However, agent traffic, which now comprises the majority of requests, leaves that later settlement with nowhere to land [2].
Cloudflare’s response is to resolve four things from its position as a reverse proxy: seeing what a request is through classification, verifying who sent it through Web Bot Auth, enforcing the site’s rules, and settling payment through the x402 exchange, all before the origin responds [1][4][7].
Identity verification lives at the edge today, and the payment gateway opens as a waitlist [1][7]. The open questions around concentration, reach, adoption, and how to price an outcome are worth tracking as the model develops.
References:
Announcing the Monetization Gateway: charge for any resource behind Cloudflare via x402
Content Independence Day, one year on: building the business model for the agentic Internet
Your site, your rules: new AI traffic options for all customers
Introducing pay per crawl: Enabling content owners to charge AI crawlers for access
Launching the x402 Foundation with Coinbase, and support for x402 transactions
Forget IPs: using cryptography to verify bot and agent traffic
2026-08-10 23:30:49
Agents are hitting your signup flow and bouncing off a browser login built for humans. Every one that gives up is a signup you never see.
WorkOS Agent Registration turns that traffic into signups. Enroll via the dashboard and AuthKit publishes an auth.md file agents read to register for scoped, short-lived credentials you control.
What does it take for a social media platform to stop rewarding clickbait content?
At first glance, this question might sound like a moderation problem that you can simply solve by enforcing better content policies and classifying posts that are engagement bait. However, the problem is much deeper. It often sits inside the component that decides which posts become part of a user’s feed in the first place.
Consider the scale of the decision. When you open a feed, the platform has a few hundred milliseconds to select a handful of posts from hundreds of millions of candidates. Scoring every candidate with an expensive model burns through the time budget. To get over this, the social media platforms rely on engagement, which is a cheap and somewhat reliable proxy for judging relevance. Such a proxy is easy to measure and optimize against, and it powered a generation of recommendation systems.
However, the problem is that this type of engagement is also easy to manufacture. For example, a post that opens with “comment DONE if you’re a real engineer” collects clicks and replies while delivering little value. But a ranking function tuned to reward interaction will promote it even though it is clearly an engagement bait. For years, the countermeasures against this were heuristics and demotions applied after ranking, but such an approach hasn’t eliminated the problem. Accounts producing bait always find ways around such measures.
Over the past two years, LinkedIn, Meta, and YouTube, three of the largest platforms, have tried to address the root of the problem. All of them have attempted to rebuild the retrieval stage around the meaning of content, matching posts to people by what a post is about and how it relates to a reader’s interests. The idea is that once the relevance depends on semantic meaning, the tactics built for engagement farming lose their potency. However, the three companies have taken three different directions to solve the same problem, which provides us with an opportunity to understand things from multiple perspectives.
In this article, we will work through the following points:
Why feeds relied on engagement signals for so long, and where that approach reached its limits.
How embeddings match users and content by meaning.
LinkedIn’s consolidation of five retrieval systems into a single language-model retriever.
Meta’s opposite choice of keeping a large family of specialized models arranged as a funnel.
YouTube’s generative approach, where the system produces the identifier of the next item.
The cold-start problem, and why pretraining helps most when a user’s history is thin.
The tradeoffs each design carries, and the limits of the engagement-bait result.
Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.
Every large feed runs on a two-step pipeline:
Retrieval: This step reduces hundreds of millions of candidates down to roughly a thousand. It has to be cheap because it touches the entire corpus.
Ranking: This step spends real compute ordering those survivors into the sequence you scroll.
Most of the recent architectural changes sit in retrieval, so that is where we will focus more.
For years, retrieval relied on behavioral signals. The system recorded which posts a user clicked, watched, and reacted to, then retrieved content that resembled those interactions or resembled the behavior of similar users. This approach scales well and produces reasonable feeds.
However, it also has a weakness.
When a system optimizes a single measurable objective, it tends to optimize the literal metric rather than the intent behind it. As mentioned, engagement is a proxy for relevance, and a proxy can be optimized directly. A retrieval stage tuned on interaction counts will surface whatever maximizes interaction counts, and engagement bait is the content that does exactly that. For example, content designed to trigger a click or a reply scores highly on the measured signal while contributing little to the experience the platform aims to deliver.
Suppressing such content through demotions and rules tries to treat the symptom without fixing the underlying disease. The retrieval stage still keeps bringing up the same material. A more durable fix changes what retrieval measures in the first place, and this happens by moving from behavior to meaning.
The alternative is to retrieve content by its semantic content rather than by its interaction history. The ability to make this possibility depends on embeddings.
An embedding is a list of numbers that positions an item as a point in a high-dimensional space, arranged so that related items land near one another. For example, a post about fixing a leaking tap and a post about reducing water waste sit close together even when they share no keywords, because their meanings are related. In more technical terms, the embedding reflects the relationship captured during model training.
To use embeddings for retrieval, platforms apply a dual-encoder design, sometimes called a two-tower model.
One encoder converts a user, along with their profile and recent activity, into a point in the space.
A second encoder converts each post into a point in the same space.
Since the two encoders operate independently, a platform can compute every post embedding in advance and store it in an index. At request time, it computes the user embedding and runs a nearest-neighbor search to find the closest posts, which keeps retrieval fast across an enormous corpus.
The value of this design comes from what the embeddings encode. A keyword system matches surface tokens, so it links “electrical engineering” to other posts containing those words. An embedding produced by a language model reflects associations present in its training data, which lets it link an “electrical engineer” to other concepts like grid optimization and renewable energy infrastructure, even when the exact terms differ.
This same pattern appears well beyond social feeds. Search ranking, retrieval-augmented generation, and product recommendation all use two-tower retrieval. The model here transfers to a wide range of systems a developer might build.
LinkedIn, Meta, and YouTube all adopt semantic retrieval, but they diverge in how they have built their respective solutions. Let us look at them in more detail.
LinkedIn’s feed previously drew candidates from separate retrieval systems, each with its own index and its own optimization logic [1].
One source supplied a chronological view of network activity, another handled trending posts by geography, another ran collaborative filtering, and several more produced embedding-based candidates. The setup worked, but maintaining five parallel systems led to rising engineering costs. Also, the sources were optimized independently rather than toward a single coherent objective.
In March 2026, LinkedIn replaced those systems with a single retrieval model built on a fine-tuned version of Meta’s LLaMA-3 [1]. The model acts as a dual encoder, converting both members and posts into one shared embedding space. It serves the entire feed through nearest-neighbor search at sub-50-millisecond latency for its member base [1].
However, consolidating five systems into a single language model raised a practical problem.
A language model processes text, while a recommendation system runs on structured features such as view counts, engagement rates, work history, and post metadata. LinkedIn bridged this with a prompt library that converts structured fields into templated text sequences the model can process [1]. See the diagram below:
In this approach, a member becomes a passage describing their profile, skills, and an ordered sequence of recently engaged posts, and a post becomes a passage describing its author, text, and engagement statistics.
We can also take a more general idea from LinkedIn’s approach. When the team fed raw popularity counts directly into the prompts, the numbers had almost no correlation with the model’s relevance scores, because large integers entered the model as arbitrary tokens. Converting each count into a ranked bucket, expressed as a percentage that the model could process in context, raised the correlation sharply and improved retrieval accuracy by roughly fifteen percent [1].
The takeaway from this is that the model is often the part that works, and the surrounding representation of the data is where the real effort goes.
As we can see, consolidation is one answer to solve the retrieval question. However, Meta made a different choice.
Meta arranges Instagram’s recommendation system as a multi-stage funnel, consisting of an ecosystem of more than a thousand models supporting it [3]. Candidates pass through a sequence of stages, and each stage applies a more expensive model to a smaller set of surviving candidates [2].
The funnel runs through four steps:
Retrieval gathers candidates from many sources across the platform.
Early-stage ranking uses a lightweight two-tower model to narrow that set.
Late-stage ranking applies a heavier model to the remaining finalists.
A final pass adjusts the result for diversity and integrity.
Where LinkedIn moved toward meaning by consolidating, Meta pursues it through specialization.
In Meta’s approach, the late-stage model predicts many possible user actions at once, and a value model combines those predictions into a single score [4]. That combination adds weight for positive actions, such as a likely save, and subtracts weight for predicted negative actions, such as a “See Fewer Posts Like This” tap [4].
The basic objective of this extends past raw engagement to include signals about content a user would prefer to avoid.
This design is the one many production recommenders resemble, which makes it a useful reference point. Many competing objectives, including engagement, diversity, integrity, and creator fairness, are easier to tune and audit as separate stages than as one model. Of course, the cost to this is operational complexity, which is the exact complexity LinkedIn set out to reduce.
To summarize, both companies looked at semantic retrieval and made opposite conclusions about how much to consolidate. However, YouTube, the third platform we are looking at, questioned whether retrieval needs to search a stored index at all.
YouTube took a third route that removes the search index from retrieval altogether [5].
The system, called PLUM, assigns every video a Semantic ID, a short sequence of discrete codes derived from the video’s own content [5]. Videos with similar content receive similar codes, so the identifier carries information about the item rather than acting as a random label.
PLUM then adapts a pretrained language model (from the Gemini family) by adding these Semantic IDs to its vocabulary and continuing to train it on video metadata and user activity [5]. After this adaptation, the model performs retrieval as a generation task. Given a user’s recent history, it produces the Semantic IDs of the videos that the user is likely to watch next, decoding several candidate identifiers through beam search, and the system maps each identifier back to a real video [5].
See the diagram below:
This approach, however, introduces a failure mode that the index-based designs avoid, which is generating an identifier that maps to no video, and PLUM reports keeping that rate below five percent after fine-tuning [5].
The payoff from this solution appears in coverage. Measured against the previous production system, PLUM surfaced a far wider range of long-tail videos, and on YouTube Shorts, it raised panel click-through by 4.96 percent [5]. The architecture also inverts where the parameters live. The prior system stored most of its parameters in large embedding tables, while PLUM holds most of its parameters in the network itself [5].
Ultimately, the three designs converge on one payoff in the cold-start problem.
A recommendation system faces the cold-start problem whenever a user arrives with little or no history. Behavioral retrieval has little to work with in this situation, because it depends on past interactions to find similar content, so new users historically saw generic material until their activity accumulated enough signal.
Semantic retrieval changes this.
Since a language model carries associations from pretraining, it can infer plausible interests from a profile alone. A member who lists a role in electrical engineering can be matched to content on grid optimization and energy infrastructure before clicking anything, using associations the model acquired during training rather than signals from that member’s own activity [1].
See the diagram below:
However, this capability also comes with a limit.
An inference drawn from a sparse profile can be wrong, and it can compress a person into a stereotype built from whatever their profile most resembles. The strength and the weakness have the same source, which is that the model fills gaps with prior associations. LinkedIn’s reported results fit this picture. The overall lift from the new system was modest, while the gains concentrated among new and low-connection members, which is exactly the group a meaning-based system can serve when behavioral history is sparse [1].
Every one of these designs has an associated cost and tradeoffs.
The first is consolidation against specialization. LinkedIn’s single model is simpler to maintain and aligns retrieval with ranking, while Meta’s many-model funnel gives independent control over each objective and provides natural redundancy. A single model also raises a more difficult rollback question, because five specialized systems offered five independent places to intervene when one of them regressed on a case such as trending content.
The second tradeoff is related to cost. Language-model embeddings capture richer associations than the lightweight methods they replaced. However, they consume more compute to produce and serve. As LinkedIn’s popularity-count finding showed, the model is rarely the bottleneck, and the load moves to the data pipeline, the feature representation, and the serving path.
The third tradeoff sits between the generative and index-based approaches. Generative retrieval composes items from compact codes and removes the large embedding table, at the cost of a failure mode where the model can produce an identifier for a video that does not exist. Index-based retrieval avoids that failure mode, but requires storage and maintenance of the index.
Semantic retrieval reduces the leverage of bait rather than removing it, since a system can still be gamed by content engineered to match high-value topics. Also, these three designs are points of emphasis on shared scaffolding more than pure types. YouTube still ranks candidates in stages, LinkedIn’s retriever feeds a separate ranking model, and Meta continues to explore generative methods.
Social media feeds are transforming retrieval approaches from behavioral engagement to semantic meaning. This helps reduce the leverage of engagement bait as a property of the architecture.
However, the three largest platforms adopted the same underlying idea and built it in three ways.
LinkedIn consolidated five retrieval systems into a single language-model dual encoder and searches one shared embedding space.
Meta kept a large family of specialized models arranged as a staged funnel with a multi-objective value model.
YouTube generates the identifier of the next item through an adapted language model and removes the retrieval index.
The reason these choices differ comes down to data.
A text-rich professional network, a multi-objective media platform, and a video service with an enormous item corpus each favor a different design, and each team optimized for the structure of their specific data.
References:
[1] Large Scale Retrieval for the LinkedIn Feed using Causal Language Models
[2] Scaling the Instagram Explore recommendations system
[3] Journey to 1000 models: Scaling Instagram’s recommendation system
[4] Powered by AI: Instagram’s Explore recommender system
[5] PLUM: Adapting Pre-trained Language Models for Industrial-scale Generative Recommendations
2026-08-06 23:31:50
Every application built on stored data performs two kinds of operations against it.
A write operation records a fact, such as a new order, a changed email address, or a deleted comment. On the other hand, a read operation answers a question, such as which orders were placed this week or what should appear on a profile page. A single database on modest hardware can handle both types of operations without much trouble, and a developer need not be bothered about which operation is more common in the application’s context.
However, high traffic can change things. Let’s say a page starts to load slowly. A fix is identified, which is creating an index on the column being filtered. Now, several months later, the same page is slow again under higher load. This time the fix is a cache in front of the query. A year after that, the database saturates during peak hours. This time the fix is a read replica with reporting traffic routed to it. In principle, each fix works, but each one is needed at a different time and for a different reason.
Now, let’s assume that a user updates a profile, reloads the page, but still sees the previous value. The bug does not reproduce locally, and it disappears on its own before anyone can investigate its root cause. However, this behavior can be a direct consequence of some of the other fixes. This is because each of the previous fixes placed a copy of some data somewhere other than its source. But the copy is not updated in sync with the source update. In other words, a seemingly simple fix on the read path can impact how things appear to work on the write path.
In this article, we will look at read path and write path operations and techniques in detail. Here’s what we will cover:
Why fast reads and correct writes require opposing data structures
Precomputation and duplication, the single operation underneath every read optimization
Two different definitions of consistency, and the bugs caused by treating them as one
Indexes, denormalization, caching, read replicas, materialized views, purpose-built read stores, fan-out on write versus read, and CQRS
For each strategy, its sync mechanism, staleness window, and characteristic failure mode
Write-heavy systems, where the ratio inverts and the decisions reverse with it.
2026-08-05 23:30:28
AI is in your engineering workflow. While the token spend shows it, the throughput doesn’t. The human is very much still in the loop, and that’s a context problem.
Join live on Aug 19 (FREE) to learn:
The 4 metrics to measure where AI gains leak out before production.
The 8 stages of context maturity, the specific walls capping your metrics, and a free tool to pinpoint where your team is
Why more MCPs and bigger context windows aren’t enough, and what it takes to get real value from your agents.
The most capable AI models are also the most expensive to run. They need specialized hardware, they consume large amounts of memory, and they add cost and delay to every request they handle.
These traits make them hard to deploy in places where resources are limited, such as a mobile device or a service that handles heavy traffic and needs fast and low-cost responses.
There is also a second fact that sounds backward at first. A small model can sometimes match or beat a much larger model on a specific task, even when the small model learned everything it knows from the larger one. On an intuitive level, a model trained on another model’s output would seem to inherit a ceiling rather than break through it, yet the results are different. The method that makes this work is called knowledge distillation, and it has become a standard part of how production AI systems get built.
In this article, we will walk through the idea from the ground up. The main points we will cover are as follows:
What distillation is, and how it differs from compression.
Why learning from a model’s output can beat learning from raw labels.
The three main methods, and which one dominates?
What distilled models achieve in practice.
Where the method breaks down, and where it is heading next.
Distillation trains a new, smaller model to copy the behavior of a larger one. The setup involves two models:
The first is a large, capable model called the teacher.
The second is a smaller model called the student, which is trained to reproduce the teacher’s outputs.
Once training finishes, the student runs on its own, and the teacher steps out of the picture.
A common assumption is that the student is the teacher in compressed form. The reality, however, is different.
Compression methods such as quantization and pruning start with one model and reduce its footprint by storing its numbers at lower precision or removing parts that contribute little to the result. The model stays the same model, smaller and lighter.
Distillation, on the other hand, produces a genuinely separate model, with its own parameters and often a different design, whose goal during training is to behave like the teacher.
One operation shrinks an existing model. The other trains a fresh one. The payoff is practical, since a small student can run inside a single service or on a phone, respond in less time, and cost far less for each request, and in some cases, it can run on the device itself without sending data elsewhere.
This method is now standard practice. For example, Google’s Gemma models are built using distillation during training, drawing on a larger model in the Gemini family. The two ideas also work together in sequence. A model is often distilled first to produce a smaller capable model, then quantized to shrink that model further for a specific device.
Keeping the distinction clear matters because it affects how we understand further concepts. A compressed model carries a copy of the original inside it. A distilled model is a separate thing that was trained to act like the original, which is exactly why it can sometimes behave in ways the original would not.
If the student only copies the teacher, why does copying work so well?
The answer is in what the teacher hands over.
Learning from a model’s output beats learning from raw data because the output carries more information than a plain answer.
Standard training data gives one answer per example. An image of a cat carries the label “cat,” and the model is rewarded for producing “cat” and penalized for anything else. A teacher model offers something richer. Instead of a single answer, its output is a set of probabilities across the options, such as cat at 0.70, dog at 0.25, and fox at 0.05. That full set of probabilities is called a soft label, in contrast to the single hard label found in ordinary data.
The extra numbers carry additional information. They show that the teacher’s output ranks dog as a plausible alternative and fox as a distant one, which says something about how the categories relate to each other. Researchers sometimes call this dark knowledge, meaning the structure hidden in a model’s confidence that a bare label leaves out.
During training, the student works to match this distribution. It is scored on how far its own probabilities sit from the teacher’s, and training pushes it to close that gap. In other words, the student learns the teacher’s whole pattern of confidence rather than a single right answer, and that pattern is a stronger training signal than a one-word label.
This is the core reason distillation works as well as it does. A single correct label discards the relationships between options, and soft labels keep them.
An early result showed the practical payoff, since a student could reach good performance from far fewer examples when trained on soft targets, because each example now carried more than a single answer. The original 2015 work added a control called temperature for exactly this purpose, where a higher temperature spreads the probabilities out and exposes more of that fine structure for the student to learn.
With the mechanism clear, the next question is how this gets done in practice, which has more than one answer.
Distillation comes in three main forms, and they differ in what the student copies:
Output distillation: The student matches the teacher’s final outputs, including the soft labels described above. This is the original form from 2015 and the most direct one.
Feature distillation: The student matches the teacher’s internal representations, meaning the intermediate values a model computes while processing an input, before it settles on a final answer. The aim is a similar internal picture, not only a similar output. Google’s EmbeddingGemma is trained this way, learning to produce internal representations close to those of a larger Gemini model.
Synthetic data distillation: The teacher generates a dataset of examples, and the student is fine-tuned on that dataset the same way it would be trained on any ordinary data. Stanford’s Alpaca was an early case, fine-tuned on examples produced by an existing large model to improve how well it followed instructions.
The third form has become the most common approach in practice, and part of the reason comes down to access.
Many strong models are reachable only through an interface that returns text, with their internal values and probabilities kept private. When those internals are out of reach, generating data is the route that still works.
The three forms also differ in what they require. Output distillation needs the teacher’s probabilities, feature distillation needs access to its internal values, and synthetic data distillation needs only the text the teacher produces, which is why it travels the furthest across closed models.
These methods can also be combined. A single training run might use a generated dataset alongside soft labels, and newer methods mix teacher and student generation during training.
These methods are not only theoretical. The next section shows what they produce.
The results in practice are strong, with one important qualifier.
A clear example came in early 2025 from a lab called DeepSeek. It used a large reasoning model to generate a set of training examples, then fine-tuned several existing smaller models on those examples. One result stood out.
A 7-billion-parameter student scored higher than a 32-billion-parameter model on a competition mathematics benchmark, even though it was produced by plain fine-tuning on the larger model’s outputs. The released family of distilled models ran from 1.5 billion parameters up to 70 billion, and the smaller ones were compact enough to run on a single graphics card, which is part of why the release drew so much attention. The practical effect was that strong performance on these narrow tasks became something a small team could run locally and cheaply, rather than only through a large hosted model.
The qualifier matters as much as the headline.
These wins tend to appear on narrow, well-defined tasks such as mathematics and code. On those tasks, a small distilled model can perform at a level its size would not suggest. Across broader measures of general knowledge, the same small models still trail the larger ones. For example, a model can become excellent at competition mathematics through distillation while remaining weaker at wide-ranging questions about the world. Therefore, a claim that a small model beats a large one is usually true in a specific, narrow sense.
If the results are this good, the natural question is where the method falls short, which the next section takes on directly.
Distillation has clear limits, and they matter when deciding whether it fits a given problem.
A ceiling effect from the teacher: A student trained on a teacher’s output tends to stay at or below the teacher’s level on the kind of data they saw. When the teacher produces a wrong answer, the student learns that wrong answer along with the right ones. The teacher’s quality sets the bar, which makes the choice of teacher one of the most consequential decisions in the process.
A wider gap can hurt: A larger, stronger teacher does not always produce a better student. When the gap between teacher and student is very wide, transfer can degrade, because the student has too little capacity to absorb everything that a much larger model expresses. Research on this capacity gap has found that the strongest available teacher is sometimes a poor choice. A set of methods exists to bridge wide gaps by adding a middle step, where the teacher trains a mid-sized model and that model trains the small student, so each handoff spans a smaller distance.
Architecture can outweigh size: The design of the base model can matter more than its parameter count. In one study, a 32-billion-parameter student outperformed a 70-billion-parameter student on the same task, because the smaller one was built on a stronger base architecture. Size alone is a weak predictor of how well distillation will go.
The teacher can pass on more than the task: In a 2025 study later published in Nature, a teacher model with a particular trait, a tendency to favor owls, was used to generate training data made up only of number sequences. A student trained on those numbers picked up the same preference for owls, even after the data was filtered to remove any visible trace of the trait. The same effect appeared with more serious behaviors, and it occurred only when the teacher and student shared the same base model. The takeaway is that distillation can carry across more than the task being taught, and that filtering the visible data is sometimes too coarse to stop it.
These limits set the boundaries, and within them, the method keeps advancing. The next section covers where it is heading.
The newest direction in distillation reduces the manual effort by automating the whole process.
In this setup, the large model runs the full loop on its own. It generates training data, fine-tunes the student, evaluates the student against a held-out set of examples it also generates, and repeats the cycle, adjusting what it produces until the student stops improving. The human role shrinks to defining the task and the success criteria at the start, with a final check on real data at the end.
Recent work in 2026 applied this to a detection task and found that it worked well, with one finding worth keeping in mind.
The choice of teacher model had a large effect on the outcome. Different teachers, given the same loop and the same student, produced students of noticeably different quality. So automation removes manual effort while making the initial choice of teacher more consequential, since that choice now drives an entire self-running process rather than a single training pass.
The same loop also points toward less hand-built pipeline work over time, as more of the data generation and evaluation moves to the model itself. For a team, the appeal is building a small, task-specific model without assembling a large hand-labeled dataset first, since the teacher supplies both the training examples and the data used to score them.
Distillation is a method for training a small, deployable model to copy the behavior of a large, expensive one. It produces a separate model rather than a compressed version of the original, and that distinction explains most of how it behaves.
It works because a model’s output carries more information than a plain label, in the form of soft labels that show a full pattern of confidence across the options.
In practice, the most common form has the teacher generate a training set that the student learns from, and the results can be strong, though usually on narrow tasks.
The limits are real:
The teacher sets a ceiling,
A wider size gap can hurt rather than help,
Architecture can outweigh size
The process can carry across traits that were never intended.
Taken together, distillation tends to be a good fit when the task is well defined and a capable teacher is available, and a weaker fit when the goal is broad, open-ended capability.
References: