2026-11-13 00:30:41
When you create an architecture diagram, you usually try to show the services, databases, queues, and the connections between these different components. With the help of these diagrams, developers can understand how a request goes through the system and how the data flows. We also get to know which components depend on other components.
However, these diagrams cannot show all the things that a component might depend on. For example, let’s say you have two services with separate code and API endpoints. At first glance, you might think that these services are independent. But imagine that whenever you change one service, you need to make a change to the other service as well. Or maybe both of these services talk to the same database whose capacity can be consumed by either service. Perhaps both of them rely on a common logging system.
You can think of these dependencies as hidden forms of coupling. If not handled properly, hidden coupling can have a big impact on the health of your system. Also, it can affect how you change things in your system, or how deployments are done. Or how recovery happens. In this article, we’ll look at the various forms of hidden coupling and how you can deal with them. Here’s what we will cover:
The various dimensions of coupling
Temporal coupling
Semantic coupling
Deployment coupling
Shared schemas and libraries
Observability coupling
Shared incidents
2026-11-12 03:58:17
Enrollment closes in 24 hours for Build Production Grade AI Systems. We kick off tomorrow, November 12, at 5pm PT.
Join Tanya Roosta, Director of AI at AMD and UC Berkeley PhD, for live, hands-on sessions covering how to:
Turn prototypes into deployable AI services.
Build RAG pipelines using company documents.
Reduce inference latency and costs.
Evaluate outputs and monitor quality in production.
Add security controls and scale with confidence.
2026-11-12 00:30:53
Writing code is fast, shipping it is still hard. Railway’s push-button compute, storage, and networking is built for both small and hyperscale software.
Humans and agents alike operate on one vertically integrated system on our own hardware. Get superior speed, better economics, and a lot more calm.
Sign up with the link below for free cloud credits and let your agents cook.
The Roundup covers insights and trends from the world of software development.
Today, we’ll cover the following topics:
More and more Agents are writing the 1st draft of most new code. According to GitHub, more than 40% of PRs in September have originated with an agent doing the work.
Anthropic has gone as far as replacing Git to keep up with a constant stream of changes.
Anthropic’s S1 filing gives an inside look at the real worth of AI coding.
Beyond AI, the very foundation of the World Wide Web has had a pretty rough time when it comes to security. PHP had to ship emergency fixes for two critical remote-code-execution bugs that attackers were already exploiting.
Disclaimer: This post is based on publicly shared details from various sources. Please comment if you notice any inaccuracies.
Git was designed in 2005 for the Linux kernel. At that time, you had 1000s of human developers. Each of them committed code a few times a day. Patches were swapped over email. For a couple of decades, these things were the normal way of doing stuff because writing code was a slow exercise.
This month, two data points have turned up that show that these norms are changing. GitHub’s Octoverse 2026 reports that more than 40% of pull requests in September started as a first draft using agents. This is up from 17% in March. For perspective, this comes to around 110 million PRs in a single month that have been started by agents.
Also, Anthropic recently released an article talking about how they replaced Git in its monorepo with an in-house system known as Scion. Daily merges in Scion grew from about 2500 in May to more than 30000 in September. A team of 7 developers moved the whole monorepo off Git in five months. The team says that the new setup uses 200X less CPU and memory than the Git mirror fleet it replaced.
Of course, there are still some questions that numbers might not answer adequately.
First, who is reviewing 110 million PRs a month? Octoverse says 90% of agent-involved PRs were merged within 2 weeks. You can think of faster merging as a sign of better code. But it could also mean that reviews are not good enough.
Second, how is GitHub even able to find out if something has been started by an agent? You can also have agents that commit stuff using a human’s account without any trace of their real identity. So it’s possible that the true share of agent work is even higher.
The third question is about Anthropic’s move to Scion. Of those 30000 daily merges, how many were later reverted? Throughput without a defect rate can’t give a complete picture.
My take is that building your own version control tool used to be a sign of a huge company because doing so required a ton of resources and bandwidth. For example, Google built Piper, and Meta built Sapling, for tens of thousands of engineers.
Anthropic solved that scale problem with a fraction of the people. They were able to do so because agents did all the coding. This is where the change has happened. Generating code is now the relatively cheaper part of software engineering. The scarce resources are time for human code review, CI capacity, and merge rate. I expect the next wave of developer tooling to be about verifying code and not writing it because that’s where the bottleneck is shifting.
Getting an agent to work in a perfect demo environment is easy. Keeping it working in the messy real world is where things get… interesting.
Get a live technical demo of Redis Iris, the unified, real-time context engine. This free session explores six capabilities that keep your agents resilient at scale.What you’ll learn:
Give agents relevant context and memory
Route requests by intent with reusable playbooks
Set up a single context entry point
Govern what agents remember and forget
Trace failures to their source with less custom glue
Come build agents with the context, memory, and messaging they need to stay accurate, accountable, and efficient in prod at scale.
Join live November 12 at 9 AM PT.
Until now, using an AI coding tool meant you had to choose a model. For example, you might select Claude for one task, GPT for another, and Gemini for something else. With the greater adoption of a model router, you can get rid of this choice. The router looks at a task and decides which model (or which mix of models) should handle each step.
At GitHub Universe, GitHub made its router, Copilot Fusion, generally available. GitHub says that Fusion Max scored within two points of Claude Opus 5.5 Max on the Artificial Analysis Coding Agent Index, at about half the cost and 40% less time per task. GitHub also shipped Copilot projects, which run agents across multiple repos in parallel.
Cloud agent sessions can now go between your phone, VS Code, the CLI, and the web. So developers can basically handle their work on any device. Also, GitHub Enterprise customers can use every Copilot feature without any separate subscription. They only need to pay for the AI credits that they consume.
Now, this development opens some interesting thoughts. Earlier, the choice of the model was with the developer. But now it’s decided by another process.
Can you now see which model handled which part of your code? Many teams have rules about where code is allowed to go. But that might be decided by a router silently.
In my view, this is pretty similar to the cloud adoption story repeating itself from years ago. For example, AWS moved organizations from owning servers to simply paying for usage. With seat-based licensing, AI spend became predictable and difficult to optimize. With usage-based billing, you make it variable. But agents that might run for hours can make things quite spiky. And hence, teams need to be a lot more thoughtful in building AI workflows.
Strategically, Fusion aims to make GitHub a model-neutral layer above the labs. This turns your models into interchangeable parts that sit behind a router. In a way, this might be good for buyers, but it might also be a little uncomfortable for any AI lab that counted on their model being the default choice. It also raises the pressure on GitHub’s reliability because a platform that runs your agents has to always stay up.
An S-1 is the document a US company has to file before it can get listed on the stock market. Unlike a funding announcement, it contains audited financials and a long list of things that might go wrong. This is quite interesting because till now, the economics of AI coding tools were mostly guesswork based on leaks. This filing helps replace some of the guesswork with real numbers.
Anthropic filed its S-1 on October 26 under the planned ticker “ANT”. Revenue for H1 2026 was $16.3B, up from $1.2B a year earlier. It has 14K+ enterprise customers, and about 250 of them are spending $10M+ a year. Also, trading could begin as soon as November. There are reports of an operating profit in Q2 and 900% net revenue retention: existing customers spent nine times more in Q3 2026 than in Q3 2025.
Recently, Anthropic also shipped reporting tools that show customers what their token spend is giving them. They also provided pooled team budgets.
There were also some important disclosures. Four models were able to reach external systems during 3rd-party security testing because a misconfiguration left access to the internet open. Anthropic also seemingly delayed pre-training a new model by more than a month to strengthen protection of its model weights.
A key question is about how much of that 9X growth per customer comes from agents running for hours, versus more people actually using the model. Also, how much of the experimental budget could be cut next year? And it’s also quite possible that the decision to launch ROI dashboards the same week as the IPO filing suggests that customers are already seeking details about what they are getting in return for their spending.
In my view, for engineering leaders, the message is that AI spend will be scrutinised pretty much the same way cloud spend was scrutinised in 2022. It’s good to start measuring outcomes now, such as cycle time, escaped defects, and review load.
Also, the disclosure about the model reaching external systems requires developers to be careful. If a well-resourced, safety-focused lab could face something like that, other companies should be more wary of their AI agent’s internet access settings.
Most security advice targets your dependencies. For example, the npm packages or the WordPress plugins. But the language runtime is a dependency too. And it’s something that’s often forgotten. If you have a bug inside the PHP engine, you can’t escape its impact with framework updates or CMS updates. Every app on that server inherits the hole.
Recently, PHP shipped emergency releases for two critical bugs. They are now nicknamed “PHP2Shell”. One of these bugs corrupts request state when memory runs out. The other one lets a later request reuse state that should have been freed. Both are rated 9.2 (CVSS v4) and allow unauthenticated remote code execution.
Apache mod_php is the worst case. The demonstrated attack runs before any application code is executed. Various researchers reported the presence of these bugs. In fact, details leaked on Hacker News two days before the planned disclosure. So PHP brought its release forward a day.
The problem is that the attack traffic came before the fix. Anyone who waited for the KEV listing had already been exposed for nine days.
As per estimates, around 2.6M internet-facing services were showing a PHP version in October. About 1.9M of them (73.5%) ran PHP 8.1 or older, which PHP doesn’t patch. PHP did publish minimal source backports for 7.4, 8.0, and 8.1, but it might be unrealistic to build PHP from source on a forgotten shared host? In fact, attempts to exploit also showed up on Amazon honeypots 3 days before the leak. Did someone have the details about this weakness even earlier?
The researchers may have used AI to help find the bugs. If AI can make old and deep bugs cheaper to find, the patch window is going to go down for everyone in the industry. The cost of running an end-of-life runtime goes up in a big way. In my view, the key lesson from this incident is that developers should also list their runtimes just like the dependency packages. Most dependency scanners don’t worry about the interpreter.
2026-11-11 00:30:55
AI is moving fast, from experiments to agents that act on their own. Few people have a better view of what that shift means for business than Lynn Martin, President of NYSE Group. At the Agentic Data Summit on December 9, Lynn joins Redpanda Founder & CEO Alex Gallego for a conversation on what it takes to adopt AI where reliability is everything, how AI is showing up in the companies coming to market, and why timely, trustworthy data matters more than ever as agents start acting in real time. Tune in at 9:19 AM PT. The summit is free and virtual and runs 8:30 AM–2:00 PM PT.
GitHub recently rewrote the software engine behind multiple Copilot products. They ported the codebase from TypeScript to Rust. Almost 800K lines of code. And the great part was that all of this was done using the GitHub Copilot app itself.
This porting exercise was triggered because the existing design was eating up a lot of resources. It was also becoming difficult to embed Copilot in other applications.
However, a project of this size would have required a big team of developers and a couple of years to get done. But by using the GitHub Copilot app, Copilot CLI, and AI agents, a single developer was able to finish the whole thing in a few months. In this article, we’ll look at how this porting was done and the learnings along the way. Here’s what we will cover:
The need for porting the Copilot runtime
The porting strategy
How the porting work started
The role of agent fleets
How the new code was reviewed
What were the issues
The performance impact
Cost of porting and learnings
Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.
You can think of GitHub’s Copilot runtime as the shared software engine that takes care of AI agent behavior across multiple products. Due to this runtime, all these products can reuse the same capabilities instead of needing to maintain their own implementation.
Originally, the GitHub engineering team had built the runtime using TypeScript that ran on Node and the V8 JavaScript engine. While this supported rapid development, the runtime was tightly coupled to the command-line application’s user interface. When GitHub needed an SDK to let other applications talk to Copilot, it built that SDK on top of the existing command-line application. When you started the SDK client, you had to launch a separate background process that was made up of the command-line application and its runtime. Requests and responses moved between the consumer app and that process.
This setup became harder to justify when the runtime spread to applications that had stricter requirements for speed, memory usage, and scale. Even if you had an application written in Python, Java, or C#, it had to hold the additional Node and V8 machinery. When you started a client, you had to launch another process and prepare JavaScript for execution. Each client had to bear roughly 100 MB of memory overhead at a bare minimum. Also, processes talking to each other added a lot of overhead. Lastly, applications had to monitor two processes, and a crash in the Node process could take the session with it. These costs were a huge problem for services that hosted many clients.
Therefore, GitHub wanted a standalone runtime library that was clearly separate from the UI. Applications should be able to load that library directly into their own process.
Rust was an ideal choice for this type of requirement. It had low overhead and predictable resource usage. An early estimate showed that roughly 130K lines of production TypeScript had to be ported. But additional logic moved out of the interface layer into the runtime. Also, you had other developers adding new features. Ultimately, 430K lines of production TypeScript code moved through the migration.
New hires, support, PMs, and your coding agents all need to know how the product works and why it works that way. Today, you’re the one pointing them in the right direction.
Unblocked works out what’s current and relevant across your code, PRs, tickets, Slack, and production data, so:
Everyone gets cited answers in Slack without pulling in an engineer
When the answer turns into a fix, a cloud agent drafts the PR
Your agents generate code that follows past decisions, no steering needed
GitHub used an incremental, in-place porting strategy.
In this strategy, you replace the runtime one piece at a time, even when the product is under active development. The development team could continue to add new features and release updates throughout the migration.
The GitHub engineering team didn’t go for a complete rewrite followed by one large switchover. This approach would have concentrated all the risk into a single release. The replacement would need to keep catching up with oncoming changes. With the smaller replacements, they were able to make each change easier to review, test, and investigate.
The process worked in a step-by-step manner. A component’s TypeScript implementation was replaced with Rust in a single pull request, which grouped the changes for review. Any TypeScript code that depended on that component called the new Rust implementation through a thin wrapper. You can think of this wrapper as a temporary bridge between two languages. The old implementation was deleted as part of the same change, so that the team had to maintain only one active version of the component. As more and more stuff moved to Rust, these temporary bridges were also removed.
As far as the tasks were concerned, the work began with small, independent helpers that already had strong test suites and didn’t manage shared state. These early changes also helped test the build setup, packaging, and communication between languages. Once they had made the approach concrete, the migration went to more interconnected parts of the system.
Throughout this process, the key aim was to maintain existing behavior and algorithms. Broader redesigns were pushed after migration. At each step, validation was performed. They had existing end-to-end tests that checked if complete workflows behaved properly as their implementation changed. If there was a test failure, the changes weren’t merged. Also, when it came to the users, the ported components reached users through incremental releases. They often appeared in a pre-release before going to a stable release.
GitHub started with a few small, working examples to test the waters before dealing with larger components.
The first couple of pull requests took care of the basic Rust setup. Things like build tools, automated checks, and coding instructions. They also introduced the runtime library and mechanisms for TypeScript and Rust to talk to each other.
The first main porting change moved three independent helper functions through the complete process. You can think of these early efforts as checking whether the translated code worked well. The team also tested how Rust code would be organized, built, packaged, and integrated with existing code. This gave the team practical experience with the entire process.
The rest of the work was ordered according to dependencies. For example, you had small helpers and supporting utilities come first. It was followed by components that managed state, then tools, model clients, and other integrations.
Session orchestration, which is supposed to manage much of the runtime, was left for the end. This is because larger subsystems often needed several rounds of work.
Agent fleets played a key role in dividing the migration work among multiple AI agents working at the same time. You could have a coordinating agent investigate a large component, break the work into smaller tasks, and create other agents to handle them. The Copilot app made it possible to check these sessions and switch between them. It also let them exchange messages as the work progressed.
The team got two different types of assistance from these agents. First, you had the subagents that worked within the parent session’s workspace and returned their findings to it. These subagents were good for investigating code and answering specific questions. Then, you had the child sessions that had their own Git branches and separate working copies of the repository. This lets them make changes independently. This separation helped multiple agents work on different areas of the implementation without any risk.
One example was around the session[dot].ts file that was roughly 30K lines and connected to many parts of the runtime. The coordinating agent spent an hour reading and checking before creating anything. After analyzing, it divided the work among 15 child sessions. It made the choice of models for each task. The parent talked with these sessions and tracked their progress from beginning to end. Finally, it brought the completed changes into its own branch.
Despite the advantages, running so many agents also created some limitations. When all 15 sessions tried to build and test their code at the same time, they overwhelmed the developer’s laptop. The workflow was adjusted so that child agents avoided expensive builds and tests. All that work was left for the parent. A separate chat session took care of build access for each of the porting sessions. It only allowed one build at a time.
Much of the agent fleet’s activities involved reading code, checking for dependencies, testing, and reviewing results. Their role wasn’t limited to writing Rust code. They also helped understand the existing system and check whether the replacement kept the same behavior. During all of this, human oversight remained critical.
Developers were constantly changing the repository, and hence, code review had to check both the translation and whether the port had the latest changes. The team used a reusable instruction called “rust-rebase-review” to handle this process.
First, the main agent pushed its work to the latest main branch and resolved any conflicts. After this, it checked that any newly incoming TypeScript changes had also been ported into Rust. Deleting the old TypeScript helped bring such changes to attention. If another developer modified a deleted file, Hit would flag a conflict that had to be handled.
Next, you had multiple review agents using different AI models compare the old TypeScript with the new Rust code line by line. Their main job was to identify changes in behavior or compatibility. They also checked whether the Rust implementation was efficient enough. Also, whether obsolete TypeScript and temporary wrappers had been removed. They also had to ensure that existing end-to-end tests remained unchanged because they were the only reference for expected behavior.
The code review was done in a repeated cycle. When agents found any issues, the coordinating agent checked the findings and fixed any valid problems. This continued until the reviews became clean. Every round of fixes could be committed and pushed so automated checks could be done along with further reviews. Copilot Code Review and additional review bots also checked every commit and left feedback on the pull request.
Human reviewers focused on architecture, interface contracts, and risk factors that might have been highlighted by agents or tests.
The porting exercise caused some regressions as well. You can think of these regressions as behavior that had worked earlier but stopped working as expected after the rewrite. The team found and fixed a bunch of regressions.
A common problem was that translated code changed the existing behaviour in a subtle manner. For example, some whole-number values were changed to decimal numbers, causing issues with certain SDKs. Other differences were around empty strings, time zones, or replacement libraries that had a different way of handling errors. The code could compile successfully. But the results could not be handled by other parts of the system.
Another major group of issues was around state management and coordination of operations. For example, when an operation was cancelled, it didn’t always stop all the related work. Also, when resources were released too early, the session got stuck. There was also a loss of some features. And some of them were only partially migrated.
There were also general integration and performance problems. For example, the Rust code sometimes performed lengthy work on Node’s main thread. This froze the interface. On Windows, missing process-launch settings made unwanted console windows appear.
The rewrite brought down the overhead of running Copilot’s agent runtime. Especially in terms of time and memory. With the use of Rust, there was no need to initialize Node and V8. Also, no need to load and prepare the JavaScript application. Applications could load the Rust runtime directly into their own process. This avoided the cost of talking with a separate runtime process.
GitHub saw performance gains during their detailed testing. For example, in a particular trial, creating a client, completing a single turn, and cleaning up took 5.25 seconds before the porting and 292 milliseconds after porting. This was an improvement of almost 18 times.
They also had capacity gains. In a test where many sessions shared one client, throughput went up from 7.55 completed session lifecycles per second to 120. Memory consumption also went down. In a 10-client test, the peak increase in private memory went down from 1383 MB to 126 MB.
The rewrite largely preserved the original algorithms. More improvements could happen if parts of the runtime are redesigned around Rust’s approach to memory ownership and concurrency.
The entire porting work cost around $120K in AI token costs. They covered roughly 136 billion tokens. Most of this volume was made up of cached input, which brought down the cost of repeatedly processing existing context.
More than that, you also had developer time that was spent on this work. Apart from the main developer, other developers also made contributions to integrations, packaging, build improvements, and reviews.
There were a lot of learnings along the way.
For one, it was critical to give clear instructions to the AI tools. Some of the initial requests to port a component sometimes led agents to translate only the central logic. The surrounding work stayed in TypeScript. The instructions had to be made more effective to handle this.
Another learning was that to perform such a migration successfully, you need strong end-to-end tests. These tests should check if complete workflows retained their original behavior. The team improved its test coverage beforehand. But additional coverage could have prevented some more regressions.
The tests also needed protection. For example, you don’t want an agent changing an implementation to make the test weaker or alter an expected result. So, you need to have some sort of oversight.
One more learning was that for such migrations, you should try to translate things first without worrying about redesigning. It’s more important to keep existing behavior and algorithms intact. If you attempt to combine the porting with other improvements, it only increases issues, time, and token costs.
References:
2026-11-10 00:30:44
The challenge of running LLM applications in production is that failures are silent, costs are unpredictable, and security vulnerabilities are hard to detect until it’s too late.
Download this Datadog eBook for practical strategies to monitor, debug, and secure your LLM application workflows end-to-end. You’ll get 4 key insights into how to overcome the challenges of deploying LLMs at scale — from tracing multi-step chain executions and detecting prompt injection attacks to tracking token costs and evaluating output quality in real time.
LinkedIn has more than 10,000 active code repositories, with tens of thousands of pull requests (PRs) being opened each week [1]. As coding agents write more of that code, the slow part of shipping software has moved from writing code to reviewing it.
To overcome the code review bottleneck, LinkedIn built its own AI reviewer. In a typical week, this in-house reviewer covers more than 40,000 PRs, and most of its reviews are posted before the first human reviewer even opens the PR [1].
To better understand the landscape of AI reviewers and how LinkedIn built theirs, we interviewed Min Chen, a Principal Staff Engineer at LinkedIn, and Balaji Srinivasan, Senior Director of Engineering at LinkedIn. Thanks to both of them for their time and for being open about what did not work.
In this article, you’ll learn:
What an AI code reviewer is, and why review is the bottleneck these days
Three main directions for teams and companies to handle the code review bottleneck
How to build a multi-agent AI reviewing system that works in production
What evaluation looks like for AI reviewers
What’s ahead and where the new bottlenecks move
Software development has two main steps. An engineer writes a change, and other engineers review it before it merges into the main codebase.
When an engineer writes the code, the change becomes a PR. Then reviewers read its diff, that is, the lines it adds, removes, or changes. Here are the typical steps from when a writer opens a PR to when the PR is merged back into the codebase:
The author opens a PR from a branch that has the change.
The code host, such as GitHub, notifies the reviewers and starts continuous integration (CI), which runs tests and linters (to flag style problems and simple errors).
A reviewer reads the PR and comments on specific lines.
The author pushes fixes. Then the reviewer looks again.
The reviewer approves, and the PR merges.
AI is changing this development process, because it can now help with some of these steps. Engineers hand those steps to AI agents and spend more of their time focusing on intent and judgement. That is AI-native development.
Specifically for the writing step, frontier coding agents have shown promise lately. Given a task, an agent implements the feature, writes the tests, then runs those tests, and iterates until they pass. Finally, it opens a PR for review.
With coding agents, writing (the first step) can now be largely automated and done much faster. Instead of an engineer spending hours or days writing a change, the agent spends minutes writing it. But if real humans still have to review the proposed changes, the process will still be slow, because engineers’ hours are still fixed and limited.
The fix is a system that can reliably review code, ideally before humans do. That system is called an AI reviewer. It reads a PR the way a human reviewer would and comments on the lines that need attention. Once the automated code review is complete, the author, in this case the coding agent, addresses them so human reviewers spend their limited time on judging the fixes.
Since models are also getting better at reading code, more teams are automating reviews this way [2]. They mostly differ in the extent to which they aim to automate the reviewing process. Full automation would let AI approve changes on its own. This is difficult, and most labs are not there yet.
Partial automation keeps human approval on changes and gives AI the first pass. This makes the process faster but keeps real humans in the loop to reduce the risk of merging buggy or insecure code. Partial automation is more feasible these days, and companies that have an AI reviewer in place, like LinkedIn, usually use partial automation. In the rest of this article, we focus on partial automation.
But why is building an AI reviewer challenging? For an AI reviewer to be reliable, it has to have four properties:
Grounded: it flags real problems in the changed lines.
High-signal: the issues it produces are real bugs and high-risk lines (not just minor style nits a linter would catch).
Codebase-aware: it follows the codebase’s own conventions.
Timely: it comments while the author can still act on it.
Getting all four at once is the hard part. LinkedIn built its own system over several iterations to get there. Let’s see how they built it.
AI agents open more pull requests, and each one reinstalls the same dependencies from scratch. That is wasted CI time and money.
WarpBuild keeps that work:
Snapshot runners boot from a saved machine image, with your toolchain already on disk
Each job still gets a fresh, ephemeral runner
The WarpBuild cache restores dependencies between jobs
Twice as fast at half the cost of default runners
Trusted by teams at Braintrust, Raycast, OpenRouter, Comcast and Jane Street.
Use code START50 for $50 in free credits.
Remember from earlier that the goal is partial automation. PRs still get a human review, but the AI review comes first. As a company, you have different options to have an AI reviewer in place. Some options are easier to set up but less flexible, while others take more work to build but give you more control over the quality of the comments.
Three common directions are:
Use a vendor’s reviewer
Use an off-the-shelf model with your rules
Build a custom multi-agent review system
A fast way to have an AI reviewer in place is to simply buy a hosted system from a vendor. The reviewer connects to the code host and comments on each new PR moving forward. So you don’t have to build the system internally.
This direction is simple to set up, does not need any particular expertise, and is probably the fastest option. But it has two major limits. First, it is generic. You can add a rule file, but the vendor controls the prompts and the harness, so many comments are generic and could apply to any project.
The second limit is control. The vendor owns the system, so it decides when the model or the instructions change. You cannot test an update before engineers see it or gradually roll out a change.
LinkedIn tried this option first. During an AI-native pilot, teams were asked to opt in to a vendor’s hosted reviewer. But the team observed that the comments were noisy and generic. They did not fit LinkedIn’s codebase and each repository’s rules. When the comments are noisy, engineers lose trust.
Soon, engineers asked to opt out, which meant the full manual review was back again. The team also had no control over the vendor’s changes. Min put it this way: “You don’t own anything. They do something, they change something, you just react to it.”
Since a hosted reviewer is too generic, a natural next step is to use an off-the-shelf model, but build your own harness around it. That way, you can choose your own prompts and write the codebase’s own rules into its instructions. With those instructions and more control over the harness, the comments will be more specific to your conventions.
This direction sounds promising: the comments would be specific to the codebase and would likely arrive quickly, before human review. But it has two main problems.
The first problem is hallucination. Because the system relies on a single model and there is no verification in place, if the model hallucinates, nothing will catch it. The author will end up with a comment about a problem that does not exist, or a real problem attached to the wrong line.
The second problem is knowledge gaps. A model learns what good and bad code look like from its training data. With a single model, it will work well on areas that overlap with its training data, but will fail on out-of-distribution inputs or inputs that were less frequent in the training data. That means a single reviewer model misses the same classes of issues repeatedly and may over-flag the same low-value issues.
LinkedIn also observed the same issues when the team built the review system with their repository-specific rules. The model gave good results but produced many hallucinations, and some of its comments did not even match the line numbers in the diff. Using a different model for review than for writing code did not fix the hallucination issue. But that idea led the team to ask whether a second model could check and verify the first model, the idea behind the multi-agent design.
This direction relies on multiple agents to produce grounded reviews. There are different design patterns for multi-agent systems, but a popular one is the orchestrator-subagent pattern [3], where one agent hands work to several subagents and combines their results. For code review, the subagents independently review the same PR, often each with a different model, and return their findings with line numbers. The orchestrator then merges overlapping findings, ranks them by confidence, and posts only reliable reviews.
This design can mitigate most of the limitations of the previous directions because you own the whole system, different models cover each other’s knowledge gaps, and findings by different agents must be verified by others to survive and be posted.
To add your rules and codebase conventions, there are two ways. One is context injection, which means adding a rule file to the repository that the AI reviewer reads on every PR. This can work in simple codebases, but is not a flexible approach because some rules apply to every repository, some to one repository, and some to one kind of change. A single file being injected as context cannot contain all three or combine the right ones for each PR.
The other approach, which LinkedIn used, is a layered design. You design several tiers, and each has its own scope, owner, and rules. Then a dedicated subagent uses those rules to perform an independent review. LinkedIn used the following three tiers:
Company-wide rules apply to every repository. These can be YAML files clarifying what the model should check.
Repository rules cover one codebase. This can be a markdown file containing the team’s conventions like coding standards and review criteria.
Pull request rules apply only to one kind of change.
To better understand how this system reviews a PR, let’s go over a concrete example and see how each step handles it. Imagine an author opens a PR that has a new python file named cache.py, implementing an in-memory cache for a Python service. But the code has two problems. It has a race condition and a bare except.
The author, who may be a coding agent, writes the change and opens a PR. The PR is then handed to the AI reviewer, which reviews it in the following three steps.
Step 1: Running parallel independent reviews
The orchestrator fetches the diff and codebase context, then dispatches them to several subagents at once. A subagent builds its own prompt, reviews the full diff in the context of the codebase and its recent changes, and then returns its structured findings.
Notice that the subagents never see each other’s findings while they review. If Subagent A could read Subagent B’s findings first, it would tend to agree, but because the subagents work independently, their agreement counts as evidence.
Step 2: Merging reviews
The independent subagents from the previous step produce findings. Many of those findings may overlap, pointing to the same issue(s). For example, if three subagents find the same five issues, the author gets fifteen comments for five problems. So the orchestrator has to merge duplicates.
This can be done in two phases. First, a deterministic check marks findings with overlapping line ranges as candidates, a cheap step that limits the search to the same code. Second, an LLM decides whether each candidate pair describes the same problem. This second phase is necessary to find overlaps semantically because two models usually do not phrase an issue the same way.
The overlaps also indicate review confidence. If multiple findings merge into a single comment, the engineer is more likely to accept it, so the code reviewer can rank it higher.
The outcome of this step is a shorter list of findings, with overlapping ones merged into single high-confidence comments. However, we still need to check the findings that only one subagent raised, which is the focus of the last step.
Step 3: Verifying reviews
A finding from only one subagent is harder to judge. It might be a real issue the other model missed, but it might also be a hallucination. To find out, the orchestrator sends those to another model to verify against the code. According to LinkedIn’s post, this often saves real issues [1]. A model that missed an issue, when asked to verify another model’s finding, confirms the issue is real. Verified findings are then rated for importance.
When the verifying model contradicts the finding, its confidence score drops. Then, low-scoring findings become candidates for filtering. In addition, findings that are cosmetic, already fixed in a later commit, or against the repository’s established patterns are suppressed before posting.
With multiple agents, layered rules, and a three-step workflow, the system has all the properties we described earlier. It merges several independent reviews into one, verifies the findings, and posts the review.
Labs and teams owning an AI reviewer system constantly change it. They may swap models, edit prompts, or add something in the harness. To tell whether a change will make the AI reviewer better, you must run evaluations, or evals, which are repeatable tests of an AI reviewer’s quality.
In this section, we cover the two phases of running evals for an AI reviewer: before a change is released, and in production after the code merges.
Before a change ships, you need to know whether the change will improve things or not. A typical process is to run the production version and the candidate on the same PRs and compare their reviews, either with an LLM judge or against a golden dataset built from human labels.
With an LLM judge, a separate model takes both reviews of the same PR as inputs, and scores them. This is cheap to run and does not need human labels. You mostly need to have a fixed rubric so the LLM follows a more predictable and reliable path to score the reviews. LinkedIn has a rubric with six weighted yes-or-no criteria, and runs every prompt, model, and harness change through this before they ship.
LLM-as-a-judge works well for comparisons [4], but it cannot reliably decide which comments are actually right. A golden dataset fills that gap with known right answers.
A typical process is to collect real PRs at the point a bug was found, and humans mark which review comments are correct. This becomes our golden dataset. We can score candidates against these labels and measure traditional metrics like precision and recall. LinkedIn has built such a dataset and is now building the eval on top of it, as a complement to its LLM-as-a-judge.
Pre-release evals run on a sample of PRs or a fixed dataset, so they cannot show how the reviewer performs on thousands of real PRs, or whether engineers find its comments useful. Those kinds of measurements can be obtained only from production. Evaluating an AI reviewer in production relies on two things: the right metrics and a feedback loop.
The right metrics determine how well your system is doing in production. There are often so many different metrics to track, such as time to first review, cost per review, and how engineers react to comments. But some are more insightful than others. Emoji reactions, for example, are noisy because they say little about whether a comment helped. On the other hand, acceptance rate is a good metric because it checks whether the code changed because of a comment. Balaji explained what counts as acceptance at LinkedIn: “Not someone giving thumbs up or thumbs down, but the code actually changed.”
LinkedIn evaluated their system in production and measured acceptance on thousands of AI comments. As the next figure shows, engineers accepted 63.9% of them [1]. These numbers show that engineers act on most of the AI reviewer’s comments, with the highest rates on concurrency bugs and dead code (100%) and logic errors (80%).
We talked about metrics, but the more important part of evaluation in production is to have a feedback loop: a mechanism that turns those measurements into actual fixes.
As you measure your system with different metrics, you learn where the gaps are. If you run and measure your system every day, you can close those gaps. For example, LinkedIn now runs its acceptance check daily and produces a trend report of which rules and prompts are gaining or losing acceptance. The team uses it to decide which rules to disable and which prompts to change.
That is evaluation. Pre-release evals check whether a change is safe to ship, and the daily loop measures the system in production and improves it over time.
In Balaji’s view, the next bottleneck after review is confidence in the quality of the software as a whole, which takes more than reviewing each PR. He does not expect review to stop being a bottleneck until automated validation is in place. Deploying, monitoring, and handling incidents also take more effort than development, so LinkedIn is aiming AI tools there too. As he put it, “the faster we are pushing out code, the faster we are potentially breaking production.”
LinkedIn’s next step for its AI reviewer is a risk heat map: an optional feature that will score each region of a PR’s diff by predicted risk so that engineers review the riskiest parts first.
[1] High-Signal AI Code Review That Adapts to Your Codebase at Scale: https://www.linkedin.com/blog/engineering/ai/high-signal-ai-code-review-that-adapts-to-your-codebase-at-scale
[2] 60 million Copilot code reviews and counting: https://github.blog/ai-and-ml/github-copilot/60-million-copilot-code-reviews-and-counting/
[3] How we built our multi-agent research system: https://www.anthropic.com/engineering/multi-agent-research-system
[4] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena: https://arxiv.org/abs/2306.05685
2026-11-08 00:30:58
Match or beat harnesses on accuracy with 28% fewer tokens. Strands harness is open source and ships with benchmarked defaults for the system prompt, tools, memory, sessions, and context management, so you can skip wiring up a harness piece by piece.
Spin it up from the CLI or in a few lines of Python or TypeScript. Nothing is a black box: swap model providers, override any default, or drop down to the SDK primitives with no rewrite. Then validate before you ship with 25+ built-in evaluators.
From idea to production with fewer tokens wasted.
This week’s system design refresher:
6 frequently used orchestration frameworks in an AI system
Top 6 Cost Optimization Strategies in AI Systems
How a SQL Query Executes Inside a Database
How does a request actually travel through Claude Code?
Orchestration in an AI system manages tasks across agents, handles failures, shared memory and context.
LangGraph: This uses a graph-based technique where workflows are represented as nodes and edges determine the state transitions.
CrewAI: A role-based abstraction technique is used here. Agents are assigned a defined role and goal that enables them to work on the tasks sequentially or via a manager agent .
Microsoft Agent Framework: This framework is designed to handle organizational workflows. It also uses graphs with built-in branching, checkpoints, and human intervention pauses.
OpenAI Agents SDK: An LLM driven runtime approach is used, where an agent decides whether it should delegate control to other agent or retain execution to complete the task itself.
Google ADK: Hybrid control approach is used where a developer defines a strategy to run tasks and a root agent splits and distributes tasks to sub-agents.
Claude Agent SDK: No separate layer for orchestration exists. It lives inside an agent which gathers context, works with tools, evaluates and keeps going until no calls left.
Your AI bills can grow quickly. The diagram below shows the top 6 techniques to make your AI system less expensive.
Prompt caching: This technique caches stable prefixes like system prompts and tool schemas. This means the model can skip redundant computation on requests with a shared prefix.
Model routing: A router sends simple queries to a small model and hard requests to a frontier model. Since a lot of queries are simple in practice, this can save a lot of money.
Batch API: This technique processes large jobs asynchronously by batching the requests first and then sending them. Providers often offer discounts on batch requests.
Semantic cache: We use an embedding model to cache queries. When there is a hit, the stored answer will be reused instead of calling the model again.
Trim the context: This technique trims the conversation history. This prevents sending the whole conversation every time, which can be costly.
Smaller model: We can also use techniques like distillation or quantization to convert a large model into a smaller model.
Over to you: Which of these techniques are most useful in production?
Join AI engineers, researchers, founders, and technical leaders at INFER/CON on November 16 in San Francisco. Convened by Datadog, this one-day conference brings together speakers from Anthropic, Together AI, Inferact, Fireworks AI, and more. Explore inference performance, production reliability, evaluation, and the economics of AI alongside the people building and operating these systems. Expect technical conversations, practical perspectives, and opportunities to connect with peers tackling similar challenges.
Attendance is free and by application only; all applications are subject to approval.
Use exclusive code to apply: Byte/VIP
Parse and validate: The database parses the SQL to check it is syntactically correct and the referenced objects, columns, and data types are valid. It converts the SQL into a logical query tree.
Rewrite and Transform: The database simplifies and restructures the query tree so it can be executed efficiently.
Optimize and choose a plan: The database evaluates the most efficient way to run the query. It compares different execution plans and selects the best plan that is supposed to use the least time and resources.
Execute the plan tree: The database runs the selected plan operations, like filtering, joining, and sorting, typically from the bottom of the parse tree to the top to produce the results.
Access cache and storage: The database retrieves the required data during execution from the cache when it is available or reads from the storage when it is not.
Return rows: The final result is formatted and sent back to the client.
Most of us type a prompt and watch the magic happen. The diagram below shows what's really going on behind the curtain, based on the Claude Code source code.
Let's trace one real request: "Fix the failing test in auth.test.ts."
Step 1: The user sends a prompt to Claude Code through their interface.
Step 2: The interface (CLI, IDE, or SDK) wraps the prompt with repo and file context and hands it to the agent loop as a request.
Step 3: The agent loop plans the next move and proposes an action: Edit(auth.ts, lines 42–58).
Step 4: The permission system checks the proposed action against the rules.
Step 5: The approved action becomes a tool call: Edit(auth.ts, patch), dispatched to the matching tool.
Step 6: The tool runs in the execution environment (shell, cloud, or sandbox) as a real syscall.
Step 7: The execution returns a tool result back to the agent loop.
Step 8: The agent persists the turn to state and streams the final message to the user.
The whole system is just this loop, repeated until the model stops asking for tools.
Over to you: which step in this loop do you think is the hardest one to get right when building your own coding agent?