2025-05-10 01:00:03
Video Friday is your weekly selection of awesome robotics videos, collected by your friends at IEEE Spectrum robotics. We also post a weekly calendar of upcoming robotics events for the next few months. Please send us your events for inclusion.
Enjoy today’s videos!
Today I learned that “hippotherapy” is not quite what I wanted it to be.
The integration of KUKA robots into robotic physiotherapy equipment offers numerous advantages, such as precise motion planning and control of robot-assisted therapy, individualized training, reduced therapist workload and patient-progress monitoring. As a result, these robotic therapies can be superior to many conventional physical therapies in restabilizing patients’ limbs.
[ Kuka ]
MIT engineers are getting in on the robotic ping-pong game with a powerful, lightweight design that returns shots with high-speed precision. The new table-tennis bot comprises a multijointed robotic arm that is fixed to one end of a ping-pong table and wields a standard ping-pong paddle. Aided by several high-speed cameras and a high-bandwidth predictive control system, the robot quickly estimates the speed and trajectory of an incoming ball and executes one of several swing types—loop, drive, or chop—to precisely hit the ball to a desired location on the table with various types of spin.
[ MIT News ]
Pan flipping involves dynamically flipping various objects, such as eggs, burger buns, and meat patties. This demonstrates precision, agility, and the ability to adapt to different challenges in motion control. Our framework enables robots to learn highly dynamic movements.
[ GitHub ] via [ Human Centered Autonomy Lab ]
Thanks, Haonan!
An edible robot made by EPFL scientists leverages a combination of biodegradable fuel and surface tension to zip around the water’s surface, creating a safe—and nutritious—alternative to environmental monitoring devices made from artificial polymers and electronics.
[ EPFL ]
Traditional quadcopters excel in flight agility and maneuverability but often face limitations in hovering efficiency and horizontal field of view. Nature-inspired rotary wings, while offering a broader perspective and enhanced hovering efficiency, are hampered by substantial angular momentum restrictions. In this study, we introduce QuadRotary, a novel vehicle that integrates the strengths of both flight characteristics through a reconfigurable design.
[ Paper ] via [ Singapore University of Technology and Design ]
I like the idea of a humanoid that uses jumping as a primary locomotion mode not because it has to, but because it’s fun.
[ PAL Robotics ]
I had not realized how much nuance there is to digging stuff up with a shovel.
[ Intelligent Motion Laboratory ]
A new 10,000-gallon [38,000-liter] water tank at the University of Michigan will help researchers design, build, and test a variety of autonomous underwater systems that could help robots map lakes and oceans and conduct inspections of ships and bridges. The tank, funded by the Office of Naval Research, allows roboticists to further test projects on robot control and behavior, marine sensing and perception, and multivehicle coordination.
“The lore is that this helps to jump-start research, as each testing tank is a living reservoir for all of the knowledge gained from within it,” said Jason Bundoff, lead engineer in research at U-M’s Friedman Marine Hydrodynamics Laboratory. “You mix the waters from other tanks to imbue the newly founded tank with all of that living knowledge from the other tanks, which helps to keep the knowledge from being lost.”
If you have a humanoid robot and you’re wondering how it should communicate, here’s the answer.
[ Pollen ]
Whose side are you on, Dusty?
Even construction robots should be mindful about siding with the Empire, though there can be consequences!
- YouTube
[ Dusty Robotics ]
This Michigan Robotics Seminar is by Danfei Xu from Georgia Tech, on “Generative Task and Motion Planning.”
Long-horizon planning is fundamental to our ability to solve complex physical problems, from using tools to cooking dinners. Despite recent progress in commonsense-rich foundation models, the ability to do the same is still lacking in robots, particularly with learning-based approaches. In this talk, I will present a body of work that aims to transform Task and Motion Planning—one of the most powerful computational frameworks in robot planning—into a fully generative model framework, enabling compositional generalization in a largely data-driven approach.
2025-05-09 02:00:03
This article is part of our exclusive career advice series in partnership with the IEEE Technology and Engineering Management Society.
Throughout my 40-year career as an electrical engineer and entrepreneur, I’ve often been asked how I achieve a work-life balance. Over time, I’ve come to realize that the question—and the way it’s framed—is inherently flawed. So my response to the inquiry is simple: I seek to live an integrated life where work and personal joy are not in conflict but in harmony.
The key is in shifting your mindset: Stop viewing work and life as opposites and start recognizing how they complement each other.
The notion of work-life balance suggests that work and life are opposing forces. Balance is seen as an elusive goal. The implication is that success in one area inevitably comes at the expense of the other. But what if the conflict is more imagined than real? Instead of trying to balance two separate entities, the goal should be to integrate them into a cohesive whole.
I won’t pretend that I have everything figured out. Life—especially with work, kids, and the everyday chaos of being part of a two-income household—is messy. But I’ve learned that work and life aren’t two things to balance; they’re two sides of the same coin. Rather than compartmentalizing them, I approach them as interconnected parts of a fulfilling journey.
Here are tips that have helped me embrace work and life as a unified whole.
Let’s be real, balancing a career with caring for children and handling daily responsibilities is chaotic, especially when both spouses are working. Between shuttling the kids to after-school activities, keeping up with household chores, and managing work deadlines, it can feel impossible to find time for everything. But here’s the thing: Balance doesn’t always come in the form of perfectly blocked time. It can come in small, intentional moments.
I’ve learned to make use of the in-between times to my advantage.
When I used to attend my child’s swim meets or was waiting for an event to start, for example, I would catch up on work with my iPad. I wasn’t always working, but in those moments where I’d otherwise just be waiting, I got things done. By strategically using downtime, you can keep on top of work while also being present for your family.
If you can find creative ways to merge the chaos of life with work demands, you can feel less overwhelmed, even when it all feels like a juggling act.
When I experience joy or setbacks in my work, I share them with my family. By doing so, I bring them into the ups and downs of my entrepreneurial journey of running five technology companies. My work isn’t a separate part of my life, and having a conversation about it with those closest to me allows us to connect more deeply.
By involving my family members in my professional world, they’ve become more than bystanders; they’ve become a supportive sounding board. The integration means that I don’t feel constantly torn between my work and my personal life. Instead, I’ve found harmony in embracing both.
By introducing the key figures in my professional life to my partner, I create context for them. It fosters empathy and understanding, allowing my spouse to offer emotional support. The transparency avoids the unrealistic pressure of “leaving work at work.” After all, we are human, and what happens at work affects how we feel at home.
For young engineers, that mindset shift is key. Don’t view work as something that competes with your personal life. View it as something that can be shared. The more your loved ones understand your professional world, the stronger your relationships can become.
For working couples, it can be especially relevant. Coordinating to give each other “catch-up time” helps create space for both partners to manage work commitments without sacrificing family experiences. It’s not about being perfect in both spheres. It’s about being present where you are.
Another way I’ve achieved the integration of work and life is by blending travel for business and pleasure.
On family vacations, I don’t treat time away as a complete break from work. I typically start my day early in the morning, catching up on email before everyone else wakes up. By the time my family is ready for breakfast, I’ve usually handled my work responsibilities and can be fully present with my spouse and children. The approach allows me to enjoy the day stress-free, knowing I’ve kept up with professional demands.
Work-life integration is also important at the personal level. On business trips, I always build in extra time to explore the area. These mini vacations transform my work trips from exhausting obligations into enriching experiences. I often visit places within a short flight from my business destination, turning a routine trip into an adventure.
Not all adventures have to be shared to be fulfilling. Sometimes solo experiences can refresh you just as much. The approach works for my spouse and I, as we each find our own ways to recharge before reuniting.
It’s important to schedule downtime, as it can make you more productive in the long run. Taking a few hours to relax without guilt is exactly what you need to tackle your next project with clarity. Sometimes after a full day of meetings, my spouse and I watch a TV show together, sharing each other’s company. On other days, we plan a dinner with no electronics, and we just talk about our day.
I’ve been able not only to achieve most of my professional goals but also to build a life rich in experiences and memories.
Life isn’t a zero-sum game between work and personal time. It’s about finding synergy between the two and designing your life so both parts can thrive. As engineers with analytical mindsets and problem-solving skills, we’re well suited to take on the challenge.
If there’s one piece of advice I’d give to engineers and young professionals, it’s this: Don’t chase balance; pursue integration. If you do, you’re likely to find that life in all its complexity becomes far more fulfilling. The experiences you create—both at work and at home—are sure to be richer, and your sense of accomplishment can extend beyond just your career.
2025-05-08 23:00:04
For months, the world has wondered about the stated goal of the president of the United States to acquire Greenland. Is he merely expressing a desire to make America greater again in terms of territorial area? Is it a question of security policy? Or are critical minerals—and especially Greenland’s immense rare earth riches—a key factor?
The first two questions can’t really be answered without a fuller understanding of the administration’s motives and strategies than is currently available publicly. But for that third question, there exists a wealth of data and context, both historical and modern-day.
Let’s start with the modern day. U.S. industries, like those of any developed economy, depend on critical materials. Lots of attention now is focused on the rare earths, a group of elements of unusual importance because of their indispensability in essential commercial, defense, and industrial applications. Roughly 90 percent of processed rare earths come from China, creating supply-chain vulnerabilities that many countries are now trying to avoid, particularly since China announced restrictions on the export of heavy rare earths in April 2025. Systematic studies have indicated that Greenland has 10 important deposits of rare earth elements.
But in mining, it’s the details that matter. To understand the value of a minerals deposit it’s important to consider the history and to make a reality check of documented resources. The facts and the obvious conclusions do not always reach decision-makers, investors, and the media.
Greenland is the largest island in the world, but people often get an exaggerated impression of how big it is because the commonly used Mercator projection distorts the size of landmasses close to the poles. Greenland is about 2 million square kilometers. However, the ice-free part—which could much more feasibly be mined than the other part—is roughly the size of California, and without any scenic coastal highways. Traveling between settlements is possible only by boat or airplane.
Greenland has had nine different mines since World War II, but only one, for the mineral anorthosite, is active today. Another one, for gold, is expected to reach full production later this year.
Greenland’s only fully operational mine as of May 2025 extracts anorthosite rock at a site called White Mountain, on the central west coast of the island.Flemming Getreuer Christiansen
Greenland took over the handling of all mining licenses from Denmark in 1998 and full authority in 2010 after self-governance was introduced. Greenland has developed a modern licensing system for mining with an element of competition between companies and transparent procedures for public input on environmental and socioeconomic concerns. The number of licenses granted by the government has been high and relatively constant for many years, but the level of actual activities, such as drilling, has been low over the past decade. Several licenses have been relinquished or revoked without any mining ever taking place.
The reasons are many. A major one is lack of human resources. The population of Greenland is only 57,000, scattered around an area three times the size of Texas. Another reason is high operational costs due to the harsh climate and lack of infrastructure. Others include restrictions favoring labor from Greenland or Denmark. Still others include puzzling recent bureaucratic changes in the legal framework of mining related to the requirements for resource assessments and feasibility studies, and also environmental and socioeconomic requirements for obtaining an exploitation license. Complex royalty schemes and relatively high corporate taxes contribute to uncertainty and risks that many investors have been unwilling to take. This is at a time when Greenland sorely needs investments to fuel its dreams of economic and political independence from Denmark.
In the summer of 2021, the Greenland government said no to issuing further petroleum licenses. It didn’t matter very much because by then, all the largest oil and gas companies had already left Greenland as a result of declining oil prices and increasing costs caused by the changes to regulations. Since the 1970s, some 40 companies have been involved, but they drilled only 15 exploration wells and made no commercial discoveries.
The Ivittuut mine, seen here in 1953, extracted huge amounts of cryolite during World War II. Ivittuut was one of the few places in the world known to have deposits of cryolite, which was necessary for the smelting of aluminum. The mine closed in 1987.Kaj Skall Sørensen/Danish Arctic Institute
Still, there have been some commercial activities. During World War II, the Ivittuut mine in South Greenland produced cryolite for aluminum production and was crucial for the U.S. war effort. Since that time, U.S. companies have shown only a modest interest in Greenland; of the total of 250 companies that have been granted exploration licenses over the past several decades, 10 have been American. Of 50 companies from various countries that have been drilling in Greenland, four were from the United States, and of the 15 companies that have applied for an exploitation license, precisely zero were American. So historically the U.S. fingerprint on mineral exploration in Greenland is negligible, even though the door has been open for decades.
According to the European Union, Greenland has great untapped potential for 25 of the 34 minerals identified in the Union’s official list of raw materials, including rare earth elements, graphite, platinum group metals, and niobium. Looking at these in more detail, however, a more complex picture emerges. In 2023, an investigation for the Center for Mineral Resources and Materials of the Geological Survey of Denmark and Greenland found that Greenland’s mineral resources included everything from minor occurrences to substantial deposits scattered around the island.
Of Greenland’s 10 important rare earth deposits, only two, Kvanefjeld (Kuannersuit) and Kringlerne (Killavaat Alannguat), have attracted much attention.
In spite of being located just a few miles apart in the same complex in South Greenland, these two rare earth deposits are very different for all parameters: geology and mineralogy, stage of exploration, and documented resources. They are alike, however, in that both deposits have some serious drawbacks when compared with active mines or advanced projects elsewhere in the West.
The Kvanefjeld plateau, near the southern tip of Greenland, is the site of large deposits of rare earth oxides, uranium, thorium, and other elements of industrial importance.Jan Richard Heinicke/laif/Redux
Kvanefjeld is the only project in Greenland with well-documented reserves, but it suffers from–or, some would say, could potentially benefit from–a high content of uranium and thorium.
In 2021, when the project was in the final phase of getting an exploitation license, the Greenlandic government dug in its heels and made it illegal to mine deposits with more than 100 parts per million of uranium. The new regulations were used against the Australian company behind the Kvanefjeld project, rendering worthless years of heavy investments; more than US $100 million was reportedly spent on drilling and other work.
The company filed a request for arbitration in 2022, and in 2024 it commenced legal proceedings against the Greenland and Danish governments. It may be years before these cases are resolved. In the meantime, this case is likely to harm investment in Greenland due to the climate of high political risk it suggests.
The rare earths at the Kringlerne deposit are contained within an igneous rock called kakortokite.Jan Richard Heinicke/laif/Redux
At Kringlerne there are much lower concentrations of uranium. The privately owned company that has invested in the site, Tanbreez, claims that it is the largest deposit of its kind in the world, but that belief is based on very little drilling. In 2020, the company got an exploration license in spite of not having provided documentation of resources, updated studies of feasibility, or of environmental and socioeconomic impact. Deadlines for providing financial security and plans for mining and closure were extended for many years. Tanbreez was partly taken over by a New York–based company registered on the NASDAQ, and the new owners were obligated to follow international standards and disclose a confidential resource estimate from 2016. The report revealed considerably more disappointing numbers with regard to ore grade and other characteristics.
Nor do Greenland’s deposits fare well in comparison with other mines or deposits. Typical ore grades at successful mines or attractive deposits are between 4 and 8 percent rare earths. The ore at the mines at Mount Weld in Australia and Mountain Pass in California, and the deposits at Nolans Bore in Australia and Bear Lodge in Wyoming, all fall within that range. In Kvanefjeld the number is 1.4 percent, and in Kringlerne, it dips as low as 0.38 percent. The Greenlandic mines would consequently require larger open pits and much more energy for crushing, separation, and refining, which would inevitably drive up costs for establishing mines and for actually operating them.
Nevertheless, the recent U.S. interest in Greenland led to some surprising reactions on the stock market, combined with high volatility. The share price of Kvanefjeld tripled after intense trading at the start of 2025. The share price of Kringlerne has dropped significantly since the new owners were listed on the NASDAQ, but with many ups and downs following all sorts of stock announcements that referenced miscellaneous analyses of old cores.
After the Trump administration touched off a tariff war, the geopolitical skirmishing between the United States and China over rare earths became a much more complicated conflagration involving the United States, the European Union, and China.
And the Greenland government has continued to fan the flames by courting Chinese investment. In March, for example, Greenland’s foreign minister, Vivian Motzfeldt, reportedly identified closer cooperation with China as a priority, and even touted the possibility of a free trade agreement between Greenland and China. According to a report in The Diplomat, Motzfeldt’s actions were ”largely driven by the belief…that a mining boom, fueled by Chinese investment, was the most realistic path toward independence from Denmark—a goal shared by most Greenlandic parties.” Maybe so, but such moves have been widely perceived as a provocation against Denmark and the United States. Whatever their motivation, they could very well lead to increased U.S. pressure. For the United States it would be a geostrategic nightmare if China opened a mine in a remote part of Greenland, with a town, communication lines, harbor, and an airfield that could obviously be used for purposes other than resource extraction and export.
Since Greenland took control of its mineral resources, Denmark has been reluctant to get involved in mining projects. That could very well change. Denmark and the E.U. may come under pressure to invest in much-needed infrastructure and energy projects and offer loans on more favorable conditions, if for no other reason than to keep the United States at bay.
Though the situation is highly unstable, it’s important to try, at least, to separate the geopolitical posturing from the realities of mining. In that vein, it’s safe to say there will be no operating rare earth mines in Greenland during the term of Donald J. Trump.2025-05-08 03:35:06
This article is crossposted from IEEE Spectrum’s careers newsletter. Sign up now to get insider tips, expert advice, and practical strategies, written in partnership with tech career development company Taro and delivered to your inbox for free!
There is a widespread misconception in the tech industry that if you work hard, you are guaranteed to be rewarded. Unfortunately, this is far from the truth. During my four and a half years at Meta, I saw many people working crazy hours who still ended up with a ‘Meets Most Expectations” rating, putting them at risk for a remedial performance improvement plan. Not only were these engineers sacrificing their evenings and taking on tons of stress, but they weren’t even being acknowledged for their efforts.
Whether you’re at a startup or Big Tech company, there is an endless amount of work you could take on. No matter how many weekends you dedicate to your team, you’ll always have more to do. In order to have a sustainably high impact, you must be deliberate about choosing what to work on.
By understanding what your team and manager care about, you can ensure great results that are commensurate with great effort. So many engineers mess this up: They go down a rabbit-hole about some interesting problem instead of thinking about their performance review and how they’ll be judged.
Tactically, here’s what that means for you:
The above steps will ensure that every action you take is loaded with value. Once you’re consistently exceeding expectations, you’re ready to take on more responsibility and grow your career.
—Rahul
ICYMI: U.S. Semiconductor Courses Surge Amid Industry Boom
The U.S. semiconductor industry is booming, driven by rapid advancements in AI, federal funding, and private sector investments. As a result, electrical engineering programs in the U.S. are now seeing an uptick in student enrollment in semiconductor coursework. But some educators and recruiters warn that new tariffs and proposed immigration restrictions could potentially complicate job prospects for students.
ICYMI: How to Avoid Ethical Red Flags in Your AI Projects
A growing number of engineers now find themselves developing AI solutions while navigating complex ethical considerations, says IBM’s AI ethics global leader Francesca Rossi. In this guest article, Rossi provides some advice for engineers based on her experience developing IBM’s internal processes for responsible AI deployment.
How Big Tech hides its outsourced African workforce
Content moderation, AI data training, and other tasks imperative to the tech sector require a lot of labor. New data reported by Rest of World reveals that the people performing this labor are often hidden. Visualized as maps, the data shows that many African workers are indirectly employed by Big Tech companies across the globe.
2025-05-07 16:30:03
At an event in Dortmund, Germany today, Amazon announced a new robotic system called Vulcan, which the company is calling “its first robotic system with a genuine sense of touch—designed to transform how robots interact with the physical world.” In the short to medium term, the physical world that Amazon is most concerned with is its warehouses, and Vulcan is designed to assist (or take over, depending on your perspective) with stowing and picking items in its mobile robotic inventory system.
In two upcoming papers in IEEE Transactions on Robotics, Amazon researchers describe how both the stowing and picking side of the system operates. We covered stowing in detail a couple of years ago, when we spoke with Aaron Parness, the director of applied science at Amazon Robotics. Parness and his team have made a lot of progress on stowing since then, improving speed and reliability over more than 500,000 stows in operational warehouses to the point where the average stowing robot is now slightly faster than the average stowing human. We spoke with Parness to get an update on stowing, as well as an in-depth look at how Vulcan handles picking, which you can find in this separate article. It’s a much different problem, and well worth a read.
Stowing is the process by which Amazon brings products into its warehouses and adds them to its inventory so that you can order them. Not surprisingly, Amazon has gone to extreme lengths to optimize this process to maximize efficiency in both space and time. Human stowers are presented with a mobile robotic pod full of fabric cubbies (bins) with elastic bands across the front of them to keep stuff from falling out. The human’s job is to find a promising space in a bin, pull the plastic band aside, and stuff the thing into that space. The item’s new home is recorded in Amazon’s system, the pod then drives back into the warehouse, and the next pod comes along, ready for the next item.
Different manipulation tools are used to interact with human-optimized bins.Amazon
The new paper on stowing includes some interesting numbers about Amazon’s inventory-handling process that helps put the scale of the problem in perspective. More than 14 billion items are stowed by hand every year at Amazon warehouses. Amazon is hoping that Vulcan robots will be able to stow 80 percent of these items at a rate of 300 items per hour, while operating 20 hours per day. It’s a very, very high bar.
After a lot of practice, Amazon’s robots are now quite good at the stowing task. Parness tells us that the stow system is operating three times as fast as it was 18 months ago, meaning that it’s actually a little bit faster than an average human. This is exciting, but as Parness explains, expert humans still put the robots to shame. “The fastest humans at this task are like Olympic athletes. They’re far faster than the robots, and they’re able to store items in pods at much higher densities.” High density is important because it means that more stuff can fit into warehouses that are physically closer to more people, which is especially relevant in urban areas where space is at a premium. The best humans can get very creative when it comes to this physical three-dimensional “Tetris-ing,” which the robots are still working on.
Where robots do excel is planning ahead, and this is likely why the average robot stower is now able to outpace the average human stower—Tetris-ing is a mental process, too. In the same way that good Tetris players are thinking about where the next piece is going to go, not just the current piece, robots are able to leverage a lot more information than humans can to optimize what gets stowed where and when, says Parness. “When you’re a person doing this task, you’ve got a buffer of 20 or 30 items, and you’re looking for an opportunity to fit those items into different bins, and having to remember which item might go into which space. But the robot knows all of the properties of all of our items at once, and we can also look at all of the bins at the same time along with the bins in the next couple of pods that are coming up. So we can do this optimization over the whole set of information in 100 milliseconds.”
Essentially, robots are far better at optimization within the planning side of Tetrising, while humans are (still) far better at the manipulation side, but that gap is closing as robots get more experienced at operating in clutter and contact. Amazon has had Vulcan stowing robots operating for over a year in live warehouses in Germany and Washington state to collect training data, and those robots have successfully stowed hundreds of thousands of items.
Stowing is of course only half of what Vulcan is designed to do. Picking offers all kinds of unique challenges too, and you can read our in-depth discussion with Parness on that topic right here.
2025-05-07 16:00:03
As far as I can make out, Amazon’s warehouses are highly structured, extremely organized, very tidy, absolute raging messes. Everything in an Amazon warehouse is (usually) exactly where it’s supposed to be, which is typically jammed into some pseudorandom fabric bin the size of a shoebox along with a bunch of other pseudorandom crap. Somehow, this turns out to be the most space- and time-efficient way of doing things, because (as we’ve written about before) you have to consider the process of stowing items away in a warehouse as well as the process of picking them, and that involves some compromises in favor of space and speed.
For humans, this isn’t so much of a problem. When someone orders something on Amazon, a human can root around in those bins, shove some things out of the way, and then pull out the item that they’re looking for. This is exactly the sort of thing that robots tend to be terrible at, because not only is this process slightly different every single time, it’s also very hard to define exactly how humans go about it.
As you might expect, Amazon has been working very very hard on this picking problem. Today at an event in Germany, the company announced Vulcan, a robotic system that can both stow and pick items at human(ish) speeds.
Last time we talked with Aaron Parness, the director of applied science at Amazon Robotics, our conversation was focused on stowing—putting items into bins. As part of today’s announcement, Amazon revealed that its robots are now slightly faster at stowing than the average human is. But in the stow context, there’s a limited amount that a robot really has to understand about what’s actually happening in the bin. Fundamentally, the stowing robot’s job is to squoosh whatever is currently in a bin as far to one side as possible in order to make enough room to cram a new item in. As long as the robot is at least somewhat careful not to crushify anything, it’s a relatively straightforward task, at least compared to picking.
The choices made when an item is stowed into a bin will affect how hard it is to get that item out of that bin later on—this is called “bin etiquette.” Amazon is trying to learn bin etiquette with AI to make picking more efficient.Amazon
The defining problem of picking, as far as robots are concerned, is sensing and manipulation in clutter. “It’s a naturally contact-rich task, and we have to plan on that contact and react to it,” Parness says. And it’s not enough to solve these problems slowly and carefully, because Amazon Robotics is trying to put robots in production, which means that its systems are being directly compared to a not-so-small army of humans who are doing this exact same job very efficiently.
“There’s a new science challenge here, which is to identify the right item,” explains Parness. The thing to understand about identifying items in an Amazon warehouse is that there are a lot of them: something like 400 million unique items. One single floor of an Amazon warehouse can easily contain 15,000 pods, which is over a million bins, and Amazon has several hundred warehouses. This is a lot of stuff.
In theory, Amazon knows exactly which items are in every single bin. Amazon also knows (again, in theory), the weight and dimensions of each of those items, and probably has some pictures of each item from previous times that the item has been stowed or picked. This is a great starting point for item identification, but as Parness points out, “We have lots of items that aren’t feature rich—imagine all of the different things you might get in a brown cardboard box.”
As challenging as it is to correctly identify an item in a bin that may be stuffed to the brim with nearly identical items, an even bigger challenge is actually getting that item that you just identified out of the bin. The hardware and software that humans have for doing this task is unmatched by any robot, which is always a problem, but the real complicating factor is dealing with items that are all jumbled together in a small fabric bin. And the picking process itself involves more than just extraction—once the item is out of the bin, you then have to get it to the next order-fulfillment step, which means dropping it into another bin or putting it on a conveyor or something.
“When we were originally starting out, we assumed we’d have to carry the item over some distance after we pulled it out of the bin,” explains Parness. “So we were thinking we needed pinch grasping.” A pinch grasp is when you grab something between a finger (or fingers) and your thumb, and at least for humans, it’s a versatile and reliable way of grabbing a wide variety of stuff. But as Parness notes, for robots in this context, it’s more complicated: “Even pinch grasping is not ideal because if you pinch the edge of a book, or the end of a plastic bag with something inside it, you don’t have pose control of the item and it may flop around unpredictably.”
At some point, Parness and his team realized that while an item did have to move farther than just out of the bin, it didn’t actually have to get moved by the picking robot itself. Instead, they came up with a lifting conveyor that positions itself directly outside of the bin being picked from, so that all the robot has to do is get the item out of the bin and onto the conveyor. “It doesn’t look that graceful right now,” admits Parness, but it’s a clever use of hardware to substantially simplify the manipulation problem, and has the side benefit of allowing the robot to work more efficiently, since the conveyor can move the item along while the arm starts working on the next pick.
Amazon’s robots have different techniques for extracting items from bins, using different gripping hardware depending on what needs to be picked. The type of end effector that the system chooses and the grasping approach depend on what the item is, where it is in the bin, and also what it’s next to. It’s a complicated planning problem that Amazon is tackling with AI, as Parness explains. “We’re starting to build foundation models of items, including properties like how squishy they are, how fragile they are, and whether they tend to get stuck on other items or no. So we’re trying to learn those things, and it’s early stage for us, but we think reasoning about item properties is going to be important to get to that level of reliability that we need.”
Reliability has to be superhigh for Amazon (and with many other commercial robotic deployments) simply because small errors multiplied over huge deployments result in an unacceptable amount of screwing up. There’s a very, very long tail of unusual things that Amazon’s robots might encounter when trying to extract an item from a bin. Even if there’s some particularly weird bin situation that might only show up once in a million picks, that still ends up happening many times per day on the scale at which Amazon operates. Fortunately for Amazon, they’ve got humans around, and part of the reason that this robotic system can be effective in production at all is that if the robot gets stuck, or even just sees a bin that it knows is likely to cause problems, it can just give up, route that particular item to a human picker, and move on to the next one.
The other new technique that Amazon is implementing is a sort of modern approach to “visual servoing,” where the robot watches itself move and then adjusts its movement based on what it sees. As Parness explains: “It’s an important capability because it allows us to catch problems before they happen. I think that’s probably our biggest innovation, and it spans not just our problem, but problems across robotics.”
Parness was very clear that (for better or worse) Amazon isn’t thinking about its stowing and picking robots in terms of replacing humans completely. There’s that long tail of items that need a human touch, and it’s frankly hard to imagine any robotic-manipulation system capable enough to make at least occasional human help unnecessary in an environment like an Amazon warehouse, which somehow manages to maximize organization and chaos at the same time.
These stowing and picking robots have been undergoing live testing in an Amazon warehouse in Germany for the past year, where they’re already demonstrating ways in which human workers could directly benefit from their presence. For example, Amazon pods can be up to 2.5 meters tall, meaning that human workers need to use a stepladder to reach the highest bins and bend down to reach the lowest ones. If the robots were primarily tasked with interacting with these bins, it would help humans work faster while putting less stress on their bodies.
With the robots so far managing to keep up with human workers, Parness tells us that the emphasis going forward will be primarily on getting better at not screwing up: “I think our speed is in a really good spot. The thing we’re focused on now is getting that last bit of reliability, and that will be our next year of work.” While it may seem like Amazon is optimizing for its own very specific use cases, Parness reiterates that the bigger picture here is using every last one of those 400 million items jumbled into bins as a unique opportunity to do fundamental research on fast, reliable manipulation in complex environments.
“If you can build the science to handle high contact and high clutter, we’re going to use it everywhere,” says Parness. “It’s going to be useful for everything, from warehouses to your own home. What we’re working on now are just the first problems that are forcing us to develop these capabilities, but I think it’s the future of robotic manipulation.”