In the 60s children’s cartoon “The Jetsons,” Rosie the Robot was featured as a futuristic mechanical maid who cleaned the clutter of the household, much to the delight of the time-strapped housewife. At that time, the idea of a robot assistant was a vision of the distant future where technology would eliminate the responsibilities of cleaning. But now, over 50 years later, a team of computer scientists has taken steps toward making this a reality. 

Meet TidyBot: a robot composed of a claw and an arm stuck into a rectangular base that cleans your room by learning your individual preferences. For example, suppose you tell the robot that it should place books in the drawer and pens on your desk. TidyBot not only puts the books and the pens in the right place but is also able to infer where you would like other items to be placed. 

The robot does this using a pre-built large language model, like OpenAI’s ChatGPT. The examples are fed into the large language model to generate more general rules. This allows the robot to be able to clean up new objects that were not part of the input examples given by the user. 

“Tidybot is a fascinating project that builds on and generalizes advances in language, vision and robotics,” said Sanmi Koyejo, an assistant professor of computer science at Stanford and the lead of the Stanford Trustworthy AI Research (STAIR) group, which is unaffiliated with the study. “Getting all of these to work separately is challenging. Getting all of them to work together is really impressive.” 

He added that, “Outside of its general usefulness, this work highlights a promising path for integrating some of the cutting-edge foundation models into new and innovative use cases.”

Tackling real world challenges 

Transitioning from a lab environment to the unpredictable cluttered landscape of a real home is not a trivial leap, but one that the researchers say they were determined to make with TidyBot. 

“We did make various simplifying assumptions to be able to build a system that works in our lab, and so to make it kind of more general for the real world, we had to remove some of those assumptions,” said Jimmy Wu, an author of this study. 

For example, instead of using standardized objects with a known size, they used objects that would be found in real-world settings, such as a messy apartment. “We wanted to bring shoes and toys and clothing from our own homes and work with these things,” said Rika Antonova, a postdoctoral fellow at Stanford and an author of this study. 

According to Antonova, TidyBot being small and agile minimizes the risk it poses to adults, but questions about its behavior around vulnerable populations, such as small children, remain. 

To help minimize these issues, the robot has various safety checks in place. “On the bottom of the robot, there are these touch sensors that ensure that whenever it bumps into anything, it’s going to immediately stop everything,” Wu said. 

In addition, Wu says the position of the robot is meticulously checked by the computer program to ensure that the robot is where it is supposed to be. “If the controller is kind of going unstable and moving the robot to where [it] shouldn’t go, then we’ll stop the robot immediately,” Wu said.

Next steps 

While TidyBot works with 85% accuracy, Wu and Antonova are quick to point out that they’re not rushing to commercialize. For one, the hardware used in the robot is very hard to come by. “We have this hardware, but … very few of them exist in the world with these particular capabilities,” Antanova said. 

A significant step toward bridging this gap, though, is the demonstration of TidyBot at the launch of the Stanford Robotics Center in March 2024. “This is a very exciting direction and another direction that is specific to Stanford,” Antonova said. 

The center is designed to mimic the environment of the real world. For example, according to Antonova, one of the areas of the center will feature an apartment with furniture and a real kitchen.

One key area that the team is excited to explore is deep reinforcement learning. The researchers say their goal is to provide TidyBot the ability to learn complex tasks through a process of trial and error much like humans do. However, Antonova says that “these types of approaches can be quite data-hungry,” an especially critical challenge in robotics where collecting data is often very expensive. 

For Wu, Antonova and the rest of the team, the next few months are about making sure that when TidyBot is ready to move from the lab to the living room, it will do so smoothly, safely and with the ability to integrate seamlessly into the daily lives of its users.