How LabOS AI-powered smart glasses can reduce human error in science

AI-powered smart glasses help beginners perform like experts

By Deni Ellis Béchard edited by Eric Sullivan

Imagine standing at the lab bench working on an experiment, when you complete one step, a screen inside your lab glasses shows you what to do next. A small camera in the frame keeps a close eye on your hands. If you reach for the wrong pipe, the display will flash a warning. Before you can make the mistake, the system tells you how to get back on track.

Laboratory goggles have finally joined the ranks of smart devices. That’s the promise behind LabOS, an AI “operating system” for scientific labs built by the Stanford-Princeton AI Coscientist Team, a group led by Stanford University bioengineer Le Cong and Princeton University computer scientist Mengdi Wang, with founding partners that include NVIDIA. Powered by NVIDIA’s vision language models for processing visual data, the system is designed to give AI real-time knowledge of lab work, so it can determine what makes experiments fail or succeed and quickly train new researchers to expert levels by guiding them through experimental protocols.

Walk into a wet lab, Cong says, and “it hasn’t changed much in the last 50 years.” This matters, he explains, because much of the time science is carried out “in the physical laboratory, in the physical world, not on computers”. As described in a recent preprint article, LabOS aims to bridge this physical-digital divide.

On supporting science journalism

If you like this article, please consider supporting our award-winning journalism by subscribes. By purchasing a subscription, you help secure the future of impactful stories about the discoveries and ideas that shape our world today.

The scientific community has long struggled with a problem known for more than a decade as a “replication crisis.” In a 2016 Nature survey, Monya Baker, then editor of the journal, reported that “more than 70% of scientists have tried and failed to reproduce another scientist’s experiments,” and more than half could not reproduce their own work. Some of this error rate can be attributed to statistical mishandling or publication pressure. But a common cause gets less attention: people doing repetitive lab work make mistakes. A reagent added at the wrong temperature, a step skipped under time pressure, a contaminated pipette tip – these are mistakes that may be too small to notice but are big enough to ruin an experiment.

The solution proposed by Wang and Cong’s team is an open-source platform and hardware kit that allows AI to see what scientists see. Researchers in early pilot tests in Cong’s lab at Stanford and Wang’s at Princeton are using augmented reality/extended reality (AR/XR) glasses that stream video directly to the system. LabOS compares what it sees with the written protocol, offering guidance to the user while collecting training data. AI can talk the researcher through each step, reminding them to keep a surface sterile or flagging errors in technique.

AI needs real-time knowledge of experiments to learn what works and what doesn’t, much in the same way that robots and self-driving cars need to gather real-world data to update their systems. “We can have 1,000 chatbots, 1,000 AI scientists trying to tell real scientists what to do,” Wang says, but if the AI ​​isn’t plugged into the physical experiment, “we’ll never have anything that can be verified.”

Normally when people do laboratory work, learning can be slow. If an experiment fails, they try to figure out what went wrong and start over. But when the AI ​​looks at an experiment and sees the result, it might be able to more quickly figure out which steps caused problems and can design a new experiment. By recording entire experiments, an AI can study the smallest details to find out what caused them to fail.

This oversight extends beyond human guidance; LabOS also uses a robotic arm to handle tedious tasks like mixing. “It’s not like replacing people,” says Cong. – We have to help people.

So far, the aid is yielding results. In an experimental procedure that involved increasing the amount of a particular protein in cells, junior researchers with just one week of LabOS training achieved results that were virtually indistinguishable from expert researchers. “I couldn’t tell the difference as a professor,” says Cong. “The results of the experiment – they are identical.”

“From a robotics and human-computer interaction perspective, this work highlights a promising direction,” said Kourosh Darvish, a researcher at the AI ​​and Automation Lab at the University of Toronto’s Acceleration Consortium, who was not involved in LabOS development. Nevertheless, he notes the importance of developing standards to better evaluate such work. “As AI systems increasingly move from analytical tools to active partners in experimentation, standardization and validation at the community level will be critical.”

The AI ​​Coscientist Team is already pushing this technology beyond the research bench. Recently, the researchers introduced MedOS, adapting their AI and AR architecture to assist surgeons with anatomical mapping and tool alignment. Ultimately, Wang says, the broader ambition is to turn “every scientific research lab”—and soon every clinic—”into an AI-perceptible and AI-operable environment,” creating a system that can train professionals faster, catch errors, and improve human outcomes.

It’s time to stand up for science

If you liked this article, I would like to ask for your support. Scientific American has served as an advocate for science and industry for 180 years, and right now may be the most critical moment in its two-century history.

I have been one Scientific American subscriber since I was 12 years old, and it helped shape the way I see the world. SciAm always educates and delights me, and inspires a sense of awe for our vast, beautiful universe. I hope it does for you too.

If you subscribe to Scientific Americanyou help ensure our coverage is centered on meaningful research and discovery; that we have the resources to report on the decisions that threaten laboratories across the United States; and that we support both budding and working scientists at a time when the value of science itself is too often not recognised.

In return, you receive important news, captivating podcasts, brilliant infographics, can’t-miss newsletters, must-see videos, challenging games, and the world of science’s best writing and reporting. You can even give someone a subscription.

There has never been a more important time for us to stand up and show why science is important. I hope you will support us in that mission.