UT Austin psychologist Chen Yu studies how young children learn. What he’s finding might one day help our most advanced machine learning models learn, too. And it all begins with a baby, some toys, and a researcher fitting that baby with an eye tracker (see the video above).
The eye tracker is an off-the-shelf model that Yu’s lab has customized, and the latest models have three cameras. Two of them point inward, one at each eye. The third points out, like a GoPro camera, to capture whatever the baby is looking at. To the babies, the tracker just feels like a heavier-than-usual hat or headband. Once it’s attached, it’s forgotten.
Yu also equips parents with eye trackers, and he studies both the parents and children and how they interact at HOME — that is, his laboratory on the UT Austin campus, the Home-like Observational Multimodal Environment. It’s an apartment, basically, but one with nearly 30 cameras embedded in the ceiling and walls.
“It’s almost like going to an Airbnb,” said Yu. “You walk in, there’s a kitchen, a living room, all the things you’d expect in a home, and you just spend time there doing your everyday activities. We can even make it really messy or really clean and see how that changes people’s behavior.”
The goal is “precision in behavioral science.” It’s to measure precisely where people look, what they do with their hands, and what they say at every second, all in various contexts — and to measure it all without intruding on the natural interactions that matter most if you want to understand something as intuitive as how people teach and learn and interact.
Early in his career, before he had HOME, Yu did some experiments at home. He rigged up his own children with eye trackers and was able to suddenly see the world as they saw it. It quickly changed how he parented.
“I used to just point to things and say, ‘Look over there!’” he remembered. “But then I saw the recordings from the babies’ point of view, and I was like ‘Wow, they can’t even see what I’m pointing at!’ That’s when I realized babies see the world very differently from adults, partly because they’re smaller, but also because of how they move their bodies. That was actually our very first finding.”
Yu now relies on AI tools to process the multimodal stream of data produced by the recordings in the lab. All the objects in every scene are automatically detected and identified. The eye-tracking data is laid over that, and it’s all correlated as well with the conversation the parents are having with their kids.
“It allows us to capture how their behavior unfolds over time, and how all these different behaviors are coordinated in everyday activities,” said Yu.
In one recent study, published in the Proceedings of the National Academy of Sciences, Yu and his colleagues began by asking parents to read to their children a picture book filled with colorful animals and objects.
Using the head-mounted eye trackers to follow both parents’ and toddlers’ attention during shared book reading, they discovered that children’s own pointing was more effective than parents’ pointing in organizing attention and reducing uncertainty about word meanings. When the kids led the process of connecting a picture to a word, they learned the word more efficiently than when they followed their parents’ lead.
This is an example, said Yu, of what researchers call “embodied inference.” Rather than passively receiving information and struggling to figure out what words mean, children actively use their bodies, attention, and social interactions to simplify the learning problem itself.
“It’s a fundamental principle of human intelligence,” said Yu. “We learn efficiently not simply because we are good at processing information, but because our actions and interactions help create the right information to learn from.”
Yu and his collaborators have brought their methods to the study of children with autism, cochlear implants, and Down syndrome. They have been able to shed light, for instance, on the fact that children with autism, because they tend to look at faces less often than other children, can sometimes be misinterpreted as not coordinating attention with parents. But in studies Yu has worked on, many of them share attention with their parents just fine. They just follow their parents’ hands instead.
Collaborators in Britain have used clips like the ones that Yu records in his lab to advise parents of children with Down syndrome on how to help their kids learn, and also to reassure them that their children often are learning even if it doesn’t always appear that they’re paying attention.
What Yu has learned about humans has implications not just for human learners, but for machine intelligence as well. He is among the leaders of a new, $30 million National Science Foundation center, led by UT Austin, that will study what happens when robots leave the lab and move into the real world to live and work alongside people.
Yu is leading the Human Environment with Robots — or HERO — Facility Network, which is a network of real places where robots will live and work next to humans. Like in the HOME lab, Yu and his colleagues will track what people do and say. They’ll also track what the robots do and how people and robots adapt to each other over time in homes, dorms, cafes, a museum, a rehabilitation hospital, and elder-care residences.
One goal is to see whether what Yu has learned from watching children can help robots learn from their surroundings. The potential is enormous, he said, because for all their power, AI systems are often far less efficient learners than a toddler is. Today’s large language models are trained on something like ten thousand to a hundred thousand times more language than a child ever hears, but even so, they can’t extrapolate to new data in ways that toddlers can.
“Think for a moment,” he said, “how a machine learning algorithm learns to recognize an object. Basically, we download a huge amount of data from the internet, and then we carefully label everything. ‘This is a car, this is a car, this is a car.’ We show the algorithm thousands of examples, and eventually it learns to recognize a car. But children learn very differently. They don’t need thousands of labeled examples. They can learn from just a few experiences, and what’s really amazing is how well they can generalize. A child can play with a toy police car at home and then see a real police car on the street and recognize it immediately even though the two may look very different. That’s something that can still be challenging for machine learning systems, especially when they encounter objects or situations that are very different from their training data.”
In one published study, Yu and his colleagues took the recorded data from head cameras worn by children and fed it into a machine learning model to help train it on recognizing different objects. They did the same with the data coming from the cameras worn by the toddlers’ parents. The models trained on the child-produced visual stream substantially outperformed those trained on adult-produced data.
“Toddlers grab things, turn them over, hold them close, and in doing so they generate exactly the kind of varied data a learner needs,” said Yu. “This is evidence that we may be able to teach robots to do the same.”