Head-to-head: Human vs. AI-human teams

With 70% of U.S. companies expected to adopt AI by 2030, organizations need to understand and address the key challenges of employee-AI interaction. To shine light on the future of AI in the workplace, Chris McComb and Jon Cagan put human and AI-human “hybrid" teams to the test. In a head-to-head matchup, researchers studied which team could better design a fleet of delivery drones under dynamic design constraints.

“We compared teams of five humans against hybrid teams of three humans and two AI teammates,” explained McComb, associate professor of mechanical engineering. “We wanted to understand failure modes in human-AI teams, so midway through the experiment we added some drastic changes that the AI agents couldn’t see.” 

The study was conducted on the online platform, HyForm, developed by researchers at CMU and Penn State University, so that while teams worked to maximize the profit of their drone delivery service, researchers could monitor both the teams’ communications and the success of their fleets.

At the end of the day, the human-AI hybrid teams performed just as well as the human teams, demonstrating the ability of hybrid teams to adapt to unexpected events.

HyForm, an online platform, enabled teams to work remotely on their drone delivery service

Researchers found that overall, the human teams communicated significantly more than the hybrid teams until the unplanned-to-the-user constraints were introduced. Communication within the human-AI teams nearly tripled from that point forward. Much of this increase was due to communication between humans and their AI teammates, indicating that some human members of the team took on the role of “AI handler” to keep the AI agents on track. 

Despite this success, members of the hybrid teams felt that they did not operate cohesively or effectively.

“It’s important for AI adopters to see that the flexibility of human-AI teams can come from the human side of the partnership,” said Cagan, department head and professor of mechanical engineering. “We don’t need to develop a perfectly adaptive AI partner.”

Humans are an essential part of human + AI teaming. Future engineers need to be trained to work effectively with AI agents.

Chris McCombAssociate Professor, Mechanical Engineering

“The bottom line here is that humans are an essential part of human + AI teaming, and future engineers need to be trained to work effectively with AI agents,” emphasized McComb.