Control-aware predictions aim to teach autonomous vehicles what's important and what isn't

Designed to address the problem of a mismatch between intermediate objectives in a modular pipeline and overall system performance, CAPOs — Control-Aware Prediction Objectives — seek to inject a little common sense into future autonomous vehicles.

author avatar

10 May, 2022

Tested in the CARLA autonomous vehicle simulator, the novel CAPO approach to autonomous vehicle training showed improved performance over the current state-of-the-art in key metrics.

Tested in the CARLA autonomous vehicle simulator, the novel CAPO approach to autonomous vehicle training showed improved performance over the current state-of-the-art in key metrics.

Autonomous vehicles have the potential to dramatically reduce collisions, both between vehicles and between vehicles and pedestrians, owing to the speed at which a machine can react compared with a human operator. Couple that with machines never getting tired, or distracted, or experiencing road rage, and autonomous vehicles can offer a dramatic improvement in safety — but sometimes their attention can be a little too broadly focused.

A team of scientists at the Toyota Research Institute and the University of California, Berkeley, have highlighted a potential shortcoming in current implementations: The autonomous vehicle concerns itself with the movement of pedestrians and other vehicles even when a collision is entirely impossible, such as when the vehicle has already passed — something they claim they can solve with CAPOs, Control-Aware Prediction Objectives.

Common sense for computers

“A problem in the literature is that forecasting metrics are typically unaware of their usage, for example as input into a downstream planner,” Rowan McAllister, first author of the paper, explains in a brief presentation unveiling the research. “This is a problem called ‘objective mismatch.’ Our proposed solution is to weigh forecasting metrics by their effect on downstream control and the benefit is this improves forecasting accuracy where it matters most — like potential collisions.”

The issue, the team explains, is that a traditional autonomous vehicle software pipeline — made up of modular components designed to address individual sub-tasks with their own sets of objectives — doesn’t necessarily provide the best outcome for overall system performance. The biggest issue: Errors in prediction modules, which are effectively treated equally whether they have little to no effect on overall outcomes or the potential for fatal consequences if ignored.

“Some predictions warrant more attention than others when deciding safe controls,” the team argues. The result: CAPOs, an approach to training prediction models which takes into account the relative effects of prediction errors on downstream control. Designed to be implemented with only forward passes, making it easily applicable to real-world autonomous vehicle planning and control systems, CAPOs are designed to offer a prediction loss function which is aware of, and makes use of, how predictions will be used downstream in the pipeline.

A question of import

By comparison, the team highlights a traditional prediction metric concentrating on two pedestrians: One walking ahead of the autonomous vehicle and the other walking behind. “This prediction metric is equally concerned with both,” the team writes. “Intuitively, accurate prediction of the pedestrian ahead of the ego vehicle is more important for safe motion planning since the ego’s planned path is more likely to intersect with [the pedestrian in front than] behind.”

Two images captured from the CARLA autonomous vehicle simulation tool. The left-hand image shows a planner predicting a collision with a crossing pedestrian and slowing accordingly; the right-hand image shows the vehicle having safely crossed the road segment where the pedestrian has already been.CAPOs could, their creators claim, improve the performance of autonomous vehicles by allowing them to effectively ignore errors which have little impact — such as the pedestrian in the right image having already crossed the road.

CAPOs, as proposed, can address this drawback with an gated recurrent unit (GRU) encoder-decoder architecture which includes an attention mechanism designed to weight predictions using attention factors influenced by the autonomous vehicle’s future trajectory. The actual trajectory prediction, however, isn’t the key: Instead, the important output is made up of the computed weighs of, in the case of the example detailed in the paper, pedestrian contributions to the vehicle’s behaviour.

The team also proposed a second approach, a counterfactual re-weighted maximization objective with the log likelihood of an agent’s trajectory weighted by its individual contribution to the autonomous vehicle’s control decision and the exchange of predicted trajectories for ground truth trajectories. Additional testing would be required to determine a clear advantage to either approach.

Testing both approaches in the CARLA autonomous driving simulator, set up for pedestrian prediction, the team found that CAPOs scored lower than competing approaches for average displacement error (ADE) but scored the highest on control error and success rate — “the metrics,” the team argues, “that matter more,” and which represent mitigation of important errors propagated downstream and thus a measurable improvement in end task performance.

The team’s paper has been published on Cornell’s arXiv preprint server under open-access terms, following its acceptance to the IEEE International Conference on Robotics and Automation 2022 (ICRA '22).


Rowan McAllister, Blake Wulfe, Jean Mercat, Logan Ellis, Sergey Levine, and Adrien Gaidon: Control-Aware Prediction Objectives for Autonomous Driving, Accepted at the IEEE International Conference on Robotics and Automation (ICRA) 2022. DOI arXiv:2204.13319 [cs.LG].

10 May, 2022

A freelance technology and science journalist and author of best-selling books on the Raspberry Pi, MicroPython, and the BBC micro:bit, Gareth is a passionate technologist with a love for both the cutting edge and more vintage topics.

Stay Informed, Be Inspired

Wevolver’s free newsletter delivers the highlights of our award winning articles weekly to your inbox.