Insights from NVIDIA Research March 22, 2022 Bill Dally Chief Scientist and SVP of Research, NVIDIA Corporation NVIDIA Research Perception & Learning Graphics Programming Systems Architecture Circuits ? Supply Demand Integration/Moonshots Networks VLSI Robotics Applied DL Research Toronto AI Tel Aviv AI AI Algorithms Autonomous Vehicles GPU Storage Systems TECHNOLOGY TRANSFER GREATEST HITS RTX NVSwitch CuDNN 4 Autonomous Vehicle Research NVIDIA DRIVE AV PLATFORM End - to - end, Open & Modular DATA COLLECTING, TESTING MAPPING, TRAINING SIMULATION DRIVE CHAUFFEUR DRIVE CONCIERGE DGX A100 DRIVE Constellation AV on DRIVE AGX IX on DRIVE AGX DRIVE Hyperion Robust & Human - centered Autonomy Data - driven Modular AV Stacks Safety Assurances x Perception Prediction Planning Control Mapping AUTONOMOUS VEHICLE RESEARCH GROUP Next - generation AV Autonomy Stacks For more information about the Autonomous Vehicle Research Group: https:// nvr - avg.github.io / TOWARDS ROBUST AUTONOMY Addressing uncertainty and error propagation throughout the autonomy stack Sensor Data Tracking Prediction Planning & Control Detection State uncertainty from object detection Class uncertainty from object classification Planning with multimodal predictions TOWARDS ROBUST AUTONOMY Making prediction robust to tracking errors Sensor Data Tracking Prediction Planning & Control Detection Object tracking errors ID Switch Ground Truth Tracked Trajectory Fragmentatio n MAKING PREDICTION MORE ROBUST Object tracking is not perfect ID Switch Fragmentatio n Ground Truth Tracked Trajectory Tracking errors detrimentally affect prediction MAKING PREDICTION MORE ROBUST Predictions after an ID switch A false positive detection causes an ID switch , yielding errant predictions. Predictions with accurate tracking Accurate tracking yields reasonable predictions. Prediction errors increase by 10 - 30x in the presence of tracking errors Is tracking necessary to make predictions? Prediction errors decrease by up to 80% when using our affinity - based prediction method Past object tracklets Predictions Data Associatio n Transformer Affinity Information Detections Removed, no more tracking MAKING PREDICTION MORE ROBUST Key Insight: Directly use detections and affinity as prediction inputs Is tracking necessary to make predictions? Standard Tracking - Prediction Pipeline Our Affinity - based Prediction Method MAKING PREDICTION MORE ROBUST HISTORY OF DRIVING SIMULATORS CarSim High - fidelity vehicle dynamic simulator 1990 Virtual KITTI Synthetic sensor image 2016 Nvidia DriveSim Photo - realistic rendering as part of Omniverse 2021 CARLA Urban driving simulation with UE4 2017 ✓ high - fidelity physics simulation ✓ high - fidelity rendering (UE, Omniverse) ? intelligence that generates realistic, human - like driving behavior Real - world driving log ML - based behavior model Learn High - fidelity simulator Deploy TOWARDS BETTER DRIVING SIMULATION Grounding traffic behavior synthesis in real human driving behaviors Desiderata Fidelity: can we synthesize agent group behaviors that resemble real - world traffic? • Diversity: can we generate a wide range of realistic traffic patterns? • Controllability: can we steer the traffic model to generate a specific scenario? Challenges • Stability: prevent the simulation from diverging out of nominal behaviors • Long - horizon: run simulation with horizon much longer than training data horizon • Adaptivity: adapt to new regions / traffic patterns with little to no tuning TOWARDS BETTER DRIVING SIMULATION Grounding traffic behavior synthesis in real human driving behaviors MODELING HUMAN BEHAVIORS BEYOND PREDICTION Closed - loop decision making requires rethinking behavior modeling Ground Truth Trajectory Rolling out a Prediction Model CLOSED - LOOP DRIVING BEHAVIOR SIMULATION Hierarchical Decision Making : Decoupling Where from How Hierarchical Policy = Spatial Map Planner + Goal - conditional Controller Rollout Spatial Probability Map SYNTHESIZE NEW AND DIVERSE TRAFFIC BEHAVIORS Sample behavior model to synthesize counterfactual scenarios Stay in the same lane Switch to the right lane SYNTHESIZE NEW AND DIVERSE TRAFFIC BEHAVIORS Decentralized Decision Making : Emergence of Complex Interactive Behaviors All agents in the scene are controlled by a learned traffic model 20 AI For Electronic Design