Autonomous vehicles encounter innumerous problems with huge variety in the real world, making it extremely crucial to find scalable ways to resolve the challenges. Data-driven paradigm with emerging end-to-end stack and world model demonstrated great potential in scalability, with the following two key questions remaining open. Can we efficiently leverage huge amounts of unlabeled data collected from real world towards foundation models for autonomy via self-supervised learning? Can we leverage the capabilities introduced by the language modality and LLM in traffic scene/behavior understanding and code generation towards generalizable autonomy stack with automated development/test pipeline? In this talk, recent works from Berkeley DeepDrive addressing the questions above are presented, such as self-supervised scene reconstruction for autonomy, dataset connecting interactive driving behavior and language reasoning, and LLM for code diagnosis and repair in autonomy stack.

In this session, you will learn more about:

• The key challenges in leveraging huge amounts of data for a data-driven paradigm of autonomy such as end-to-end stack
• Novel techniques on self-supervised learning and language modality utilized in leveraging large scale data collected from the real world and on the internet for autonomy stack and automated development pipeline
• New dataset connecting interactive driving behavior and language reasoning