Pete Florence: Dense Visual Representations, NeRFs, and LLMs for Robotics

In episode 54 of The Gradient Podcast, Andrey Kurenkov speaks with Pete Florence.Note: this was recorded 2 months ago. Andrey should be getting back to putting out some episodes next year. Pete Florence is a Research Scientist at Google Research on the Robotics at Google team inside Brain Team in Google Research. His research focuses on topics in robotics, computer vision, and natural language -- including 3D learning, self-supervised learning, and policy learning in robotics. Before Google, he finished his PhD in Computer Science at MIT with Russ Tedrake.Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on TwitterOutline:* (00:00:00) Intro* (00:01:16) Start in AI* (00:04:15) PhD Work with Quadcopters* (00:08:40) Dense Visual Representations * (00:22:00) NeRFs for Robotics* (00:39:00) Language Models for Robotics* (00:57:00) Talking to Robots in Real Time* (01:07:00) Limitations* (01:14:00) OutroPapers discussed:* Aggressive quadrotor flight through cluttered environments using mixed integer programming * Integrated perception and control at high speed: Evaluating collision avoidance maneuvers without maps* High-speed autonomous obstacle avoidance with pushbroom stereo* Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation. (Best Paper Award, CoRL 2018)* Self-Supervised Correspondence in Visuomotor Policy Learning (Best Paper Award, RA-L 2020 )* iNeRF: Inverting Neural Radiance Fields for Pose Estimation.* NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fields.* Reinforcement Learning with Neural Radiance Fields* Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language.* Inner Monologue: Embodied Reasoning through Planning with Language Models* Code as Policies: Language Model Programs for Embodied Control Get full access to The Gradient at thegradientpub.substack.com/subscribe