Autonomous Car - what goes into sensing for autonomous driving?

Here is a great, very accessible video where the CTO of MobileEye (which provides some of the components for Tesla's "autopilot") explains his views on the challenges of end-to-end learning for autonomous vehicles, and why it's preferable to decompose the problem instead (still using deep learning for the decomposed modules). [this video] I'm inclined to agree, especially because it helps in 1) providing diagnostic information (such as the great driving visualizations shown in the video), and 2) makes it easier to incorporate algorithms and sensors (like with Google's cars) as a redundancy in case the neural network hits a crazy edge case.

Those are good points. Here's some backup. Here's Mobileeye's CTO talking in detail about their technology.[1] They use a a deep neural network that recognizes cars. Just cars. Look at this point in the video.[2] There's a stationary obstacle on the left, and it doesn't recognize that. Maybe this is why Tesla, using Mobileye hardware, plowed into three different stationary vehicles at the left of a highway. They were all trucks (one was a street sweeper), and seen from the back while stationary, may not have matched the trained model of "car". Mobileeye's algorithms are good at recognizing moving cars from most angles, and that's valuable for feeding into the model of "what's the other player doing". But it's not enough for the "where are all the obstacles" and "where is the road surface" questions. Those questions have objective answers, which can be answered with a LIDAR, and maybe millimeter radar if it has the angular resolution. [1] [this video] [2] [this video]