Seeing the world through Tesla Autopilot V9 – 3D bounding boxes, human detection and beyond
Oct 23, 2018 | by Iqtidar Ali
Tesla’s V9 software update is a major release that adds and improves in-car features of the entire Tesla fleet as well as attempts to take the Tesla’s proprietary driver assistance system called ‘Autopilot’ to the next level.
A Tesla enthusiast and hacker who goes by the moniker of ‘greentheonly’ had previously hacked the Autopilot 8.1 system to record footage of how the Autopilot software looks at the world around a Tesla vehicle.
Now the same hacker has recorded how Autopilot sees the world around a Tesla car after the V9 update (videos below) — and yes this update includes major improvements over the previous ones.
Tesla Autopilot V9 now scans objects with 3D bounding boxes (3D Vehicle Detection) as opposed to 2D bounding boxes in the previous version, this is how the vehicle is able to detect different objects like Cars, SUVs, Trucks, Bikes and even Humans.
The video description describes why there is some lag in different camera angles:
This is running 18.39.6 firmware. All autopilot cameras are utilized.
This video does not show “narrow” stream because it’s somewhat redundant with main.
You may notice certain choppiness on all the cameras other than “main” – this is because while main camera is captured at 36fps and is therefore very smooth, the “fisheye” is only captured at 6fps and the sides are at 9fps (the car captures all cameras at 36fps and backup camera at 30fps) – I needed to limit the rate of the intake to not overwhelm my storage device (it is still struggling a bit and that’s why sometimes framerate drops on some cams).
It’s amazing that the Autopilot V9 cameras are now able to capture the video at 1280×960 resolution each except the backup camera — this translates to a ultra HD Autopilot V9 vision combined.
In the next part of the video below we can see Tesla Autopilot V9 in action on the highway and in the first few seconds of the video the vehicle encounters road debris and is able to detect it as an object but only when it came very close, let’s check out.
Even after the Autopilot V9 update, Tesla’s driver assistance software is not able to detect traffic lights or classify road signs yet, so let’s just wait and see if this is enabled before the release of Autopilot V10.
You can find additional notes on the videos from u/greentheonly on the Tesla Subreddit here.
- Testing ‘Autopark’ and ‘Autopilot’ with Tesla V9 update
- Tesla Model 3 V9 Update: Full release notes, screens and walk through video
- Tesla Model 3 update 2018.18.1 adds ‘Auto High Beam’
- 2018.14.1 update brings Chill Mode to Tesla Model 3
- Tesla asking customers to share external car camera footage to improve Autopilot safety
- Elon Musk’s Autopilot 2.0 Conference Call (full transcript and audio)