A Tesla hacker who goes with the twitter handle “Green”, in his recent tweet has revealed what Tesla’s driver-facing camera in Model 3 and Model Y is looking for — hinting at driver monitoring feature.
Tesla Hacker Reveals The Use Of Driver Facing Camera, Capable Of Doing More Than Just Monitoring Passengers
You may question how legitimate this guy is. Let me remind you back in January he was the first to claim, “Tesla has added a wireless device charger to its two oldest car models. The charger is allegedly integrated into the center console. Green also claims to have uncovered plans for a new type of charging port.” via his tweet. He didn’t stop there; he went on to claim that new battery options and a suspension redesign were also inbound.
While Tesla hadn’t confirmed or denied the hacker’s findings, and it hadn’t announced plans to update the Model S and the Model X, it quietly added a wireless phone charger to the list of standard features posted on its website. The remaining leaks are speculated to be included either in the Tesla Model S 2021, which is set to release in Jan 2021, or the new Tesla Plaid.
Why Tesla Model 3 is equipped with a driver-facing camera
In an equally controversial way, back in 2017 another tweet from Jason Hughes aka WK057 brought it to everyone’s attention that the official photos of the then-recent Tesla Model 3 had driver-facing cameras above the rearview mirror.
Am I the only one that noticed the @TeslaMotors #Model3 has a camera above the rear-view mirror facing inside the car? @elonmusk's #SpyCam?? pic.twitter.com/WYQnRwmklG
— Jason Hughes (@wk057) August 1, 2017
Hours after the tweet, that blew up and was spreading like wildfire on various online media outlets, Tesla confirmed that the Model 3 is equipped with the camera, but it is not currently active. It will only become active after future software updates, which Tesla didn’t want to elaborate on. At the time multiple automobile companies that offered any level of autonomous mode had started implementing a similar solution. Several autonomous driving experts believed that a driver-facing camera to monitor driver engagement is needed for level 3 autonomous driving – a level of autonomous driving where the driver is used as a backup to the autonomous system with a reasonable period to transfer the controls. The camera’s basic function was to provide safety and it always has been the same.
However, for almost 3 years the camera in Tesla Model 3 was inactive. As Tesla progressed and started offering multiple levels of autonomy in their cars the camera’s role became crucial. We may question why put a camera and not use it for 3 years? But this just shows the meticulous planning of Tesla. They didn’t have to recall a single car and every customer in the future could enjoy the autonomous features. Tesla saved enormous man-hours by adding the camera well in advance. In April this year, Elon Musk revealed the reason behind the driver-facing camera. Tesla has yet to implement a feature using this camera, but a software update once referred to the camera being linked to Autopilot and “Tesla Network,” which is supposed to be an upcoming fleet of Robo-taxis operated by Tesla.
The camera’s role was to capture, relay, and make sure there is an action taken on any type of misconduct, vandalism, etc. Again on the scene was Tesla hacker, now gaining followers in heaps, “Green”. This time the hacker was able to retrieve the cameras field of vision.
Other quashed dreams: none of the mirrors are visible and overall pretty much nothing outside the car, so people hoping this will add coverage for autopilot are out of luck.
Now what are YOUR theories about the purpose of this cam in light of this info?
— green (@greentheonly) April 12, 2020
The driver and passenger seats, as expected, are easily visible. The same cannot be said about the rear passengers. Important to note is that the dash and glove box arent visible in the above images. After seeing the tweet, unsurprisingly, speculations surrounding what the actual purpose of this camera is gained pace. Green tweeted, “The sensor is the same aptina as the rest of the cams. The color filter appears to be RCCB but delayering failed so far. Resolution is also the same 1280×960 Besides driver pose eye-tracking also seems to be possible at least to a degree” he teased his followers by adding, “Other quashed dreams: none of the mirrors are visible and overall pretty much nothing outside the car, so people hoping this will add coverage for autopilot are out of luck. Now, what are YOUR theories about the purpose of this cam in light of this info?”
The latter quote hinted at the possibility of driver reaction track. Today we got the confirmation from Green himself.
Decoding the leaks
When Tesla initially activated the cameras in the Tesla Model 3, they started collecting images and clips for research purposes with consent from the drivers:
“Help Tesla continue to develop safer vehicles by sharing camera data from your vehicle. This update will allow you to enable the built-in cabin camera above the rearview mirror. If enabled, Tesla will automatically capture images and a short video clip just before a collision or safety event to help engineers develop safety features and enhancements in the future. As usual, you can adjust your data sharing preferences by tapping Controls > Safety &Security > DATA Sharing > Camera Analytics.”
Nowhere in this quote does Tesla suggest that they will actively capture drivers’ attention, but hinted at using the data to improve future features of the car. It led some to believe that Tesla could be working on eventually releasing active driver monitoring features.
Now Tesla hacker ‘green‘, known for revealing many features in Tesla’s software, has discovered what events the automaker is trying to detect with the driver-facing camera:
- BLINDED
- DARK
- EYES_CLOSED
- EYES_DOWN
- EYES_NOMINAL
- EYES_UP
- HEAD_DOWN
- HEAD_TRUNC
- LOOKING_LEFT
- LOOKING_RIGHT
- PHONE_USE
- SUNGLASSES_EYES_LIKELY_NOMINAL
- SUNGLASSES_LIKELY_EYES_DOWN
Tesla’s only active driver monitoring feature when Autopilot is engaged is detecting if torque is being applied to the wheel.
This information may seem controversial at first glance but it is aimed at improving driver monitoring because it’s currently one of the biggest weaknesses of Tesla’s Autopilot. Tesla has its customers’ consent and it seems that they will surely pay back this trust.
