Why did a Tesla car mistake a truck with a bright sky?
Tesla's technology is assistive, as Alexey points out, so this is not a case of an autonomous system (e.g. an AGI) doing some fatal stunt (the product name AutoPilot is famously misleading). Now on why the car assistance led to this tragic accident, there is some information related to AI technologies.



Warning: I cannot find again the source critical to the next paragraph, and reading again pages over pages, I cannot find similar argument in other reports. I still remember vividly the point below, but please keep in mind it may be incorrect. The rest of the answer is weakly related, so I leave it all, with this warning.

An independent report (link needed, I can't find it...) explained that the assistive system was unable to detect the truck due to an exceptionally low contrast (bright sky perceived as white---colour of the truck). The report also said that a human driver would have been unable to make the difference either. In other words, it is possible that car sensors (presumably camera) and the human eye could not have detected an obstacle, and could not have triggered any safety measure. This short graphical explanation sums up the car sensors: Camera, radar, GPS, etc.

The assistive sub-system is based on proprietary AI technologies. We can only speculate under some hypothesis. _This is not very useful, honestly, except for illustration purpose. Assuming the assistive system relies on ML technologies to learn about obstacles from a video stream (such systems do exist):


It may be that the learning data was not "good enough" to cover the truck scenario.
It may be the technology was not powerful enough yet (lack generalization power, or simply too slow).
It may be a hardware problem, notably from the sensors: If the "car's eyes" are defective, the "car's brain" (the assistive system) is unable to react properly.


Why those technologies did not work in that case will remain a secret. We can say however that any system---whether built with AI technology or not---has limits. Beyond these limits, the system reaction is unpredictable: It could stop, reset, shutdown. The difficulty here is to define what a "default behaviour" is. A machine will basically do whatever it is designed to do, so an AI-based system too.

We could speculate even more on what would happen if the assistive system was really autonomous, the elusive AGI, but that is really not the case here.