AI STRATEGY
“ That means there is usually already a process,” he says.“ There is already some cost associated with that process. There is probably a baseline for what would have happened without the prediction. So, you can look back and ask: having had this prediction available, did we make a better decision or a worse one?”
Juan José notes that it might mean cost avoided, service level improved, response time reduced, inventory better allocated, fraud prevented, risk managed or simply being better prepared for one of the scenarios that the model identified.
“ But there is an important trap here,” he says.“ You cannot judge the quality of a past decision using information that only became available later. That is hindsight bias. A good decision can have a bad outcome and a bad decision can occasionally have a good outcome.”
Juan José says the quality of the decision has to be evaluated using the information that was available at the moment the decision was made:“ If you repeatedly discover that critical information was missing at that moment, then that tells you something useful: perhaps the problem is not the decision rule. Perhaps you need to change the information you make available to the decision-maker.”
This is also why model accuracy on its own can be misleading.
“ A 95 % accuracy number can look wonderful in a presentation,” he says,“ but if the remaining 5 % contains the cases that cost you the most money, it may be a terrible model for the business.
Juan José López Murphy
Global Head of Data Science and AI
Globant