OpenAI just launched its decision model, making it widely available. This move puts them in direct competition with players like Jev, aiming for a share of the market for AI tools that help make choices. While tech giants push these models, new research questions their reliability, especially when answers are not clear-cut.
Decision models are AI systems that take options and return a probability distribution, indicating the likelihood of each choice being correct. Downstream applications use these probabilities to pick options. Providers, including OpenAI, emphasize their reliability, which is important for adoption.
However, a recent arXiv paper (arXiv:2610.29217v1) examined how three decision models—Jev, Liquid d1, and GPT-6 Luna Decisions—handle situations with multiple correct choices. The findings show that even when these models rank all correct answers above distractors, in 28% to 50% of those lists, at least one correct choice received a probability below the uniform probability of 1/n, where 'n' is the number of choices. This indicates the model considers a correct answer less likely than a random guess.
The research also found that if a model is given five identical copies of a correct choice, it does not assign them equal probabilities. Mean gaps in probability between these identical correct choices ranged from 17 to 79 percentage points. This asymmetric probability allocation affects how well these models identify multiple correct answers using a probability threshold.
Another study, also on arXiv (arXiv:2605.24771v2), looked at vision-language models (VLMs) for medical imaging. It combined small sets of expert labels with zero-shot pseudo-labels from VLMs like BiomedCLIP and PubMedCLIP. This is a common practice to boost accuracy when expert data is scarce.
The study found a crossover point. Once a classifier trained only on expert labels reaches the accuracy of the weak labeler, adding more AI-generated pseudo-labels lowers test performance. For example, on the ISIC 2019 dermoscopy dataset, adding 10,000 pseudo-labels after 20-50 expert labels reduced test macro-AUC by up to 0.14 points with BiomedCLIP. This means that past a certain point, the AI's 'help' introduces noise instead of value.
OpenAI is betting on decision models, hoping to capture enterprise clients who need reliable, automated decision-making. If these models do not allocate probabilities accurately, particularly in complex scenarios with multiple correct outcomes, their perceived value decreases. Companies investing in these tools for critical applications like medical diagnostics or financial trading could face significant risks, from flawed decisions to regulatory issues. The winners are the researchers exposing these limitations, pushing for more robust, transparent AI. The companies adopting these models without due diligence, or the AI providers whose claims of reliability exceed actual performance, could lose money.
Watch how OpenAI and its rivals address these calibration issues. Will they refine their models to handle multiple correct choices more gracefully, or will they continue to push them into markets where limitations could have serious consequences? We also need to see if researchers can develop better metrics or calibration methods to truly assess the reliability of these powerful but imperfect AI tools.
