< KAIST 백세범 석좌교수, (상단)KAIST 천정환 석사 >

 

"AI Must Also Be Able to Say 'I Don't Know'"

A new approach has been proposed to solve the problem of "overconfidence"—a phenomenon where Artificial Intelligence (AI) shows high confidence in incorrect predictions—which has been identified as a major risk in fields such as autonomous driving and medical diagnosis. A research team at our university (KAIST) has developed a learning method that enables AI to recognize situations it does not know, laying the foundation for reducing overconfidence and enhancing reliability.

A research team led by Professor Se-Bum Paik of the Department of Brain and Cognitive Sciences identified that random initialization—a widely used technique in deep learning where neural network weights are set randomly according to a probability distribution at the start of training—may be the fundamental cause of AI overconfidence.

To address this, the team proposed a "warm-up" strategy: briefly training the neural network with random noise (meaningless, arbitrary input data) before it learns actual data.

The researchers noted that the issue of AI overconfidence appears not only after training but is already present at the initialization stage, the very starting point of learning.

 

<AI-generated image: Comparison of confidence calibration between an AI model with warm-up training and one without>

 

Experiments revealed that when arbitrary data was fed into a randomly initialized neural network, the model exhibited high confidence levels even before learning anything. This characteristic can lead to hallucinations in generative AI, where the system convincingly produces false or non-existent information.

The research team found a clue to the solution in the biological brain. Even before birth and in the absence of external stimuli, the human brain forms neural circuits through "spontaneous neural activity"—brain signals generated internally without external input.

Applying this concept to artificial neural networks, the team introduced a "warm-up stage" where the model undergoes brief pre-training with random noise inputs before actual training begins. This process allows the AI to calibrate its own uncertainty before it starts learning in earnest. AI models that underwent this warm-up had their initial confidence levels aligned to low values—close to chance levels—significantly mitigating the overconfidence bias found in traditional initialization.

 

In essence, the AI learns the state of "I don't know anything yet" before it learns real data. As a result, the model's accuracy (the rate of correct predictions) and confidence (the degree to which the model believes it is correct) naturally aligned.

 

< Reliability test comparing accuracy and confidence of responses in artificial neural network models>

 

The difference was particularly evident in responses to unfamiliar data. While conventional models tended to provide wrong answers with high confidence for data they hadn't learned, the model with warm-up training improved its ability to judge "I don't know" by lowering its confidence.

 

This led to high performance in out-of-distribution (OOD) detection, a technique for identifying new types of data that were not included in the training set.

 

< Random noise warm-up training mimicking the development process of the brain>

 

This research suggests the possibility of AI possessing meta-cognition—the ability to recognize one's own cognitive state—going beyond simply getting the right answer to distinguishing between "what it knows" and "what it does not know."

Professor Se-Bum Paik explained, "This study is an example of how AI can recognize its knowledge state more similarly to humans by mimicking the brain's developmental process. Its significance lies in presenting a principle for AI to judge its own uncertainty, moving beyond just increasing accuracy."

This technology is expected to serve as a core method for increasing the overall reliability of AI. It can be applied to initialization methods for almost all deep learning models, as well as fields requiring high reliability such as autonomous driving, medical AI, and generative AI.

The research, with Jeonghwan Cheon (Master’s student in the Department of Brain and Cognitive Sciences, currently in the ROK Army) as the first author, was published online on April 9, 2026, in the international AI journal Nature Machine Intelligence. It was also featured in the "News & Views" section as a noteworthy paper.

• Paper Title: Brain-inspired warm-up training with random noise for uncertainty calibration (DOI: 10.1038/s42256-026-01215-x)
• News & Views Feature: Learning to be uncertain before learning from data (DOI: 10.1038/s42256-026-01205-z)

This research was conducted with support from the Basic Science Research Program of the National Research Foundation of Korea and the KAIST Singularity Research Project.