Challenges in Auditing Frontier Models: Failing Rates on the Rise

AI agents have become an integral part of enterprise workflows, but they still struggle with reliability, failing in about one out of three attempts on structured benchmarks. This gap between capability and reliability is the main operational challenge for IT leaders in 2026, as highlighted in Stanford HAI’s ninth annual AI Index report.

Referred to as the “jagged frontier” by the AI Index, this uneven and unpredictable performance is where AI excels in certain tasks but then suddenly fails. AI models can excel in complex tasks like winning a gold medal at the International Mathematical Olympiad but still struggle with basic tasks like telling time accurately.

In 2025, there were significant advancements in AI models across various benchmarks. For example, frontier models saw a 30% improvement in just one year on Humanity’s Last Exam, a challenging test that includes questions from various fields. Leading models also performed well on benchmarks like MMLU-Pro, τ-bench, GAIA, SWE-bench Verified, WebArena, and MLE-bench, showcasing their progress in different domains.

AI agents have shown improvement in cybersecurity tasks, with frontier models solving a high percentage of problems on benchmarks like Cybench. Additionally, models have made significant progress in video generation, with the ability to simulate real-world behaviors and interactions.

Despite these advancements, AI systems still face challenges in areas like visual reasoning, hallucinations, and multi-step reasoning. Models struggle with basic perception tasks like telling time accurately and often fail in tasks that require multi-step workflows.

Furthermore, the reliability of AI benchmarks is also facing challenges, with error rates reaching as high as 42% on some evaluations. Benchmark saturation is becoming a concern, as models achieve high scores that no longer differentiate their performance effectively.

As AI capability continues to surge, concerns about data bottlenecks, responsible AI practices, and the transparency of model development are also growing. Researchers are exploring hybrid approaches using synthetic data to improve model performance and are emphasizing the importance of responsible AI practices in light of increasing capabilities.

In conclusion, while AI is advancing rapidly in terms of capability, the gap between what AI can do in a controlled environment versus real-world applications remains a significant challenge. Transparency, responsible AI practices, and reliable benchmarking are crucial aspects that need to be addressed to ensure the successful integration of AI in enterprise workflows.