Raising the Bar: Google’s ‘FACTS’ Benchmark and the Importance of Factuality in Enterprise AI

The realm of generative AI benchmarks is vast, with numerous assessments aimed at evaluating the performance and accuracy of models in completing various enterprise tasks. From coding to instruction following to agentic web browsing and tool use, these benchmarks have traditionally focused on measuring a model’s ability to solve specific problems rather than its factual accuracy in generating information tied to real-world data, particularly in dealing with imagery and graphics.

In critical industries such as legal, finance, and medical fields where accuracy is paramount, the absence of a standardized method to measure factuality has been a significant challenge. However, Google’s FACTS team and Kaggle’s data science unit have introduced the FACTS Benchmark Suite to address this gap effectively.

The FACTS Benchmark Suite presents a well-defined evaluation framework that delves into the concept of factuality, categorizing it into two operational scenarios: contextual factuality (grounding responses in provided data) and world knowledge factuality (retrieving information from memory or the web).

Despite the headline news of Gemini 3 Pro’s top-tier position in the benchmark, the industry faces a notable “factuality wall.” Initial results indicate that no model, including Gemini 3 Pro, GPT-5, or Claude 4.5 Opus, managed to achieve a 70% accuracy score across a range of problems. This underscores the ongoing need for technical leaders to adopt a “trust but verify” approach.

The FACTS suite goes beyond traditional question-and-answer formats, comprising four distinct tests that simulate real-world failure modes encountered during development and production. These tests include the Parametric Benchmark, Search Benchmark, Multimodal Benchmark, and Grounding Benchmark v2, each assessing different aspects of a model’s performance.

With Google making 3,513 examples public and Kaggle holding a private set to prevent data contamination, developers can engage with the suite without the risk of biased training.

The leaderboard reveals Gemini 3 Pro leading with a FACTS Score of 68.8%, followed by Gemini 2.5 Pro and GPT-5. However, a deeper analysis exposes areas of improvement for engineering teams, particularly in the Search and Multimodal benchmarks.

For developers focusing on RAG systems, the Search Benchmark emerges as a critical metric, highlighting the disparity between a model’s internal knowledge and its ability to search for information. This emphasizes the importance of integrating search tools into models for enhanced accuracy.

The Multimodal tasks pose challenges for product managers, with low accuracy rates across the board indicating that Multimodal AI may not be ready for unsupervised data extraction. This has implications for tasks such as data scraping and financial chart interpretation, where errors could be introduced without human oversight.

Overall, the FACTS Benchmark is set to become a standard reference point for evaluating models in enterprise settings. Technical leaders are advised to scrutinize specific sub-benchmarks aligned with their use cases to make informed decisions about model selection.

As models continue to evolve, the industry must acknowledge that while AI is advancing, it is not infallible. Designing systems with the understanding that errors may occur is crucial for maintaining accuracy and reliability in AI applications.