Accelerating AI Infrastructure Spending Outpaces Visibility and Control

Investments in AI infrastructure across 107 enterprises are rapidly increasing, but the ability to monitor and manage the economics of these investments lags behind. While most organizations currently rely on hyperscalers and model-provider APIs for their AI operations, there is a clear shift towards specialized compute that few are utilizing at present. A majority of enterprises are considering changing or adding providers within the next year, with many planning to do so within a quarter. The focus of buying decisions is on integration and total cost of ownership rather than just the token price, as many enterprises struggle to accurately track the costs of their compute resources. This disparity results in a significant compute gap, with heavy investments being made without the necessary visibility to control them.

Enterprise AI Infrastructure and Compute Landscape

According to VentureBeat Pulse Research, the current landscape of enterprise AI infrastructure and compute is characterized by a significant gap between investment ambition and operational reality. Only about 21% of enterprises are running AI at scale in production, indicating that the majority are still in the early stages of deployment. Despite this, spending intentions are high, with a focus on specialized AI clouds that are not currently being utilized by most enterprises. This mismatch between investment and operational maturity underscores the need for better visibility and control over AI infrastructure economics.

Key Findings

Finding 1: Ambition Outpaces Production

Only one in five enterprises are running AI in production at scale, with the majority still in the experimentation phase. This early stage of deployment has significant implications for infrastructure decisions, as organizations are likely to see their compute footprint and costs increase as they scale up their AI operations.

Finding 2: Enterprises Run on Hyperscalers and Model APIs

Currently, enterprises predominantly rely on hyperscalers and model APIs for their AI operations, with specialized GPU clouds playing a minimal role. The leading providers are Google Cloud, followed by other major cloud providers and model APIs. The shift towards specialized AI clouds is a notable trend, indicating a potential re-platforming of AI infrastructure in the near future.

Finding 3: The Next Dollar Goes to Infrastructure They Don’t Yet Run

Enterprises are planning to evaluate AI-specialized clouds and alternative accelerators in the coming year, despite the fact that these technologies are not widely used at present. This shift in evaluation priorities suggests a significant re-platforming of AI infrastructure is on the horizon, with a focus on new and emerging technologies.

Finding 4: A Switching Wave is Building

Six in 10 enterprises plan to change or add infrastructure providers within a year, with a significant number intending to do so within the next quarter. This high level of switching intent is unusual for a foundational category like compute, indicating a dynamic and rapidly evolving market landscape.

Finding 5: Nobody Buys on Token Price

Integration with existing systems and total cost of ownership are the primary factors influencing enterprise decisions when selecting an AI infrastructure provider. Token price is not a significant consideration, highlighting the importance of overall fit and economic viability in the decision-making process.

Finding 6: Expensive GPUs, Idle Most of the Time

Despite significant investments in GPU infrastructure, the majority of enterprises report GPU utilization rates of 50% or less. This inefficiency in resource utilization underscores the need for better visibility and control over AI infrastructure costs and performance.

Finding 7: Spending Fast, Measuring Slowly

Less than half of enterprises rigorously track the cost and return of their AI compute, indicating a significant gap between spending and measurement. While overall satisfaction with current infrastructure is moderate, many organizations struggle to accurately assess the value and cost-effectiveness of their AI investments.

Finding 8: The Next Bottleneck Few are Watching

As large-scale inference shifts from compute to memory, many enterprises are unprepared for this emerging constraint. The lack of awareness and preparedness for this shift highlights the need for better visibility and planning in AI infrastructure investments.

Conclusion

The current landscape of enterprise AI infrastructure is characterized by rapidly increasing investments that outpace the ability to monitor and control these investments effectively. Enterprises are facing a significant compute gap, with heavy spending on AI infrastructure occurring without the necessary visibility and control over costs and performance. The key takeaway is the importance of gaining better visibility and control over AI infrastructure economics to ensure that investments are made strategically and cost-effectively.

Based on survey responses from 107 qualified enterprise respondents (100+ employees), the findings highlight the need for organizations to prioritize visibility and control in their AI infrastructure investments to achieve long-term success and sustainability.