AI Talk: The AI consumption squeeze — balancing performance, sovereignty, and cost at enterprise scale

Views on news 

For the last 24 months, one narrative justified every over-provisioned data centre and bloated IT budget: the GPU scramble. Gartner estimates AI infrastructure is adding $401 billion in new spending this year. Real-world audits tell a darker story: average GPU utilization in the enterprise is stuck at 5%. Many organizations locked in GPU capacity under traditional three- to five-year depreciation cycles. That means the infrastructure purchased during the peak of the “GPU scramble” is now a fixed cost, regardless of how much it is actually used. This is forcing a shift in mindset from acquiring capacity to maximizing the economic output of what is already deployed. The underlying reason for much below capacity GPU use is that businesses don’t have access to the right information fast enough. As a result, so called AI factories are shifting from single to multi-tenant environments. The challenge now is to move data sprawls to GPUs with minimum complexity. However, underutilisation isn’t specific to GPUs but affects the whole chain: even using less GPU capacity doesn’t necessary mean better business outcomes, as organisations often lack the ability to operationalise AI efficiently and securely.  

 

The trade-offs of hybrid cloud for AIaaS 

Companies want to see the business value before they make investments in GPUs. Being aware of how much data AI generates, businesses want to ensure that they can access this data, as well as maximise its utilisation. A hybrid approach allows you to use data efficiently on-prem, while also continuing workflows that you’ve started in the cloud.  At the same time, there is also a move away from vendor lock-in to free, open source platforms. As an emerging trend, 70 per cent of enterprise customers now want to have a repatriating strategy. Recent EU regulations have also made companies more aware of where their data sits and whether it’s stored securely. Choices about public and on-prem cloud will impact a series of decisions from governance to integrations to how processes should be reengineered and the type of talent the company needs to hire.  

 

Data requirements depend heavily on the type of AI used. Training, for example, uses huge amounts, fine-tuning is less data intensive but must rely on company data and therefore tends to be on prem. With agentic AI, however, you need a completely different model to access all sorts of different data at speed. Neoclouds or AI hyperscalers charge clients in tokens. But compared to human users, agentic AI burns tokens much faster. So much so, that some companies have decided to switch off their AI agents to save money. This makes a case for keeping data on-prem, as this way the extortionate fees can be avoided. The aaS model can provide some level of predictability in terms of costs, as well as flexibility against spikes.  Having carefully considered objectives and business outcomes and any organisational transformations including M&A, a business must also factor in term capacity and regulation – recently the most relevant being the one on AI sovereignty. Once all factors have been considered, an aaS platform matching the company’s needs can be a good model to achieve predictability of costs.    

 

Netapp has been considering launching a service designed for organisations that want to be data sovereign by EU standards. The business value of data sovereignty is hard to assess at this point with some points of the legislation clear and others still taking shape. However, businesses must assess what data sovereignty brings for them in the medium and long term. Eventually, concerns that EU sovereignty regulation will slow down innovation may prove unjustified, provided sovereignty is applied proportionally. Businesses embracing AI must perform a risk management exercise to identify where their critical risks lie and where they can be more relaxed about guardrails.  Where companies deploying AI are prone to make mistakes is data access. In Netapp’s experience, about 80 per cent of proofs of concept fail because the GPU can’t get access to data. Moreover, with  the rapid speed of AI advancement, it’s also key for businesses to strike a balance between control and adaptability.  

 

The panel’s advice 

• There is too much focus on GPUs and not enough on infrastructure.  
• Not all data has the same value for an enterprise and different data can generate different amounts of revenue. 
• With the latest AI models relying more firmly on context, you will need less data.   
• Sovereignty can include both the on-prem management of corporate data or a countrywide platform that keeps data and revenue generated with that data within a certain jurisdiction. However, in some geographies, sovereignty also applies to the country where the platform or solution has been developed.