About cookies on this site Our websites require some cookies to function properly (required). In addition, other cookies may be used with your consent to analyze site usage, improve the user experience and for advertising. For more information, please review your options. By visiting our website, you agree to our processing of information as described in IBM’sprivacy statement. To provide a smooth navigation, your cookie preferences will be shared across the IBM web domains listed here.

- Generative AI is unlike any technology that has come before. It’s swiftly disrupting business and society, forcing leaders to rethink their assumptions, plans, and strategies in real time.
- To help CEOs stay on top of the fast-shifting changes, the IBM Institute for Business Value (IBM IBV) is releasing a series of targeted, research-backed guides to generative AI on topics from data security to tech investment strategy to customer experience.
- This is part 22: Cost of compute.
Generative AI is unlike any technology that has come before. It’s swiftly disrupting business and society, forcing leaders to rethink their assumptions, plans, and strategies in real time.
To help CEOs stay on top of the fast-shifting changes, the IBM Institute for Business Value (IBM IBV) is releasing a series of targeted, research-backed guides to generative AI on topics from data security to tech investment strategy to customer experience.
This is part 22: Cost of compute.
Generative AI is unlike any technology that has come before. It’s swiftly disrupting business and society, forcing leaders to rethink their assumptions, plans, and strategies in real time.
To help CEOs stay on top of the fast-shifting changes, the IBM Institute for Business Value (IBM IBV) is releasing a series of targeted, research-backed guides to generative AI on topics from data security to tech investment strategy to customer experience.
This is part 22: Cost of compute.
Catalyze cost of compute
Cost of compute may seem like an IT concern—and as recently as 24 months ago it would have been. But generative AI is elevating it to the C-suite. That’s because, left unchecked, the massive computing resources required to power gen AI can quickly lead to a corresponding surge in unexpected costs that, in turn, might jeopardize innovation and business transformation.
By understanding what drives the computing costs of gen AI, CEOs can make more informed investment decisions, setting strategic priorities that make innovation and transformation more cost-efficient.
For instance, enterprises must make significant capital expenditures or operational investments to ensure they have the dedicated cloud capacity gen AI demands. But compute capabilities and servers are only part of the equation. There’s also storage, data centers, and networking equipment and services to consider—as well as all the energy it takes to power gen AI systems.
When added together, these unexpected costs can send budgets skyrocketing. The CEOs that best manage these costs will be able to run their business like a high-performance machine—reducing drag while using the latest technology to outpace the competition. In this way, cost of compute can offer a competitive advantage. While other organizations struggle to fit gen AI into their budget, those that wrangle costs effectively can overcome financial obstacles and leapfrog into the future.
IBM Institute for Business Value research has identified three things every leader needs to know:
1. Costs can derail your best-laid generative AI plans.
2. Hybrid by design makes scaling generative AI affordable.
3. Generative AI stretches your computing budget.
And three things every leader needs to do right now:
1. Get a grip on your cost of compute.
2. Forge a unified front with generative AI and hybrid cloud.
3. Move lightning-fast at a lower cost.
Additional content
Meet the authors
Anthony Marshall, Senior Research Director, Thought Leadership, IBM Institute for Business ValueCindy Anderson, Global Executive for Engagement and Eminence, IBM Institute for Business Value
Christian Bieck, Europe Leader & Global Research Leader, Insurance, IBM Institute for Business Value
Jacob Dencik, Ph.D, Research Director, IBM Institute for Business Value
Download report translations
Originally published 07 October 2024
1. Scaling
+ Generative AI
What you need to know
Costs can derail your best-laid generative AI plans
Gen AI is shifting computing costs into high gear. Overall, the average cost of compute is expected to climb 89% between 2023 and 2025—and 70% of executives say generative AI is playing a key part in driving this increase.
As a result, many organizations are pulling back on the throttle. Every single executive we surveyed said their organization has cancelled or postponed at least one gen AI initiative due to cost of compute concerns. On average, 15% of projects have been put on hold and 21% of gen AI initiatives have failed to scale for this reason.
While many gen AI activities contribute to cost of compute—from training and fine-tuning models to storing data and powering processing—most of these will be channeled through cloud computing. Cloud costs associated with deploying gen AI are now twice as high as the cost of the models themselves—and this gap is widening as clouds become the engine rooms where gen AI is built and run. It’s a catch 22: without the proper oversight, the cloud services required to scale generative AI can become the top cost barrier to scalability.

To break out of this paradox, CEOs must set clear cost objectives for gen AI programs, establish a cost governance framework, find ways to collaborate with partners to reduce costs, and invest in more efficient architectures that can help optimize costs.
For instance, a hybrid cloud architecture lets organizations bring gen AI directly to the data and applications it must be paired with to deliver business value. Using a hybrid cloud platform that includes a common control plane and FinOps capabilities gives leaders the visibility they need to run data, workloads, and applications in the lowest-cost environments. However, despite its potential, only 26% of organizations are currently leveraging cloud platforms and container orchestration technologies to a great extent to reduce their cost of compute.
As gen AI complicates the cost equation, CEOs will need to put expenses under the microscope to manage them more effectively. A new class of AI costs, from data labeling to model customization, could quickly balloon budgets if they aren’t carefully controlled. Understanding model optimization best practices—and deploying compute power with precision—will be essential to improve profitability with gen AI.
What you need to do
Get a grip on your cost of compute
Pinpoint the factors driving up generative AI expenses—and stay ahead of the curve as your projects scale. Set clear cost guardrails and assess compute needs as early as possible in project planning to avoid expensive surprises down the line.
- Identify cost catalysts. Understand how different elements influence gen AI costs—including hardware, cloud services, model selection and training, data collection and cleaning, integration, and maintenance—and how these drivers can change as you move from pilots to projects at scale. Set clear cost control parameters to guide every gen AI decision and give your teams the tools they need to assess, monitor, and manage the cost of compute implications of gen AI at every stage.
- Recalibrate your computing resources. Conduct a full lifecycle cost assessment to preempt and anticipate compute needs. Invest in more cost-efficient infrastructure, fit-for-purpose models, and workload optimization tools to keep costs manageable as you scale. Collaborate with partners to reduce training, fine-tuning, and development costs.
- Use FinOps and cloud optimization to cut gen AI costs. Use a hybrid cloud platform as your cost of compute control tower. Deploy kubernetes to manage workloads and services in containers to roll out gen AI applications in the most consistent and efficient manner. Monitor the evolving costs that come with gen AI, from data storage to model retraining and fine-tuning to security and compliance, to avoid taking an unexpected hit to the balance sheet.
2. Hybrid cloud
+ Generative AI
What you need to know
Hybrid by design makes scaling generative AI affordable
Not every application of gen AI is created equal. Each use case has its own compute, data, and privacy requirements. That’s why hybrid cloud, which lets organizations use the most cost-effective infrastructure for every workload, is quickly becoming the go-to architecture for helping organizations achieve their gen AI goals at scale—without breaking the bank.
Overall, 72% of executives agree that hybrid cloud will be essential for scaling gen AI and managing the cost of compute. This figure rises to 85% for organizations that have moved beyond pilots and into full-fledged gen AI projects. But to tap the full potential of hybrid cloud for gen AI, you need to extend its principles across your platforms, security, AI, cloud, and data efforts

This type of hybrid-by-design architecture delivers both a powerful engine for raw processing power, such as on-prem processing capability, and the agility of cloud for rapid scaling and data access. It aligns disparate technologies toward clearly defined business outcomes through wise design and intentional integration.
That’s why organizations looking to scale their gen AI initiatives can benefit from a consistent hybrid-by-design approach. Today, 53% of organizations are managing their cost of compute governance centrally, with 73% expected to do so by 2026. Hybrid by design is poised to play a critical role in this shift by providing a unified view of the compute resources leaders need to monitor, optimize, and control costs.
What you need to do
Forge a unified front with generative AI and hybrid cloud
Channel the combined power of gen AI and hybrid cloud to deliver on concrete business goals. Optimize and orchestrate with hybrid by design and containerized workloads to corral compute costs and streamline operations.
- Establish your nerve center. Gain visibility into where and how the demand for compute resources increases as you scale generative AI. Find opportunities to orchestrate these resources more cost-effectively, deploying only the precise amount of compute power needed for each task. Extend hybrid cloud architectural principles across your technology estate.
- Contain cost creep. Design for modularity and flexibility. Follow architectural principles that let your organization choose the best and most cost-effective environment for each AI use case and initiative.
- Measure success with clarity. Centralize your cost of compute management and create enterprise-wide guidelines, driven by clearly defined business objectives. Create a governance structure with a defined responsibility assignment matrix and performance metrics.
3. Optimization
+ Generative AI
What you need to know
Generative AI stretches your computing budget
Gen AI may be the cause of the cost of compute crunch—but it can also be the solution. In fact, 73% of executives agree that gen AI can make their use of computing resources more efficient—and they’re already putting this theory into practice.
For example, 67% of organizations are using gen AI to accelerate the development of new and more efficient models, algorithms, and applications. This not only reduces the time and effort required to develop these resources but also help organizations create more efficient solutions that require fewer computing resources.
In addition, 65% of organizations are using gen AI to reduce required compute resources by automating tasks. How is this different from traditional automation? One unique aspect is that gen AI models can be designed to process data in parallel, taking advantage of multiple processing units and reducing the overall processing time and compute resources required to complete an automated task.

Another promising application of gen AI lies in making the mainframe more cost-efficient. While the mainframe has developed a reputation for being costly to manage and difficult to use, there are plenty of scenarios where it’s still the best option. For instance, banks, insurance companies, and airlines rely on mainframes to maintain business continuity in a crisis. Its ability to shift workloads when systems are compromised—and process transactions at unmatched speeds—has given the mainframe real staying power.
And gen AI can take that speed and resilience to the next level. It can optimize system utilization while also transforming mainframe operations through AI-powered automation, predictive analytics, and self-tuning capabilities. Plus, by using gen AI to optimize data center layouts, organizations can reduce energy consumption, lower costs, and improve overall efficiency. In 2023, 25% of organizations were applying gen AI for this purpose, and this figure is expected to rise to 70% by the end of 2024.
What you need to do
Move lightning-fast at a lower cost
Arm managers with intelligent decision support tools that slash compute costs and fuel real-time adaptability. Automate workflows and prune models to unlock a new era of efficiency, reduce costs, and unleash innovation.
- Inject generative AI into the heart of IT operations. Give IT managers gen AI tools that help them create automation scripts, document operations, and spend less time on compliance. Revolutionize mainframe management with automated problem detection and resolution, predictive capacity management, and real-time performance monitoring.
- Optimize and automate your way to efficiency. Tap gen AI for synthetic data generation, automated code generation and optimization, and dynamic resource allocation to reduce cost of compute.
- Adapt to changing market conditions in real time. Use gen AI to analyze real-time demand, market trends, and competitor pricing to optimize pricing strategies, maximize revenue, and reduce price-related losses. Assess historical spending patterns to predict budget requirements, optimize budget allocation, and reduce waste.
The statistics informing the insights in this report are sourced from two proprietary surveys conducted by the IBM Institute for Business Value and Oxford Economics. The first surveyed 207 US-based executives regarding their perspectives on cost of compute and generative AI in June and July 2024. The second surveyed 1,110 executives regarding sustainable IT practices from December 2023 through April 2024. Insights from The Wall Street Journal are also referenced.
Bookmark this report
Share
Additional content
Meet the authors
Anthony Marshall, Senior Research Director, Thought Leadership, IBM Institute for Business ValueCindy Anderson, Global Executive for Engagement and Eminence, IBM Institute for Business Value
Christian Bieck, Europe Leader & Global Research Leader, Insurance, IBM Institute for Business Value
Jacob Dencik, Ph.D, Research Director, IBM Institute for Business Value
Download report translations
Originally published 07 October 2024
How can IBM help you?
You might also like
You might also like