AI factory strategy: How to decide where AI workloads should run

Many of our client organizations start their AI journey in the cloud, running proofs of concept (POCs), prototypes, and pilots, and prioritizing their use cases. At the outset, it is a sound decision. It’s straightforward, and resources are available, elastic, and performant enough. In general, the low risk and cost of entry allow them to achieve a fast time to value. But once those pilots move into production and multiply, data gravity and scale may make the cloud cost-prohibitive. This is often the point where it makes sense to evaluate an on-premises AI factory.

On the other hand, I’ve also seen clients beginning in the opposite direction, purchasing expensive on-prem AI infrastructure before truly understanding the business outcomes they’re trying to achieve. While on-prem AI factories are becoming an important operating model at scale, any strategy should start with defining the desired business outcomes and preparing the organization before deployment. Starting there is often the key to understanding the workloads you need to run and building a mix of environments that suit your overall goals. This also generally lowers cost and risk.

Why cloud vs. on premises shouldn’t be your first question

When IT leaders decide where to run an AI workload, many factors come into play. For starters, you’re dealing with copious amounts of potentially unorganized data, governance and security requirements, sometimes sovereignty needs, and total cost of ownership (TCO) considerations. I believe most leaders in the space are looking for the highest rate of return with the lowest cost of entry that still meets the needs of the organization. For pilots, testing, prototyping, and sandboxes, the low cost of entry is why many start their projects in the cloud.

Before you weigh cloud vs. on premises, I always recommend starting your AI factory strategy with the business outcomes you are trying to achieve. Without a clear outcome, that killer use case that generates revenue, reduces cost, or improves organizational efficiency is not yet worth an AI factory model investment. Most enterprises also haven’t reached AI-factory scale, but they’re quickly figuring out where AI can make an impact in their organizations.

I’ve seen the mistakes play out firsthand. For example, a retail organization invested over $1 million in an AI pod infrastructure, thinking it would “grow into it.” The reality, however, was that the business, data science team, developers, and IT department weren’t actually ready and didn’t have any meaningful use cases to take advantage of the infrastructure. Within a year or so, the cost to run it, without any return on investment, had the organization looking to sell the assets. That’s because, from the start, the company put the cart before the horse and didn’t align business outcomes with the infrastructure needed to support the result.

With the high cost of resources required to deploy true enterprise-scale AI infrastructure, this type of mentality is not ideal, nor is it easy on the pocketbook. It’s one of the biggest mistakes we are seeing enterprises make right now.

I am a proponent of the “don’t boil the ocean” strategy. Find a few use cases with specific and real business outcomes. Pilot and learn from them. Then scale toward an AI factory, where multiple AI use cases and workloads create intelligence in a meaningful and cost-effective way.

What factors should drive AI workload placement?

Once you understand your desired business outcome, these are the factors I’d weigh most closely:

1. Data. Start with your data, including where it lives, how it is governed, and how it is organized. AI performs best when relevant, accurate, and well-governed data is available for the workload. This is the familiar garbage-in, garbage-out principle. Disjointed, outdated, irrelevant, or inaccurate data can produce poor results and make it harder to evaluate or trust model outputs.
   
2. Security. Security is important in any IT environment, but AI can increase the speed, scale, and impact of security and privacy risks. Organizations should establish policies, procedures, and technical controls before deploying AI. Preparation should address risks such as data sovereignty and regulatory requirements, isolation and access controls, model, prompt, and output protection, supply chain vulnerabilities, and inadequate monitoring.
   
3. Latency. The first question is not simply, “How fast is the GPU?” It is, “How quickly does the business need an answer?” A customer-facing assistant, fraud detection system, industrial control process, or computer vision application may require responses in milliseconds or seconds. For those workloads, network distance, data access, queueing, model loading, and response generation all matter. Running the workload closer to the user, device, or source data may provide more predictable performance.
   
4. Capacity and availability. In my 20 years in IT, I’ve never experienced availability and time constraints like we’re seeing now, and an on-premises AI factory can take six to 12 months to deploy, depending on scale, procurement, and site readiness. That catches many folks by surprise. So, while you wait for the equipment, you can work on data and governance or start in the cloud. In many cases, we point customers to neoclouds, which can be a more cost-effective way to get GPU resources and bridge the gap until your equipment arrives.
   
5. Facilities. Power and cooling are among the biggest limitations moving forward. With the Vera Rubin generation, NVIDIA’s highest-density rack-scale AI systems are designed around direct liquid cooling. Retrofitting an air-cooled facility for that type of deployment can require significant changes to power distribution, cooling loops, rack design, and facility infrastructure. In some cases, a new or adjacent AI facility, colocation site, or neocloud may be more practical than modifying an existing data center.
   
6. Cost. This is the big one. Look past the initial sticker price to full lifecycle economics and TCO, including GPU utilization, storage, network, and data transfer. Many teams don’t ask whether they have enough workloads to use the GPU scale they’re buying, and idle GPUs tie up expensive capital.

Portability and time to value are also worth weighing. Starting in the cloud can deliver value while you wait for on-premises equipment, but ask up front whether you can move that workload on premises, to a neocloud, or to hybrid AI infrastructure later, and what the cost implications will be.

How on premises, public cloud, neocloud, and hybrid models compare

The reality is that no single environment will fit every workload. Here’s how I like to think about each one:

• On premises. When demand is sustained and utilization is high, owning infrastructure may improve TCO and make costs more predictable. The trade-offs are long lead times, operational requirements, power, networking, and increasingly, high-density liquid cooling.
  
• Public cloud. Cloud platforms provide rapid access to compute and services, making them useful for experimentation, pilots, bursty workloads, and teams that want to avoid upfront capital investment. You’ll also still need to evaluate GPU availability, quotas, data movement, egress, and ongoing consumption costs.
  
• Neocloud. Neoclouds specialize in GPU infrastructure and may provide faster access or lower unit costs than hyperscalers for certain workloads. They can serve as a bridge while on-premises capacity is being built, or as a longer-term option for GPU-intensive workloads. However, data protection depends on the provider’s controls, data location, contractual terms, and workload architecture.  A neocloud can also give you access to the latest infrastructure while keeping your data more protected.
  
• Hybrid. A hybrid model provides flexibility. An organization running on premises can burst to the cloud or a neocloud when GPU or storage capacity is constrained. A team that starts in the cloud can later add on-premises capacity, colocation, or neocloud resources when utilization and economics justify the investment.

Deciding where a workload belongs really comes down to asking the right questions, and a scorecard is a good place to start.

A practical scorecard for AI workload placement

As you consider each workload, I’d suggest scoring these questions:

• Where does the data live, and is it organized, governed, ready, and accessible?
• What security, compliance, and data sovereignty requirements apply?
• What latency, throughput, and availability does the workload require?
• What is the support and operations model, and how much control do you need over the infrastructure?
• Do you have adequate redundancy, portability, and options to move the workload if requirements change?
• What will it cost over time in the cloud, on premises, through a neocloud, or in a hybrid environment?
• Is the workload steady, bursty, or seasonal?
• Do you have enough sustained demand to keep the GPU capacity well-utilized?
• What should run where, and what is the interim plan while additional capacity comes online?

I’d weigh data readiness and acceptable latency most, and cost is a big one as well. For cost, I’d look beyond that upfront price to the next three to five years. This infrastructure doesn’t follow a traditional refresh cycle, and new GPUs often come out within a year. So, ask yourself whether this will last three to five years, or whether you’ll need to invest again next year for performance or cooling.

Where does an AI factory operating model fit?

With an AI factory, you take data and energy and turn them into intelligence. And while an AI factory can run on premises, it doesn’t necessarily have to. It’s about having a repeatable process to turn AI use cases into value for your organization. That process can follow your workloads wherever they run, whether that’s on premises, in the cloud, in a neocloud, or in a hybrid mix.

At GDT, we’re helping our customers formalize their teams, create repeatable architectures, and move from AI pilots to NVIDIA-aligned AI factories. Our technology partners provide the accelerated compute, software, networking, and storage. And we help our clients architect, integrate, deploy, and operationalize AI factory solutions.

I believe that AI factories are really the future of the AI space. But your AI factory strategy has to begin with a full understanding of your business outcomes.

Are you considering where your AI workloads should run? Talk with a GDT expert or explore our AI infrastructure workshop.