Scheduling AI jobs across five data centres and an external cloud
Which AI job runs when, in which data centre and on which GPU generation, and what is rented externally, so that important jobs finish on time, the own GPUs stay busy through the day and little is rented?
Drag to rotate, right-drag or shift-drag to move the map, scroll or pinch to zoom. The top of each rack shows its GPU generation. Its two light panels (32 GPUs each) show the job running there: blue online inference, purple training, amber fine-tuning, teal batch inference. Each data centre chip shows how many of its GPUs are busy. Light green panels are experiments. The top row of the schedule shows the GPUs in use over the day: green own, red rented on demand, yellow rented as spot, the dashed line is the own capacity. Click a rack to find its job in the schedule; hover a bar for details. Plan computed with a Timefold model.
The planning question
This example is a fictional composition. The data centres, fleets, prices, jobs and performance figures are invented. The GPU model names (NVIDIA A100, H100, H200, B200, AMD MI250X, MI300X) only name the hardware type, with public, rounded specs. There is no affiliation with any vendor, operator or cloud provider.
Five data centres on an abstract map run 9,728 GPUs. DC North has 1,536 NVIDIA H100 and 1,024 A100, DC West 1,024 H200 and 1,024 H100, DC Centre 1,024 B200 and 512 H200, DC South 1,024 AMD MI300X and 512 MI250X, DC East 1,536 A100 and 512 MI250X. An external cloud rents NVIDIA H100 with 80 GB in two ways: on demand for €3.00 per GPU hour, always available, or as spot for €1.20 per GPU hour. Spot capacity may be taken back at short notice, so only jobs with checkpoints use it, and a surcharge of 15 % covers the reruns.
On a Tuesday five teams send 480 jobs, about 117 % of what the own GPUs can do in a day. Online inference is the base load: per region and 3-hour block a fixed throughput, with the peak in the evening. 16 training runs of 256 to 1,024 GPUs have a target for today; what is not done by 24:00 continues tomorrow. 52 fine-tuning jobs arrive between 05:00 and 15:00, most of them from 06:30 to 09:00, each due 3 to 6 hours later. 150 batch inference jobs are flexible; 39 of them must be ready by 07:00. 230 experiments (research runs, evaluations, sweeps) are best effort: they run on own GPUs only, and what does not fit waits for tomorrow. A job runs on all its GPUs in one place at the same time. Some need 141 GB per GPU, some run on CUDA only.
How we plan it
The Timefold model plans each job with four variables: the GPU pool (data centre and generation, the cloud on demand or as spot, or for experiments: tomorrow), the start, and for checkpointable jobs the position and length of one pause. A job finishes its work faster on a newer generation (assumed throughput relative to an A100: H100 2.0, H200 2.4, B200 4.0, MI250X 0.9, MI300X 2.2). Online inference needs a throughput, so a faster generation needs fewer GPUs. Copying the data to another data centre takes 15 minutes, to the cloud 30 minutes; a resume after a pause takes 15 minutes. Hard rules are the GPUs per pool in every quarter hour. Late important jobs (fine-tuning, batch due 07:00) come next. Then the plan weighs money: the external rent against the value of the work not done today, assumed at €2.00 per A100 hour for training behind its target and €1.00 for a waiting experiment. Electricity is left out on purpose, because it hardly changes the decision compared with rent. Timefold plans in two steps: first the jobs that must run, then the experiments fill the gaps, and everything may still move.
The comparison is a common way of working, written as a fixed rule and computed with the same data. Every team uses its home data centre. Jobs are served in the order they arrive, in the first free slot of the first pool with enough memory, experiments last. When a job would wait more than an hour or miss its deadline, it is rented: as spot if it has checkpoints, else on demand. The Timefold plan starts from this plan and improves it for 2 × 2 minutes. The result is the best plan found in that time. Whether a better one exists is not proven.
What the plan shows
| KPI | Optimized plan | Comparison: fixed team quotas |
|---|---|---|
| important jobs on time | 100 % +9.9 | 90.1 % |
| own GPUs busy | 99.1 % +10.5 | 88.6 % |
| rent on demand | 0 € −14,820 | 14,820 € |
| rent spot | 37,492 € −26,352 | 63,844 € |
| experiments done today | 71.2 % −0.4 | 71.6 % |
Decisions
- data centre and GPU generation, or the external cloud (on demand or spot), for every job
- start of every job in 15-minute steps
- a pause for checkpointable jobs (training, batch inference): when, and for how long
Constraints
- enough GPUs with enough memory in one place at the same time
- deadlines and priorities: online inference first, then deadlines, then the rest
- spot only for jobs with checkpoints, experiments only on own GPUs
Objective
important jobs on time, then little external rent and much work done today: own GPUs full, spot before on demand
Questions
Are the data centres, prices and figures real?
No. Everything is a fictional composition: the data centres, fleets, rents, jobs, teams, values and performance figures. The vendor names NVIDIA and AMD only name the GPU type; memory per GPU is the public, rounded spec, throughput is an assumption. There is no affiliation with any vendor, operator or cloud provider.
Why is there more work than capacity?
Because that is the normal case. In a GPU cluster the queue is hardly ever empty: experiments, evaluations and sweeps wait for free GPUs. The plan decides what runs today and what waits. This keeps the own GPUs busy until midnight instead of running dry in the evening.
What does the comparison rent?
81 jobs for €78,664. 23 fine-tuning jobs have no checkpoints and go on demand (€14,820); 4 training runs and 54 batch jobs go to spot (€63,844). Nine important jobs finish late. In the evening the own GPUs fall to 60 % at 22:00, because experiments only start in their home data centre.
What does the plan rent?
2 training runs and 27 batch jobs, all as spot, for €37,492, and nothing on demand. The fine-tuning jobs stay on the fast own GPUs, five checkpointable jobs pause to make room, and all 91 important jobs finish on time. Renting batch on spot costs less than the experiments it makes room for are worth.
What happens over the day?
The top row of the schedule shows it. The own GPUs run at 94 to 100 % all day, 99 % on average, compared with 89 % in the comparison. The experiments fill every gap: 71 % of their work is done today, as much as in the comparison, and 38 of 230 wait for tomorrow (comparison: 49). Two training runs continue a few minutes after midnight.
Can it plan again during the day?
Yes. Jobs that have started stay fixed. A new urgent job, a failed rack or a cheaper rent offer are planned in with the same rules.
More use cases
Your planning problem looks different?
Every operation has its own rules. Tell us about yours. We assess feasibility and expected impact in a free initial call.
Book a free call Write to us