AI jobs across five data centres: Timefold plans ahead, the scheduler carries it out
Which AI job runs when, where and on which GPUs, and what is rented, so that important jobs finish on time, the own GPUs stay busy and little is rented, while new jobs arrive all day and nodes fail?
Drag to rotate, right-drag or shift-drag to move the map, scroll or pinch to zoom. Each rack holds 64 GPUs; its eight lights show the jobs on its nodes: blue inference, purple training, amber fine-tuning, teal batch, green experiments, red a node that is down. On top of the schedule: the GPUs in use (green own, red on demand, yellow spot, dashed the own capacity), the queue, and the events (purple a Timefold re-plan, red a node failure). Switch to the scheduler alone to compare; hover a bar for details. Plan computed with a Timefold model.
The planning question
Companies and research centres run GPUs in several data centres and rent extra capacity in the cloud. Their scheduler (Slurm or Kubernetes) decides within seconds which job starts on free GPUs. As commonly set up, it plans little ahead and rarely across sites and prices: large jobs wait for gaps, every team stays in its own data centre, and when the queue gets long, capacity is rented.
Here Timefold plans in front of the scheduler. Every 30 minutes it looks 24 hours ahead and decides for every known job where and when it runs, which running jobs pause briefly, and what is rented. The scheduler carries the plan out. Over five simulated days, 98 % of the important jobs finish on time instead of 95 %, the own GPUs are 86 % busy instead of 82 %, and the rent falls from about €7,000 to about €300 a day. On this fleet that is the work of about 380 GPUs more and some €2 million less rent a year, without new hardware.
The example is a fictional composition: data centres, fleets, prices, jobs and performance figures are invented, the GPU names (NVIDIA A100 to B200, AMD MI250X, MI300X) only name the hardware type, and there is no affiliation with any vendor or provider. Five data centres run 9,728 GPUs in racks of 64. On a Tuesday the teams send 863 jobs: reserved inference, ten long training runs, 56 fine-tuning jobs with deadlines a few hours after submission, 130 batch jobs and 600 experiments, more than fits. The day is simulated: jobs become known when they are submitted, run shorter than estimated, nodes fail, and the cloud has limited capacity at short notice.
How we plan it
Timefold plans from the current state: running jobs keep their racks, failed nodes are blocked, inference stays reserved. GPUs per rack are a hard rule; deadlines come next; then euros: rent, the value of work not done (assumed at €2 per A100 hour for training, €1 for experiments), moving data, and a small cost for every change against the last plan, so the plan stays calm. It copies the data of a job ahead of its start and plans with a cautious forecast of the run times. The plan goes to the scheduler as reservations, start times and pauses.
The comparison is the scheduler alone, set up as is common: every team in its home data centre, priority then order of arrival, backfill, no pauses; a job that waited an hour or would miss its deadline is rented. Both run on the same simulated days with the same jobs, run times and failures. Timefold gets 12 seconds per re-plan; each plan is the best found in that time, without a proof that no better one exists.
What the plan shows
| KPI | Optimized plan | Comparison: scheduler with team quotas |
|---|---|---|
| important jobs on time | 97.8 % +1.2 | 96.6 % |
| own GPUs busy | 84.5 % +3 | 81.5 % |
| rent on demand | 0 € −1,000 | 1,000 € |
| rent spot | 437 € −6,267 | 6,704 € |
| experiments done | 526 +2 | 524 |
Decisions
- place of every job: data centre, GPU generation and racks, or the cloud (on demand or spot)
- start of every job, and short pauses of running jobs with checkpoints
- every 30 minutes anew for the next 24 hours, handed to the scheduler
Constraints
- all GPUs of a job at once, close together, with enough memory
- deadlines and priorities: inference reserved, then deadlines, then the rest
- spot only for jobs with checkpoints, experiments only on own GPUs
Objective
important jobs on time, then little rent and much work done on the own GPUs
Questions
Are the data centres, prices and figures real?
No. Data centres, fleets, rents, jobs, teams and performance figures are a fictional composition. NVIDIA and AMD only name the GPU type. There is no affiliation with any vendor, operator or cloud provider.
Why does the scheduler stay?
It runs in production, reacts within seconds to every job that ends and every failed node, and knows the details of the machines. Replacing it would be a risk without benefit. What it lacks is the look ahead and the costs across sites. Timefold adds exactly that, and without the plan everything runs as before.
What is the difference between the scheduler and the planning layer?
The scheduler asks: which waiting job fits on the GPUs that are free now? It follows fixed rules: priority, quotas, first free place, one cluster. The planning layer asks: what is the best plan for all known jobs over the next 24 hours? It weighs deadlines, rent and the value of the work across all data centres and the cloud, decides pauses, and copies data ahead. On this day no fine-tuning job goes to the cloud, and fewer important jobs are late.
How does Timefold get into the architecture?
Every 30 minutes it reads the state from the scheduler: queue, running jobs, failed nodes. It plans and writes the result back: in Slurm as reservations, start times and priorities, in Kubernetes as queue quotas and the target cluster of a job. A pause is a checkpoint signal. It starts in shadow mode next to the scheduler, then gives advice that a person confirms, then steers.
Could Timefold be the scheduler itself?
For smaller clusters or batch work where a decision within a minute is enough: yes, Timefold can re-plan in seconds after every event. For large clusters the split works better: the scheduler handles the second-by-second work on the machines, Timefold the plan.
Why does the comparison rent only €1,000 on demand?
It rents spot whenever a job writes checkpoints: €6,704 on this day. Only fine-tuning, which writes no checkpoints, goes on demand, and these jobs are small. The plan rents almost nothing, because it places fine-tuning on free GPUs in other data centres, with the data copied ahead.
Why Timefold?
The rules are many and differ from company to company: topology, memory, deadlines, quotas, prices, maintenance. Timefold describes them as rules with weights, finds good plans within seconds, explains a plan by its score, and runs in the own data centre.
What would a pilot look like?
A few weeks of your job history calibrate the simulation. Then the planning runs in shadow mode next to your scheduler, and both are compared on the same days before it advises or steers.
More use cases
Your planning problem looks different?
Every operation has its own rules. Tell us about yours. We assess feasibility and expected impact in a free initial call.
Book a free call Write to us