Submit and monitor SLURM jobs

SLURM schedules access to blazar's shared computational resources: GPU, CPU and RAM. You submit a job describing your usage needs. SLURM puts the job in a queue based on the resources requested and your usage history.

Available partitions¶

A partition is a queue with its own limits. Choose a partition with --partition when you submit a job.

If you don't set --partition, SLURM uses the default partition shown above.

A higher scheduling priority runs first when jobs compete. See How SLURM ranks your job in the queue.

Submit a batch job¶

1. Write a job script. At the top, set #SBATCH directives for the resources that your job needs:

#!/bin/bash
#SBATCH --job-name=my-training-run
#SBATCH --partition=batch
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=8
#SBATCH --mem=32G
#SBATCH --time=01:00:00
#SBATCH --output=%x-%j.log

python train.py

1. Submit it:

sbatch job.sh

SLURM prints the job ID that it assigned. For example: Submitted batch job 42.

Request the GPU¶

Add --gres=gpu:1 to request the GPU. If you do not add it, your job still runs, but it cannot use the GPU.

Request CPU cores¶

Add --cpus-per-task=<N> to request N CPU cores (up to 8 cores). If you do not set it, SLURM gives your job one core. Request more cores only when your job uses several cores or threads, for example for parallel data loading.

Request memory¶

Add --mem=<size> to request memory for the whole job, for example --mem=32G. If you do not set it, SLURM uses the partition's default memory limit.

Request a time limit¶

Add --time=<D-HH:MM:SS> to set the maximum run time, for example --time=01:00:00 for one hour. If you do not set it, SLURM uses the partition's maximum run time (see Available partitions).

Run an interactive job¶

To get a shell with resources reserved for you, request a session instead of a batch script:

srun --partition=debug --gres=gpu:1 --pty bash

The command waits until SLURM grants the resources. Then it opens a shell on the allocated node.

Monitor and manage jobs¶

How SLURM ranks your job in the queue¶

The server has one GPU. When more jobs wait than the GPU can run, SLURM decides the order in two steps.

1. Partition tier. SLURM considers every job in a higher-priority partition before any job in a lower-priority partition. A pending debug job is therefore dispatched ahead of a pending batch job. See the partition table for each partition's scheduling priority.
2. Priority score. Among the jobs in the same partition tier, SLURM ranks each job by a priority score. The score is a weighted sum of a few factors:

Fair-share weighs about ten times more than any other factor, so it usually decides the order.

Fair-share: your share against your recent usage¶

Fair-share compares how much your account used recently against the share it is entitled to.

• Equal shares. Every top-level account gets the same fair-share weight (1). An account is one standalone user, or one family — a sponsor plus the people that sponsor invited. So each account is entitled to an equal slice of the resources over time, whatever its number of members. See Request an account.
• Usage lowers your factor. When your account uses less than its share, its fair-share factor rises toward 1 and your jobs gain priority. When your account uses more, the factor falls toward 0 and your jobs lose priority.
• Recent usage counts most. Old usage fades. A busy week fades over about the following week. So your priority recovers when you pause.
• A family shares one bucket. Every member of a family draws down the same share. When several members run jobs at the same time, the whole family's fair-share factor falls.

Run sshare to see your account's current fair-share factor and recent usage.

What counts as usage¶

SLURM does not measure usage as plain time. It converts the resources that your job holds into billing units, then multiplies the units by how long the job runs. The GPU, the CPU cores, and the memory all count.

The rates are balanced. A whole GPU, all the CPU cores, or all the RAM each cost about the same. Two results follow:

• A longer job costs more. A job that runs for two hours costs twice as much as the same job for one hour.
• Idle resources still cost. When you request CPU cores or memory that your job does not use, they still add to your usage and lower your fair-share factor.

How your requests change your place¶

Three parts of your job request change where you land in the queue.

--partition (the queue)

Each partition has a scheduling priority. A job in a higher-priority partition runs before a job in a lower-priority partition. The debug partition has the highest priority, but the shortest time limit.

--time (the time limit)

A short, accurate time limit lets SLURM fit your job into a gap before a larger, higher-priority job starts. This is called backfill. A time limit that is too long makes you wait for that gap to grow. A time limit that is too short stops your job before it finishes.

--gres, --cpus-per-task, --mem (the resources)

The server has one GPU, so most jobs wait for the same GPU. When you also ask for more CPU cores or memory than are free, you wait for those too. Every resource that you hold also counts as usage and lowers your fair-share factor — see What counts as usage. Ask only for what you need.