Simple Slurm

A simple Snakemake profile for Slurm without --cluster-config

• Features
• Limitations
• Quick start
• Customizations
• Use speed with caution
• License

The option --cluster-config is deprecated, but it's still possible to set default and rule-specific resources for submitting jobs to a remote scheduler using a combination of --default-resources and the resources field in individual rules. This profile is a simplified alternative to the more comprehensive official Slurm profile for Snakemake. For more background, this blog post by Sichong Peng nicely explains this strategy for replacing --cluster-config.

[!WARNING] The Slurm profile and documentation in this repository currently support Snakemake versions >= 5.15.0 and < 8.0.0. This is because Snakemake 8 completely overhauled how it submits jobs to external clusters, which broke this and all the other existing profiles. We are in the process of updating the documentation to support Snakemake 8 (please help contribute if you can). If you plan to continue to use Snakemake 7, you can find the Snakemake 7 version of the docs in the v7 branch of this repository.

Features

• Only requires a single configuration file to get started submitting jobs to Slurm

• Easily add or remove options to pass to sbatch

• Automatically saves the log files as logs/{rule}/{rule}-{wildcards}-%j.out, where {rule} is the name of the rule, {wildcards} is any wildcards passed to the rule, and %j is the job number

• Automatically names jobs using the pattern smk-{rule}-{wildcards}

• Fast! It can quickly submit jobs and check their status because it doesn't invoke a Python script for these steps, which adds up when you have thousands of jobs (however, please see the section Use speed with caution)

• No reliance on the deprecated option --cluster-config to customize job resources

• By default it relies on Snakemake's built-in ability to understand the job statuses PENDING, RUNNING, COMPLETING, and OUT_OF_MEMORY

• (Optional, but recommended) You can pass a simple script (see extras/) to --cluster-status to additionally handle the job statuses TIMEOUT and CANCELED

• New Support for cluster-cancel feature introduced in Snakemake 7.0.0 (see examples/cluster-cancel/)

• New Full support for multi-cluster setups (using a custom status script requires Snakemake 7.1.1+). See the section Multiple clusters below

• New Adaptable for use with AWS ParallelCluster. See Christian Brueffer's profile snakemake-aws-parallelcluster-slurm

Limitations

• If you use job grouping, then you can't dynamically name the jobs and log files based on the name of the rules. This doesn't prevent you from using this profile and benefiting from its other features, but it is less convenient. Also note that job grouping isn't easy to use in the first place, since it sums resources like mem_mb and threads, but that is a limitation of Snakemake itself, and not anything in particular with this profile UPDATE: As of Snakemake 7.11, there is improved support for managing the maximum resources requested when submitting a grouped job that executes multiple rules. It's still non-trivial, but now at least possible. See the example in examples/job-grouping/ for a demonstration of how to use the new features

• Wildcards can't contain / if you want to use them in the name of the Slurm log file. This is a Slurm requirement (which makes sense, since it has to create a file on the filesystem). You'll either have to change how you manage the wildcards or remove the {wildcards} from the pattern passed to --output, e.g. --output=logs/{rule}/{rule}-%j.out. Note that you can still submit wildcards containing / to --job-name

• Requires Snakemake version 5.15.0 or later (released 2020-04-21, see changelog). You can test this directly in your Snakefile with min_version()

Quick start

1. Download the configuration file config.yaml to your Snakemake project. It has to be in a subdirectory, e.g. simple/

2. Open it in your favorite text editor and replace all the placeholders surrounded in angle brackets (<>) with the options you use to submit jobs on your cluster

3. You can override any of the defaults by adding a resources field to a rule, e.g.

   rule much_memory:
       resources:
           mem_mb=64000
   

4. Invoke snakemake with the profile:

   snakemake --profile simple/
   

Customizations

See the directory examples/ for examples you can experiment with on your cluster.

A fixed argument to sbatch, e.g. --account

To pass an additional argument to sbatch that will be fixed across all job submissions, add it directly to the arguments passed to sbatch in the field cluster. For example, to specify an account to use for all job submissions, you can add the --account argument as shown below:

cluster:
  mkdir -p logs/{rule} &&
  sbatch
    --partition={resources.partition}
    --qos={resources.qos}
    --cpus-per-task={threads}
    --mem={resources.mem_mb}
    --job-name=smk-{rule}-{wildcards}
    --output=logs/{rule}/{rule}-{wildcards}-%j.out
    --account=myaccount

A variable argument to sbatch, e.g. --time

To pass an additional argument to sbatch that can vary across job submissions, add it to the arguments passed to sbatch in the field cluster, list a default value in the field default-resources, and update any rules that require a value different from the default.

For example, the config.yaml below sets a default time of 1 hour, and the example rule overrides this default for a total of 3 hours. Note that the quotes around the default time specification are required, even though you don't need quotes when specifying the default for either partition or qos.

cluster:
  mkdir -p logs/{rule} &&
  sbatch
    --partition={resources.partition}
    --qos={resources.qos}
    --cpus-per-task={threads}
    --mem={resources.mem_mb}
    --job-name=smk-{rule}-{wildcards}
    --output=logs/{rule}/{rule}-{wildcards}-%j.out
    --time={resources.time}
default-resources:
  - partition=<name-of-default-partition>
  - qos=<name-of-quality-of-service>
  - mem_mb=1000
  - time="01:00:00"

# A rule in Snakefile
rule more_time:
    resources:
        time = "03:00:00"

Note that sbatch accepts time defined using various formats. Above I used hours:minutes:seconds, but the simple slurm profile is agnostic to how you choose to configure this. It's a good idea to be consistent across rules, but it's not required. From Slurm 19.05.7:

A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds".

Thus to instead use minutes, you could achieve the same effect as above with:

cluster:
  mkdir -p logs/{rule} &&
  sbatch
    --partition={resources.partition}
    --qos={resources.qos}
    --cpus-per-task={threads}
    --mem={resources.mem_mb}
    --job-name=smk-{rule}-{wildcards}
    --output=logs/{rule}/{rule}-{wildcards}-%j.out
    --time={resources.time}
default-resources:
  - partition=<name-of-default-partition>
  - qos=<name-of-quality-of-service>
  - mem_mb=1000
  - time=60

# A rule in Snakefile
rule more_time:
    resources:
        time = 180

See examples/time-integer/ and examples/time-string/ for examples you can play with. Note that specifying the time as a string requires a minimum Snakemake version of 5.15.0.

Using a cluster status script

By default, snakemake can monitor jobs submitted to slurm. I realized this when reading this detailed blog post, in which the author decided not to use the cluster-status.py script provided by the official Slurm profile. Thus if you don't find that your jobs are silently failing often, then there's no need to worry about this extra configuration step.

However, if you start to have jobs silently fail often, e.g. with status TIMEOUT for exceeding their time limit, then you can add a custom script to monitor the job status with the option --cluster-status.

The directory extras/ contains multiple options for checking the status of the jobs. You can choose which one you'd like to use:

• status-sacct.py - This is the example from the Snakemake documentation. It uses sacct to query the status of each job by its ID

• status-sacct.sh - (recommended) This is a Bash translation of the example from the Snakemake documentation. The Python script is simply shell-ing out to sacct, so running Bash directly removes the overhead of repeatedly starting Python each time you check a job

• status-scontrol.sh - This is a Bash script that uses scontrol to query the status of each job by its ID. The scontrol command is from slurm-status.py in the official profile. If your HPC cluster doesn't have sacct configured, you can use this option

• status-sacct-multi.sh - Support for multi-cluster setup (see section Multiple clusters)

To use one of these status scripts:

1. Download the script to your profile directory where config.yaml is located

2. Make the script executable, e.g. chmod +x status-sacct.sh

3. Add the field cluster-status to your config.yaml, e.g. cluster-status: status-sacct.sh

4. Add the flag --parsable to your sbatch command (requires Slurm version 14.03.0rc1 or later)

Multiple clusters

It's possible for Slurm to submit jobs to multiple different clusters. Below is my advice on how to configure this. However, I've worked with multiple HPC clusters running Slurm, and have never encountered this situation. Thus I'd appreciate any contributions to improve the documentation below.

1. If you have access to multiple clusters, but only need to submit jobs to the default cluster, then you shouldn't have to modify anything in this profile

2. If you want to always submit your jobs to a cluster other than the default, or use multiple clusters, then pass the option --clusters to sbatch, e.g. to submit your jobs to either cluster "c1" or "c2"

   cluster:
     mkdir -p logs/{rule} &&
     sbatch
       --clusters=c1,c2
   

3. To set a default cluster and override it for specific rules, use --default-resources. For example, to run on "c1" by default but "c2" for a specific rule:

   # config.yaml
   cluster:
     mkdir -p logs/{rule} &&
     sbatch
       --clusters={resources.clusters}
   default-resources:
     - clusters=c1
   

   # Snakefile
   rule different_cluster:
       resources:
           clusters="c2"
   

4. Using a custom cluster status script in a multi-cluster setup requires Snakemake 7.1.1+. After you add the flag --parsable to sbatch, it will return jobid;cluster_name. I adapted status-sacct.sh to handle this situation. Please see examples/multi-cluster/ to try out status-sacct-multi.sh

Use speed with caution

A big benefit of the simplicity of this profile is the speed in which jobs can be submitted and their statuses checked. The official Slurm profile for Snakemake provides a lot of extra fine-grained control, but this is all defined in Python scripts, which then have to be invoked for each job submission and status check. I needed this speed for a pipeline that had an aggregation rule that needed to be run tens of thousands of times, and the run time for each job was under 10 seconds. In this situation, the job submission rate and status check rate were huge bottlenecks.

However, you should use this speed with caution! On a shared HPC cluster, many users are making requests to the Slurm scheduler. If too many requests are made at once, the performance will suffer for all users. If the rules in your Snakemake pipeline take at least more than a few minutes to complete, then it's overkill to constantly check the status of multiple jobs in a single second. In other words, only increase max-jobs-per-second and/or max-status-checks-per-second if either the submission rate or status checks to confirm job completion are clear bottlenecks.

License

This is all boiler plate code. Please feel free to use it for whatever purpose you like. No need to attribute or cite this repo, but of course it comes with no warranties. To make it official, it's released under the CC0 license. See LICENSE for details.