All use of Scientific Computing and Data resources in publications and presentations must acknowledge Mount Sinai’s CTSA funding award. Click here

Scientific Computing and Data / High Performance Computing / Documentation / Load Sharing Facility (LSF) Job Scheduler

Load Sharing Facility (LSF) Job Scheduler

This guide covers job submission on both Minerva and Minerva-Res. The two systems share the same login process, the same bsub command and options, and the same sample job scripts below. The one difference is queue availability: Minerva-Res offers only the express and premium queues, while Minerva offers the full set listed in the queue table.

When you log into Minerva (or Minerva-Res), you are placed on one of the login nodes. Login nodes should only be used for basic tasks such as file editing, code compilation, data backup, and job submission. The login nodes should not be used to run production jobs. Production work should be performed on the system’s compute nodes. Note that compute nodes do not have internet access by design for security reasons. You need to be on a login node or one of the interactive nodes to talk to the outside world.

Running Jobs on Compute Nodes

Access to compute resources and job scheduling are managed by IBM Spectrum LSF (Load Sharing Facility) batch system. LSF is responsible for allocating resources to users, providing a framework for starting, executing and monitoring work on allocated resources, and scheduling work for future execution. Once access to compute resources has been allocated through the batch system, users have the ability to execute jobs on the allocated resources.

You should request compute resources (e.g., number of nodes, number of compute cores per node, amount of memory, max time per job, etc.) that are consistent with the type of application(s) you are running:

  • A serial (non-parallel) application can only make use of a single compute core on a single node, and will only see that node’s memory.
  • A threaded program (e.g. one that uses OpenMP) employs a shared memory programming (SMP) model and is also restricted to a single node, but can run on multiple CPU cores on that same node. If multiple nodes are requested for a threaded application it will only see the memory and compute cores on the first node assigned to the job while those on the rest of the nodes are wasted. It’s highly recommended the number of threads is set to that of compute cores.
  • An MPI (Message Passing Interface) parallel program can be distributed over multiple nodes: it launches multiple copies of its executable (MPI tasks, each assigned unique IDs called ranks) that can communicate with each other across the network.
  • An LSF job array enables large numbers of independent jobs to be distributed over more than a single node from a single batch job submission.

Submitting Batch Jobs with bsub

To submit a job to the LSF queue you must have a project allocation account first and it has to be provided using the -P option flag. To see the list of project allocation accounts you have access to:

$ mybalance

If you need access to a specific project account, you will have to have the project authorizer (owner/delegate) send us a request at hpchelp@hpc.mssm.edu.

A batch job is the most common way users run production applications. To submit a batch job to one of the queues use the bsub command:

bsub [options] command

$ bsub -P acc_hpcstaff -q premium -n 1 -W 00:10 -o hello.out echo "Hello World!"

In this example, a job is submitted to the premium queue (-q premium) to execute the command

echo "Hello World!"

with a resource request of a single core (-n 1) and the default amount of memory (3 GB) for a 10 min. of walltime (-W 00:10) under a specific project allocation account that Minerva staff members have access to (-P acc_hpcstaff). The result will be written to a file (-o hello.out) with a summary of resource usage information when the job is completed. The job will advance in the queue until it has reached the top. At this point, LSF will allocate the requested compute resources to the batch job.

Typically, the user submits a job script to the batch system.

bsub [options] < YourJobScript

Here “YourJobScript” is the name of a text file containing #BSUB directives and shell commands that describe the particulars of the job you are submitting.

$ bsub < HelloWorld.lsf

where HelloWorld.lsf is:

#!/bin/bash
#BSUB -P acc_hpcstaff
#BSUB -q premium
#BSUB -n 1
#BSUB -W 00:10
#BSUB -e "hello.err"
#BSUB -o "hello.out"

echo "Hello World!"

Note that if an option is given on both the bsub command line and in the job script, the command line option overrides the option in the script. The following job will be submitted to the express queue, not to the premium queue as specified in the job script.

$ bsub -q express < HelloWorld.lsf

Commonly Used bsub Options

-P acc_ProjectName Project allocation account name (required)
-q queue_name Job submission queue (default: premium)
-W WallClockTime Wall-clock limit in the form HH:MM (default 1:00)
-J job_name Job name
-n Ncore Total number of CPU cores requested (default: 1)
-R rusage[mem=#] Amount of memory per core in MB (default: rusage[mem=3000]). Note that this is not per job.

  • Max memory per node: ~1.4 TB (Chimera, himem, CATS, GPU H100, L40S), ~325 GB (GPU V100, A100), ~1.9 TB (himem-GPU A100-80GB), ~435 GB (GPU H100-80GB)
-R span[ptile=#n's per node] Number of CPU cores per physical node
-R span[hosts=1] All cores on the same node
-o output_file Direct job standard output to output_file (without -e, error also goes to this file). LSF appends output to the specified file; use -oo to overwrite instead.
-e error_file Direct job error output to error_file. Use -eo to overwrite instead.
-L login_shell Initializes the execution environment using the specified login shell

For more details about bsub options, click here.

Although there are default values for all batch parameters except -P, it is always a good idea to specify the name of the queue, the number of cores, and the walltime for all batch jobs. To minimize time spent waiting in the queue, specify the smallest walltime that will safely allow the job to complete.

LSF inserts job report information into the job’s output. This includes the submitting user and host, the execution host, the CPU time (user plus system time) used by the job, and the exit status. Note that the standard LSF configuration allows emailing job output to the user on completion, but this feature has been disabled for the reason of stability.

Sample Batch Jobs

Serial Jobs

A serial job is one that only requires a single computational core. There is no queue specifically configured to run serial jobs. Serial jobs share nodes, rather than having exclusive access. Multiple jobs will be scheduled on an available node until either all cores are in use, or until there is not enough memory available for additional processes on that node. The following job script requests a single core and 8GB of memory for 2 hours in the “express” queue:

#!/bin/bash
#BSUB -J mySerialJob                   # Job name
#BSUB -P acc_YourAllocationAccount     # allocation account
#BSUB -q express                       # queue
#BSUB -n 1                             # number of compute cores = 1
#BSUB -R rusage[mem=8000]              # 8GB of memory
#BSUB -W 02:00                         # walltime in HH:MM
#BSUB -o %J.stdout                     # output log (%J : JobID)
#BSUB -eo %J.stderr                    # error log
#BSUB -L /bin/bash                     # Initialize the execution environment

ml gcc
cd /sc/arion/work/MyID/my/job/dir/
../mybin/serial_executable < testdata.inp > results.log

Multithreaded Jobs

In general, a multithreaded application uses a single process which then spawns multiple threads of execution. The compute cores must be on the same node and it’s highly recommended the number of threads is set to the number of compute cores. The following example requests 8 cores on the same node for 12 hours in the “premium” queue. Note that the memory requirement (-R rusage[mem=4000]) is in MB and is PER CORE, not per job. A total of 32GB of memory will be allocated for this job.

#!/bin/bash
#BSUB -J mySTARjob                # Job name
#BSUB -P acc_PLK2                 # allocation account
#BSUB -q premium                  # queue
#BSUB -n 8                        # number of compute cores
#BSUB -W 12:00                    # walltime in HH:MM
#BSUB -R rusage[mem=4000]         # 32 GB of memory (4 GB per core)
#BSUB -R span[hosts=1]            # all cores from the same node
#BSUB -o %J.stdout                # output log (%J : JobID)
#BSUB -eo %J.stderr               # error log
#BSUB -L /bin/bash                # Initialize the execution environment

module load star                  # load star module
WRKDIR=/sc/arion/projects/hpcstaff/benchmark_star

STAR --genomeDir $WRKDIR/star-genome --readFilesIn Experiment1.fastq --runThreadN 8 -outFileNamePrefix Experiment1Star

MPI Parallel Jobs

An MPI application launches multiple copies of its executable (MPI tasks) over multiple nodes that are separate but can communicate with each other across the network. The following example requests 48 cores and 2 hours in the “premium” queue. Those 48 cores are distributed over 6 nodes (8 cores per node) and a total of 192GB of memory will be allocated for this job, with 32GB on each node.

#!/bin/bash
#BSUB -J myMPIjob                 # Job name
#BSUB -P acc_hpcstaff             # allocation account
#BSUB -q premium                  # queue
#BSUB -n 48                       # total number of compute cores
#BSUB -R span[ptile=8]            # 8 cores per node
#BSUB -R rusage[mem=4000]         # 192 GB of memory (4 GB per core)
#BSUB -W 02:00                    # walltime in HH:MM
#BSUB -o %J.stdout
#BSUB -eo %J.stderr
#BSUB -L /bin/bash

module load selfsched
mpirun -np 48 selfsched < test.inp

Please refer to the GPU section for further information about options relevant to GPU job submissions.

Array Jobs

Sometimes it is necessary to run a group of jobs that share the same computational requirements but with different input files. Job arrays can be used to handle this type of embarrassingly parallel workload. They can be submitted, controlled, and monitored as a single unit or as individual jobs or groups of jobs. Each job submitted from a job array shares the same job ID as the job array and is uniquely referenced using an array index.

To create a job array, add an index range to the jobname specification:

#BSUB -J MyArrayJob[1-100]

In this way you ask for 100 jobs, numbered from 1 to 100. Each job in the job array can be identified by its index, which is accessible through the environment variable LSB_JOBINDEX. LSF provides the runtime variables %I and %J, which correspond to the job array index and the jobID respectively. They can be used in the #BSUB option specification to diversify the jobs. In your commands, however, you must use the environment variables LSB_JOBINDEX and LSB_JOBID.

#!/bin/bash
#BSUB -P acc_hpcstaff
#BSUB -n 1
#BSUB -W 02:00
#BSUB -q express
#BSUB -J "jobarraytest[1-10]"
#BSUB -o out.%J.%I
#BSUB -e err.%J.%I

echo "Working on file.$LSB_JOBINDEX"

A total of 10 jobs, jobarraytest[1] to jobarraytest[10], will be created and submitted to the queue simultaneously by this script. The output and error files will be written to out.JobID.1 ~ out.JobID.10 and err.JobID.1 ~ err.JobID.10, respectively.

Interactive Jobs

Interactive batch jobs give users interactive access to compute resources. A common use for interactive batch jobs is debugging and testing. Running a batch-interactive job is done by using the -I option with bsub.

Here is an example command creating an interactive shell on compute nodes with internet access:

bsub -P AllocationAccount -q interactive -n 8 -W 15 -R span[hosts=1] -XF -Is /bin/bash

This command allocates a total of 8 cores on one of the interactive compute nodes, reserved exclusively for jobs running in interactive mode. All cores are on the same node. The -XF option enables X11 forwarding so graphics applications can open on your screen. Once the interactive shell has started, the user can execute jobs interactively there. In addition to those dedicated nodes, regular compute nodes with no internet access can also be used to open an interactive session by submitting a batch interactive job to non-interactive queues such as “premium”, “express”, and “gpu” using the bsub -I, -Is, and -Ip options. On these nodes, internet access can be enabled after issuing the command module load proxies.

Useful LSF Commands

bjobs — Show job status. Check the status of your own jobs in the queue. To display detailed information about a job in a multi-line format use bjobs -l JobID. If a job has already completed, bjobs won’t show information about it — use bhist to retrieve the job’s record from the LSF database.

bkill — Cancel a batch job. A job can be removed from the queue or killed if running, using bkill JobID. There are many ways to terminate a specific set of jobs:

  • To terminate a job by its job name: bkill -J myjob_1
  • To terminate a bunch of jobs using a wildcard: bkill -J myjob_*
  • To kill some selected array jobs: bkill JobID[1,7,10-25]
  • To kill all your jobs: bkill 0

bmod — Modify the resource requirement of a pending job. Many batch job specifications can be modified after submission and before it runs — typically job size (number of nodes), queue, and wall clock limit. Job specifications cannot be modified once a job enters the RUN state. For example: bmod -q express jobID changes the job’s queue to express; bmod -R rusage[mem=20000] jobID changes the job’s memory requirement to 20 GB per core. Note that -R replaces ALL R fields, not just the one you specify.

bpeek — Displays the stdout and stderr output of an unfinished job.

bqueues — Displays information about queues.

bhosts — Displays hosts and their static and dynamic resources.

For the full list of LSF commands, see the IBM Spectrum LSF Command Reference.

Pending Reasons

There could be many reasons for a job to stay in the queue longer than usual: the whole cluster may be busy; your job may request a large amount of compute resources (memory, cores) that no compute node can currently satisfy; or your job may overlap with a scheduled PM. To see the pending reason, use the bjobs command with the -l flag:

$ bjobs -l

LSF Queues And Policies

The command to check the queues is bqueues. To get more details about a specific queue, type bqueues -l, e.g. bqueues -l premium.

* The default memory for all queues is 3000 MB.

Queue Description Max Walltime Minerva Minerva-Res
premium Normal submission queue 144 hrs ✓ ✓
express Rapid turnaround jobs 12 hrs ✓ ✓
interactive Jobs running in interactive mode 12 hrs ✓ —
long Jobs requiring extended runtime 336 hrs ✓ —
gpu Jobs requiring GPU resources 144 hrs ✓ —
gpuexpress Short jobs requiring GPU resources 15 hrs ✓ —
ondemand Jobs dedicated to Open OnDemand 12 hrs ✓ —
private Jobs using dedicated resources Unlimited ✓ —
others Any other queues are for testing by the Scientific Computing group N/A ✓ —

Minerva-Res note: only express and premium are available on Minerva-Res. All other queues in the table above are Minerva-only.

Policies

LSF is configured to do “absolute priority scheduling (APS)” backfill. Backfilling allows smaller, shorter jobs to use otherwise idle resources.

To check the priority of pending jobs: $ bjobs -u all -aps

In certain special cases, the priority of a job may be manually increased upon request. To request a priority change, contact the Minerva HPC team at hpchelp@hpc.mssm.edu. We will need the job ID and reason to submit the request.

Improper Resource Specification

Minerva is a shared resource. Improper specification of job requirements not only wastes them but also prevents other researchers from running their jobs.

1. If your program is not explicitly written to use more than one core, specifying more than one core wastes cores that could be used by other users. E.g.:

IF bsub -n 6 < single_core_program THEN:

  • 6 cores allocated
  • program runs on 1 core
  • 5 cores are idle

2. If your program can use more than one core but does not use tools like mpirun, mpiexec, or torchrun, then all cores must be on the same node. This is a shared-memory multiprocessing (SMP) program. You must specify -R span[hosts=1] to ensure all cores are on the same node.

E.g.: IF bsub -n 6 < SMP_program THEN:

  • 6 cores are allocated: 1 on node 1, 2 on node 2, 3 on node 3
  • 1 core on node 1 is used to run the program
  • The other 5 cores sit idle

or, perhaps, IF bsub -n 6 < SMP_program THEN:

  • 6 cores are allocated: 3 on node 1, 1 on node 2, 2 on node 3
  • 3 cores on node 1 are used to run the program
  • The other 3 cores sit idle

But with -R span[hosts=1]: IF bsub -n 6 -R span[hosts=1] < SMP_program THEN:

  • 6 cores are allocated: 6 on node 1
  • All 6 cores on node 1 are used to run the program

3. Memory specified by -R rusage[mem=xxx] is reserved for your use and cannot be used by anyone else until released at the end of the job. If most/all of the memory on a node is reserved, no additional jobs can run even if CPUs are still available.

Example — IF:

  • A 192GB node has 3 jobs dispatched to it
  • Each job requests 1 core and 64GB of memory (-n 1 -R rusage[mem=64G])
  • Each is actually using only 1GB

THEN:

  • All memory on the node is reserved, so no other job can be dispatched to it.
  • Only 3GB of memory is being used.
  • 189 GB of memory and 43 cores are idle and cannot be used.

4. Check to see how much memory your program is actually using and request accordingly. If you’re running a series of jobs using the same program, run one or two test jobs with a large amount of memory and then adjust for the production runs.

Test:

bsub -R rusage[mem=6G] other options < testJob.lsf

Output:

Successfully completed.
Resource usage summary:
    CPU time :             1.82 sec.
    Max Memory :           8 MB
    Average Memory :       6.75 MB
    Total Requested Memory : 6144.00 MB
    Delta Memory :         6136.00 MB
    Max Swap :             -
    Max Processes :        3
    Max Threads :          3
    Run time :              2 sec.
    Turnaround time :      50 sec.

The test run shows this program used 8MB maximum. Memory usage varies between runs depending on what else is running on the node, so scale the request down but leave some “wiggle room”:

bsub -R rusage[mem=15M] other options < productionJob.lsf
Successfully completed.
Resource usage summary:
    CPU time :             2.93 sec.
    Max Memory :           5 MB
    Average Memory :       4.00 MB
    Total Requested Memory : 15.00 MB
    Delta Memory :         10.00 MB
    Max Swap :             -
    Max Processes :        3
    Max Threads :          3
    Run time :              4 sec.
    Turnaround time :      47 sec.

Minerva Training Session

Each spring and fall, the Scientific Computing and Data team hosts a series of training sessions, including one on the LSF Job Scheduler. Click here to see past and future Minerva training sessions.