Scientific Computing and Data

Partnering with researchers to advance scientific discovery

GPU Etiquette

The GPU nodes in Minerva can be in great demand. Because of the manner in which LSF allocates GPU resources to a job, it is important to specify the job requirements carefully. Incorrect LSF resource requests can cause resources to be unnecessarily reserved and, hence, unavailable to other researchers. 

 

The bsub options that need particular attention are:

 

-n The number of slots assigned to your job.
-R rusage[mem=] The amount of memory assigned to each slot.
-R rusage[ngpus_excl_p=] The number of GPU cards per node assigned to your job.
-R span[] The arrangement of the resources assigned to your job across Minerva nodes.

 

-n

This option specifies how many job slots you want for your job. View this value as how many packets of resources you want allocated to your job. By default, each slot you ask for will have 1 cpu and 3GB of memory. For most cases, job slots and cpus are synonymous. Note that if your program is not parallelized, adding more cores will not affect the performance.

 

-R rusage[mem=]

This option specifies how much memory is to be allocated per slot. Default is 3GB. Note that request is per slot, not per job. The total amount of memory requested will be this value times the number of slots you have requested with the -n option.

 

-R rusage[ngpus_excl_p=]

This option specifies how many GPU cards you want allocated per compute node, not slot, that is allocated to your job. The GPU cards/devices are numbered 0 through 3. 

 

Because your program needs to connect to a specific GPU device, LSF will set an environment variable, CUDA_VISIBLE_DEVICES, to the list of GPU devices assigned to your job. E.g, CUDA_VISIBLE_DEVICES=0,3. Do not change these values manually and, if writing your own code, you must honor the assignment. The installed software on Minerva that use GPUs honor these assignments.

 

-R span[]

This option tells LSF how to lay out your slots/cpus on the compute nodes of Minerva. By default, slots/cpus can be, and usually are, spread out over several nodes. 

 

There are two ways that this option is generally specified: using ptile=x, to specify the tiling or hosts=x to specify how many nodes to use.

 

Example A:

-R span[ptile=<value>] where value is the number of slots/cores to be placed on a node.

 E.g.

 

#BSUB -n 6

#BSUB -R rusage[mem=5G]

#BSUB -R span[ptile=4]

 

Will result in 2 nodes being allocated: one with 4 cpus and 20GB (4*5GB) of memory and a second with 2 cpus and 10GB (2*5GB) of memory.

 

Example B:

-R span[hosts=<value>] where value is the number of hosts to use

Eg.

 

#BSUB -n 6

#BSUB -R rusage[mem=5G]

#BSUB -R span[hosts=1]

 

Will result in 1 node being allocated with 6 cpus and 30GB (6*5GB) of memory.

 

If your program Is not parallelized or is a Shared Memory Parallel (SMP) program, and most parallel programs in our environment are, only the resources on the first node of those nodes assigned to your job will be used. The remainder will be unavailable to other users and may cause significant backlogs in job processing.

 

Some hints that your program cannot run in a distributed manner are

  • Documentation does not say that it can.
  • The mpirun command is not used to start the program.
  • Has program options such as 
    • -ncpu
    • -threads

 

In these cases all resources must be on a single node and you should specify

-R span[hosts=1]

to ensure all resources are placed on a single node.

 

This is crucial when GPUs are involved. GPU devices are valuable and there are not a large number of them. LSF will allocate the number of GPUs you request with the  rusage[ngpus_excl_p=] option on each node that is allocated to your job. 

 

#BSUB -n 6

#BSUB -R rusage[mem=5G]

#BSUB -R [ngpus_excl_p=2]

#BSUB -R span[ptile=4]

 

Will result in 2 nodes being allocated: one with 4 cpus, 20GB of memory, and 2GPUs  and a second with 2 cpus, 10GB of memory, and 2GPUs.

 

If your program is single thread or an SMP program, only the first node will be used and the GPUs on the second node will be allocated but unused and probably preventing another job from being dispatched. Even worse, if you do not use the span option, LSF could distribute one core to each of 6 nodes where each node will have 2 GPU’s reserved. In this case 10 GPUs would be wasted.