[et_pb_section fb_built=”1″ _builder_version=”3.22″][et_pb_row _builder_version=”3.25″ background_size=”initial” background_position=”top_left” background_repeat=”repeat”][et_pb_column type=”4_4″ _builder_version=”3.25″ custom_padding=”|||” custom_padding__hover=”|||”][et_pb_text _builder_version=”4.9.0″ background_size=”initial” background_position=”top_left” background_repeat=”repeat” hover_enabled=”0″ sticky_enabled=”0″]<\/p>\n
We have added 2 new A100-80GB GPU nodes to the LSF queue. Each node is equipped with 2<\/span> TB of memory and 7<\/span> TB of local NVMe PCIe SSD to provide higher performance over the prior generation.<\/span><\/p>\n \u00a0<\/span><\/p>\n What are the A100-80GB GPU nodes on Minerva?<\/span><\/u><\/b><\/p>\n 8 A100 GPUs<\/span><\/b>\u00a0in 2 nodes\u00a0<\/span><\/p>\n \u00b7<\/span>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>64 Intel Xeon Platinum 8358 2.6 GHz CPU Processors per node,\u00a02<\/b><\/span> <\/b>TB memory per node<\/span><\/b>, for a total of 128 CPU cores<\/span><\/p>\n \u00b7<\/span>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>7.68<\/span><\/b> <\/b>TB NVMe PCIe SSD\u00a0<\/span><\/b>\u00a0(7.0TB usable) per node, which can deliver a sustained read-write speed of 3.5 GB\/s in contrast with SATA SSDs that limit at 600 MB\/s<\/span><\/p>\n \u00b7<\/span>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>4 A100 GPUs per node,\u00a080 GB of memory for each GPU,\u00a0<\/b>for a total 320 GB per node<\/span><\/p>\n \u00b7<\/span>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>The A100 is connected via NVLink<\/span><\/p>\n \u00a0<\/span><\/p>\n How to submit jobs to the A100-80GB GPU nodes?<\/span><\/u><\/b><\/p>\n \u00b7<\/span>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>A100-80GB GPU nodes are available in the GPU queue ( use the LSF flag\u00a0“-q gpu”\u00a0<\/b>).<\/span><\/p>\n \u00b7<\/span>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>To submit your jobs to those A100-80GB GPU nodes, flag “<\/span>-R a10080g<\/span><\/b>” is required.\u00a0 I.e., add\u00a0<\/span>\u00a0#BSUB -R a10080g\u00a0<\/span><\/b>to your LSF script or\u00a0<\/span>-R a10080g<\/span><\/b>\u00a0to your LSF command line. For example, the following requests 1 GPU card, 8 CPUs and 256GB of memory for 1hr:<\/span><\/p>\n \u00a0<\/span><\/p>\n bsub -P acc_xxx -q gpu -n 8 -R rusage[mem=32000] -R a10080g\u00a0-R rusage[ngpus_excl_p=1] -R span[hosts=1]\u00a0-W 01:00 -Is \/bin\/bash<\/span><\/p>\n \u00a0<\/span><\/p>\n \u00b7<\/span>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>Note, the gpu queue also contains other GPU nodes with V100 and A100 GPU cards. You can access those resources with the corresponding flags “-R v100”, “-R a100”. If GPU model flag is not specified, your job will start on the earliest available GPU nodes.<\/span><\/p>\n \u00a0<\/span><\/p>\n How to use the ssd on the A100-80GB GPU nodes?<\/span><\/u><\/b><\/p>\n \u00b7<\/span>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>The slink\u00a0<\/b><\/span>\/ssd<\/span><\/b>\u00a0points to the local\u00a0NVMe SSD storage. You can specify \/ssd in your job script and direct your temporary files there. At the end of your job script, please remember to clean up your temporary files.<\/b>\u00a0<\/span><\/p>\n \u00a0<\/span><\/p>\n What cuda version is supported on the A100-80GB GPU nodes?<\/span><\/u><\/b><\/p>\n \u00b7<\/span>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>Cuda 11.x or later is supported on those A100-80GB nodes. Please load the cuda module by\u00a0<\/span>ml cuda\/11.1\u00a0 or ml cuda\u00a0<\/span><\/b>( cuda\/11.1 is the default version currently)<\/span><\/p>\n