{"id":1447,"date":"2020-11-18T14:08:05","date_gmt":"2020-11-18T19:08:05","guid":{"rendered":"https:\/\/labs.icahn.mssm.edu\/minervalab\/?page_id=1447"},"modified":"2021-07-01T12:17:02","modified_gmt":"2021-07-01T16:17:02","slug":"new-a100-gpu-nodes-are-available-on-minerva","status":"publish","type":"page","link":"https:\/\/labs.icahn.mssm.edu\/minervalab\/new-a100-gpu-nodes-are-available-on-minerva\/","title":{"rendered":"New A100 GPU nodes are available on Minerva"},"content":{"rendered":"<div><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\">We have completed all the\u00a0tests on the<\/span><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\">\u00a0new A100 GPU<\/span><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\">\u00a0nodes. Those new<\/span><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\">\u00a0A100 GPU nodes are added to LSF gpu\u00a0queues with resource -R a100 needed.\u00a0<\/span><span style=\"font-family: Arial,sans-serif,serif,EmojiFont;font-size: medium\"><span id=\"x_m_853539228081763126x_x_docs-internal-guid-22dd755f-7fff-c62d-11da-69ed2717c328\"><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\">A100 provides higher performance over the prior generation with a detailed datasheet\u00a0<\/span><a href=\"https:\/\/www.nvidia.com\/content\/dam\/en-zz\/Solutions\/Data-Center\/a100\/pdf\/nvidia-a100-datasheet.pdf\" target=\"_blank\" rel=\"noopener noreferrer\"><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\">here<\/span><\/a><\/span><\/span><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\">\u00a0.<\/span><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\"><br \/>\n<\/span><\/div>\n<div><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\">\u00a0<\/span><\/div>\n<div><span style=\"color: #006fc9;font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\"><i><u>What are the A100 GPU<\/u><\/i><\/span><span style=\"color: #006fc9;font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\"><i><u>\u00a0nodes<\/u><\/i><\/span><span style=\"color: #006fc9;font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\"><i><u> on Minerva?<\/u><\/i><\/span><span style=\"font-family: Calibri,serif,EmojiFont;font-size: small\"><br \/>\n<\/span><\/div>\n<div><span style=\"color: black;font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\"><b>32\u00a0<\/b><\/span><span style=\"color: black;font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\"><b>A100 GPUs<\/b><\/span><span style=\"color: black;font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\">\u00a0in 8 nodes\u00a0<\/span><span style=\"color: black;font-family: Arial,sans-serif,serif,EmojiFont;font-size: medium\"><br \/>\n<\/span><\/div>\n<ul>\n<li><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: small\"><span style=\"color: black;font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\">48<\/span><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\">\u00a0<\/span><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\">Intel Xeon Platinum 8268 2.9 GHz CPU Processors per node,<\/span><span style=\"font-family: Arial,sans-serif,serif,EmojiFont;font-size: medium\"><span id=\"x_m_853539228081763126x_x_docs-internal-guid-22dd755f-7fff-c62d-11da-69ed2717c328\"><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\">\u00a0384 GB memory per node,\u00a0<\/span><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\">for a total of 384 CPU cores<\/span><span id=\"x_m_853539228081763126divtagdefaultwrapper\"><\/span><\/span><\/span><\/span><\/li>\n<li><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: small\"><span style=\"font-family: Arial,sans-serif,serif,EmojiFont;font-size: medium\"><span id=\"x_m_853539228081763126x_x_docs-internal-guid-22dd755f-7fff-c62d-11da-69ed2717c328\"><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\"><b>1.92 TB SSD\u00a0<\/b><\/span><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\">\u00a0(1.8 TB usable) per node<\/span><\/span><\/span><\/span><\/li>\n<li><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: small\"><span style=\"font-family: Arial,sans-serif,serif,EmojiFont;font-size: medium\"><span id=\"x_m_853539228081763126x_x_docs-internal-guid-22dd755f-7fff-c62d-11da-69ed2717c328\"><span id=\"x_m_853539228081763126divtagdefaultwrapper\"><\/span><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\">4\u00a0A100 GPUs per node,\u00a0<\/span><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\"><b>40 GB of memory for each GPU,\u00a0<\/b><\/span><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\">for a total 160 GB per node<\/span><\/span><\/span><\/span><\/li>\n<li><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: small\"><span style=\"font-family: Arial,sans-serif,serif,EmojiFont;font-size: medium\"><span id=\"x_m_853539228081763126x_x_docs-internal-guid-22dd755f-7fff-c62d-11da-69ed2717c328\"><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\">The A100\u00a0 is connected via PCIe<\/span><\/span><\/span><\/span><\/li>\n<\/ul>\n<div><span style=\"color: #006fc9;font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\"><i><u>How to submit jobs to the A100 GPU<\/u><\/i><\/span><span style=\"color: #006fc9;font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\"><i><u>\u00a0nodes<\/u><\/i><\/span><span style=\"color: #006fc9;font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\"><i><u>?<\/u><\/i><\/span><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\"><br \/>\n<\/span><\/div>\n<ul>\n<li><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\">A100 GPU nodes are available in the GPU queue (\u00a0use\u00a0LSF\u00a0flag <b>&#8220;-q gpu<\/b><b>&#8221;\u00a0<\/b>).<\/span><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\"><br \/>\n<\/span><\/li>\n<li>To submit your jobs to those A100 GPU nodes, fag &#8220;<span style=\"color: #ff0000\"><b>-R a100<\/b><\/span>&#8221;\u00a0is required.\u00a0\u00a0I.e., add\u00a0<span style=\"color: #ff0000\"><b>\u00a0<\/b><b>#BSUB<\/b><b>\u00a0<\/b><\/span><b><span style=\"color: #ff0000\">-R a100<\/span>\u00a0<\/b>to your LSF script or<span style=\"color: #ff0000\">\u00a0<b>-R a100<\/b><\/span>\u00a0to your LSF command line.<\/li>\n<li><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\">Note, the gpu queue also contains GPU nodes with V100 and P100 GPU cards. You can access those resources with the corresponding flags &#8220;-R v100&#8221;, &#8220;-R p100&#8221;.\u00a0If GPU model\u00a0flag\u00a0is not specified, your job will start on the earliest available GPU nodes.<\/span><\/li>\n<\/ul>\n<div><\/div>\n<div><span style=\"color: #006fc9;font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\"><i><u>How to use the ssd on the A100 GPU<\/u><\/i><\/span><span style=\"color: #006fc9;font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\"><i><u>\u00a0nodes<\/u><\/i><\/span><span style=\"color: #006fc9;font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\"><i><u>?<\/u><\/i><\/span><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\"><br \/>\n<\/span><\/div>\n<ul>\n<li><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\">To take advantage of local\u00a0<\/span><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: small\">1.8 TB<\/span><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\"> SSD, please request the resource such as\u00a0<\/span><span style=\"color: red;font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\"><b>&#8220;<\/b><\/span><span style=\"color: red;font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\"><b>-R rusage[ssd_gb=1000]<\/b><\/span><span style=\"color: red;font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\"><b>&#8220;<\/b><\/span><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\"><b>.<\/b><\/span><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\">\u00a0This flag will allocate 1000GB of dedicated SSD\u00a0space to your job. We would advise you to request the ssd_rg less or around 1500 (1.5T). <\/span><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\"><br \/>\n<\/span><\/li>\n<li><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\">The\u00a0slink<\/span><b> <\/b><span style=\"color: red;font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\"><b>\/ssd<\/b><\/span><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\"> points to the ssd storage. You can specify \/ssd in your job script and direct your temporary files there. At the end of your job script, please remember to clean up your temporary files.\u00a0<\/span><\/li>\n<\/ul>\n<div><\/div>\n<div><span style=\"color: #006fc9;font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\"><i><u>What cuda version is supported on\u00a0the A100 GPU<\/u><\/i><\/span><span style=\"color: #006fc9;font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\"><i><u>\u00a0nodes<\/u><\/i><\/span><span style=\"color: #006fc9;font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\"><i><u>?<\/u><\/i><\/span><\/div>\n<div>\n<ul>\n<li><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\">Cuda 11.x or later is supported on those A100 nodes. Please load the Cuda module by\u00a0<\/span><span style=\"color: red;font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\"><b>ml cuda\/11.1\u00a0 or ml cuda <\/b><\/span><span style=\"color: black;font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\">( cuda\/11.1 is the default version currently)<\/span><\/li>\n<\/ul>\n<\/div>\n<div><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\"><b>\u00a0<\/b><\/span><\/div>\n<div><\/div>\n<div><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\">If you have any question on this,\u00a0<\/span><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\">please send us a ticket at<\/span><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\">\u00a0<\/span><a href=\"mailto:hpchelp@hpc.mssm.edu\" target=\"_blank\" rel=\"noopener noreferrer\"><span style=\"font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: small\"><span style=\"color: #1155cc;font-family: Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size: medium\">hpchelp@hpc.mssm.edu<\/span><\/span><\/a><\/div>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>We have completed all the\u00a0tests on the\u00a0new A100 GPU\u00a0nodes. Those new\u00a0A100 GPU nodes are added to LSF gpu\u00a0queues with resource -R a100 needed.\u00a0A100 provides higher performance over the prior generation with a detailed datasheet\u00a0here\u00a0. \u00a0 What are the A100 GPU\u00a0nodes on Minerva? 32\u00a0A100 GPUs\u00a0in 8 nodes\u00a0 48\u00a0Intel Xeon Platinum 8268 2.9 GHz CPU Processors per [&hellip;]<\/p>\n","protected":false},"author":396,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":"","footnotes":""},"class_list":["post-1447","page","type-page","status-publish","hentry"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/labs.icahn.mssm.edu\/minervalab\/wp-json\/wp\/v2\/pages\/1447","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/labs.icahn.mssm.edu\/minervalab\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/labs.icahn.mssm.edu\/minervalab\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/labs.icahn.mssm.edu\/minervalab\/wp-json\/wp\/v2\/users\/396"}],"replies":[{"embeddable":true,"href":"https:\/\/labs.icahn.mssm.edu\/minervalab\/wp-json\/wp\/v2\/comments?post=1447"}],"version-history":[{"count":6,"href":"https:\/\/labs.icahn.mssm.edu\/minervalab\/wp-json\/wp\/v2\/pages\/1447\/revisions"}],"predecessor-version":[{"id":1462,"href":"https:\/\/labs.icahn.mssm.edu\/minervalab\/wp-json\/wp\/v2\/pages\/1447\/revisions\/1462"}],"wp:attachment":[{"href":"https:\/\/labs.icahn.mssm.edu\/minervalab\/wp-json\/wp\/v2\/media?parent=1447"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}