{"id":965,"date":"2020-01-16T15:26:42","date_gmt":"2020-01-16T20:26:42","guid":{"rendered":"http:\/\/labs.icahn.mssm.edu\/minervalab\/?page_id=965"},"modified":"2025-12-12T14:41:17","modified_gmt":"2025-12-12T19:41:17","slug":"minerva-quick-start","status":"publish","type":"page","link":"https:\/\/labs.icahn.mssm.edu\/minervalab\/minerva-quick-start\/","title":{"rendered":"Minerva Quick Start"},"content":{"rendered":"<p>[et_pb_section fb_built=&#8221;1&#8243; fullwidth=&#8221;on&#8221; _builder_version=&#8221;4.16&#8243; _module_preset=&#8221;default&#8221; global_colors_info=&#8221;{}&#8221;][et_pb_fullwidth_menu menu_id=&#8221;15&#8243; menu_style=&#8221;centered&#8221; fullwidth_menu=&#8221;on&#8221; active_link_color=&#8221;#d80b8c&#8221; dropdown_menu_bg_color=&#8221;#221f72&#8243; dropdown_menu_line_color=&#8221;#221f72&#8243; dropdown_menu_active_link_color=&#8221;#d80b8c&#8221; _builder_version=&#8221;4.16&#8243; _module_preset=&#8221;default&#8221; menu_font=&#8221;|600|||||||&#8221; menu_text_color=&#8221;#FFFFFF&#8221; menu_font_size=&#8221;16px&#8221; background_color=&#8221;#221f72&#8243; background_layout=&#8221;dark&#8221; sticky_position=&#8221;top&#8221; global_colors_info=&#8221;{}&#8221;][\/et_pb_fullwidth_menu][\/et_pb_section][et_pb_section fb_built=&#8221;1&#8243; _builder_version=&#8221;4.16&#8243; _module_preset=&#8221;default&#8221; custom_padding=&#8221;0px||0px||false|false&#8221; global_colors_info=&#8221;{}&#8221;][et_pb_row _builder_version=&#8221;4.16&#8243; _module_preset=&#8221;default&#8221; custom_padding=&#8221;||0px||false|false&#8221; global_colors_info=&#8221;{}&#8221;][et_pb_column type=&#8221;4_4&#8243; _builder_version=&#8221;4.16&#8243; _module_preset=&#8221;default&#8221; global_colors_info=&#8221;{}&#8221;][et_pb_text admin_label=&#8221;Breadcrumb&#8221; _builder_version=&#8221;4.16&#8243; _module_preset=&#8221;default&#8221; global_colors_info=&#8221;{}&#8221;]<\/p>\n<p><a href=\"https:\/\/labs.icahn.mssm.edu\/minervalab\/scientific-computing-and-data\/\">Scientific Computing and Data<\/a> \/ <a href=\"https:\/\/labs.icahn.mssm.edu\/minervalab\/\">High Performance Computing<\/a> \/ Minerva Quick Start<\/p>\n<p>[\/et_pb_text][\/et_pb_column][\/et_pb_row][\/et_pb_section][et_pb_section fb_built=&#8221;1&#8243; _builder_version=&#8221;4.16&#8243; custom_padding=&#8221;||0px||false|false&#8221; global_colors_info=&#8221;{}&#8221;][et_pb_row _builder_version=&#8221;4.16&#8243; background_size=&#8221;initial&#8221; background_position=&#8221;top_left&#8221; background_repeat=&#8221;repeat&#8221; custom_padding=&#8221;||0px||false|false&#8221; global_colors_info=&#8221;{}&#8221;][et_pb_column type=&#8221;4_4&#8243; _builder_version=&#8221;4.16&#8243; custom_padding=&#8221;|||&#8221; global_colors_info=&#8221;{}&#8221; custom_padding__hover=&#8221;|||&#8221;][et_pb_text admin_label=&#8221;Minerva Quick Start Header&#8221; _builder_version=&#8221;4.16&#8243; header_font=&#8221;|600|||||||&#8221; header_text_color=&#8221;#221f72&#8243; background_size=&#8221;initial&#8221; background_position=&#8221;top_left&#8221; background_repeat=&#8221;repeat&#8221; global_colors_info=&#8221;{}&#8221;]<\/p>\n<h1>Minerva Quick Start Guide<\/h1>\n<p>[\/et_pb_text][\/et_pb_column][\/et_pb_row][et_pb_row column_structure=&#8221;1_2,1_2&#8243; admin_label=&#8221;Row&#8221; _builder_version=&#8221;4.16&#8243; _module_preset=&#8221;default&#8221; custom_padding=&#8221;||0px||false|false&#8221; global_colors_info=&#8221;{}&#8221;][et_pb_column type=&#8221;1_2&#8243; _builder_version=&#8221;4.16&#8243; _module_preset=&#8221;default&#8221; global_colors_info=&#8221;{}&#8221;][et_pb_text admin_label=&#8221;Partitions details&#8221; _builder_version=&#8221;4.16&#8243; _module_preset=&#8221;default&#8221; header_text_color=&#8221;#221f72&#8243; header_2_text_color=&#8221;#221f72&#8243; header_2_font_size=&#8221;24px&#8221; global_colors_info=&#8221;{}&#8221;]<\/p>\n<h2>Partitions<\/h2>\n<p><strong>The Chimera partition:<\/strong><\/p>\n<ul>\n<li>286 compute nodes &#8211; 48 Intel 8168 cores (2.7GHz) and 192 GB memory<\/li>\n<li>4x high memory nodes &#8211; 48 Intel 8168 cores (2.7GHz) and 1.5 TB memory<\/li>\n<li>48 V100 GPUs in 12 nodes &#8211; 32 Intel 6142 cores (2.6GHz) and 384 GB memory &#8211; 4x V100-16 GB GPU<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><strong>The BODE2 partition:<\/strong><\/p>\n<ul>\n<li>78 compute nodes \u2013 48 Intel 8268 cores (2.9GHz) and 192 GB memory<\/li>\n<li>Only BODE-enabled users have access to the BODE2 partition<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><strong>The CATS partition:<\/strong><\/p>\n<ul>\n<li>3,520 64-core 2.6 GHz Intel IceLake processors in 55 nodes<\/li>\n<li class=\"m_4748730477779924083MsoListParagraph\">1.5 TB\u00a0of memory per node<\/li>\n<li class=\"m_4748730477779924083MsoListParagraph\">82.5 TB memory (collectively)<\/li>\n<li>Eligible for all NIH-funded projects<\/li>\n<\/ul>\n<p>[\/et_pb_text][\/et_pb_column][et_pb_column type=&#8221;1_2&#8243; _builder_version=&#8221;4.16&#8243; _module_preset=&#8221;default&#8221; global_colors_info=&#8221;{}&#8221;][et_pb_image src=&#8221;https:\/\/labs.icahn.mssm.edu\/minervalab\/wp-content\/uploads\/sites\/342\/2021\/09\/Minerva-1024&#215;624.png&#8221; title_text=&#8221;Minerva&#8221; admin_label=&#8221;Minerva Image&#8221; _builder_version=&#8221;4.16&#8243; _module_preset=&#8221;default&#8221; global_colors_info=&#8221;{}&#8221;][\/et_pb_image][\/et_pb_column][\/et_pb_row][et_pb_row _builder_version=&#8221;4.16&#8243; background_size=&#8221;initial&#8221; background_position=&#8221;top_left&#8221; background_repeat=&#8221;repeat&#8221; global_colors_info=&#8221;{}&#8221;][et_pb_column type=&#8221;4_4&#8243; _builder_version=&#8221;4.16&#8243; custom_padding=&#8221;|||&#8221; global_colors_info=&#8221;{}&#8221; custom_padding__hover=&#8221;|||&#8221;][et_pb_text _builder_version=&#8221;4.27.4&#8243; header_text_color=&#8221;#221f72&#8243; header_2_text_color=&#8221;#221f72&#8243; header_2_font_size=&#8221;24px&#8221; background_size=&#8221;initial&#8221; background_position=&#8221;top_left&#8221; background_repeat=&#8221;repeat&#8221; global_colors_info=&#8221;{}&#8221;]<\/p>\n<h2>Connecting to Minerva<\/h2>\n<p>For security, Minerva uses the Secure Shell (ssh) protocol and Two Factor authentication. Unix systems typically have an ssh client already installed. Windows systems can download one of several ssh clients that are available for free such as <a href=\"http:\/\/www.chiark.greenend.org.uk\/~sgtatham\/putty\/\">PuTTY<\/a>.<\/p>\n<p>Two Factor authentication requires you to enter a password that is the combination of your Sinai password and a generated token (Either Software or Hardware token).<br \/>Software Token:<br \/>On an Android and\/or iPhone, the application is called &#8220;VIP Access&#8221; and is published by Symantec. Blackberry, Windows Mobile, etc are also supported.<\/p>\n<p>Hardware Token:<br \/>You can obtain a Hardware Token from the IT Helpdesk. We don&#8217;t have Hardware Token available now.<br \/>To setup two factor authentication visit the <a href=\"https:\/\/itsecurity.mssm.edu\/vip-two-factor-setup\/\">ASCIT website<\/a><\/p>\n<p><b>From on-site and off-site<\/b><\/p>\n<p>All users can login to Minerva cluster via ssh to minerva.hpc.mssm.edu. As a part of our HIPAA compliance activities, we need to shut down the external gateway access to Minerva. The High Performance Computing team has made adjustments so that all users can connect to internal login nodes, thus<strong>\u00a0all users<\/strong>\u00a0will need a VPN account\u00a0<strong>for off-campus login.<\/strong> Please refer to<a href=\"https:\/\/labs.icahn.mssm.edu\/minervalab\/internal-users-minerva-action-required-external-login-nodes-being-removed-on-april-15\/\"> here<\/a> for details.<\/p>\n<p>For example:<\/p>\n<p>&gt; ssh your_userid@minerva.hpc.mssm.edu<br \/>Password: &gt; your_Sinai_password123456<\/p>\n<p>( the &gt; sign indicates what you would type in; 123456 represents the numeric sequence obtained from your token)<\/p>\n<p><a href=\"https:\/\/labs.icahn.mssm.edu\/minervalab\/documentation\/logging-in\/\">Click here<\/a> for more about logging in.<\/p>\n<hr \/>\n<h1><\/h1>\n<h2>File System<\/h2>\n<table style=\"border: 1px solid #cccccc\">\n<tbody>\n<tr>\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\"><span style=\"color: #339966\">\/hpc\/users\/<\/span>&lt;userid&gt;<\/td>\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\">User HOME directories. 30 GB quota.\u00a0 <strong>It is NOT purged and is backed up.<\/strong>\u00a0 Generally used for all the \u2018rc\u2019 and configuration files for various programs.<\/td>\n<\/tr>\n<tr>\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\"><span style=\"color: #339966\">\/sc\/arion\/work\/<\/span>&lt;userid&gt;<\/td>\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\">A WORK directory for each user. 100 GB quota.\u00a0 <strong>It is NOT purged and it is NOT backed up<\/strong>. To be used for whatever purpose the user desires.<\/td>\n<\/tr>\n<tr>\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\"><span style=\"color: #339966\">\/sc\/arion\/scratch\/<\/span>&lt;userid&gt;<\/td>\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\">\n<p>A folder for each user inside the \/sc\/arion\/scratch directory.<\/p>\n<p>\/sc\/arion\/scratch has a 100TB quota and it is shared by all users.\u00a0 This should be used in lieu of \/tmp for temporary files as well as short term storage up to a maximum of 14 days.\u00a0 <strong>Files older than 14 days are purged automatically by the system.<\/strong><\/p>\n<\/td>\n<\/tr>\n<tr>\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\"><span style=\"color: #339966\">\/sc\/arion\/projects\/<\/span>&lt;projectid&gt;<\/td>\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\">\n<p>PI\u2019s can request project storage. <a href=\"https:\/\/labs.icahn.mssm.edu\/minervalab\/minerva-project-allocation\/\">Click here<\/a> to submit an allocation request and renew annually.<\/p>\n<p>A directory for each approved project. The quota is set to the approved allocation for the project.\u00a0 <strong>It is NOT purged but it is NOT backed up.\u00a0 <\/strong><\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>&nbsp;<\/p>\n<hr \/>\n<h1><\/h1>\n<h2>Queues<\/h2>\n<p>The queues that are available are:<br \/>Default memory per core is set as 3000MB for all the queues.<\/p>\n<table style=\"border: 1px solid #cccccc;height: 264px\">\n<tbody>\n<tr style=\"height: 24px\">\n<td style=\"padding: 5px;border: 1px solid #cccccc;height: 24px;width: 312px\"><b>Queue<\/b><\/td>\n<td style=\"padding: 5px;border: 1px solid #cccccc;height: 24px;width: 312px\"><b>Description<\/b><\/td>\n<td style=\"padding: 5px;border: 1px solid #cccccc;height: 24px;width: 312px\"><b>Max Walltime<\/b><\/td>\n<\/tr>\n<tr style=\"height: 72px\">\n<td style=\"padding: 5px;border: 1px solid #cccccc;height: 72px;width: 312px\"><b>Premium<\/b><\/td>\n<td style=\"padding: 5px;border: 1px solid #cccccc;height: 72px;width: 312px\">Jobs requesting high priority with APS doubled as 200. Charged at 150% of alloc rate<\/td>\n<td style=\"padding: 5px;border: 1px solid #cccccc;height: 72px;width: 312px\">144 hrs.<\/td>\n<\/tr>\n<tr style=\"height: 24px\">\n<td style=\"padding: 5px;border: 1px solid #cccccc;height: 24px;width: 312px\"><b>express<\/b><\/td>\n<td style=\"padding: 5px;border: 1px solid #cccccc;height: 24px;width: 312px\">Jobs requiring less than 12 hours walltime<\/td>\n<td style=\"padding: 5px;border: 1px solid #cccccc;height: 24px;width: 312px\">12 hrs.<\/td>\n<\/tr>\n<tr style=\"height: 24px\">\n<td style=\"padding: 5px;border: 1px solid #cccccc;height: 24px;width: 312px\"><b>interactive<\/b><\/td>\n<td style=\"padding: 5px;border: 1px solid #cccccc;height: 24px;width: 312px\">Jobs running in interactive mode for CPU or GPU computing<\/td>\n<td style=\"padding: 5px;border: 1px solid #cccccc;height: 24px;width: 312px\">12 hrs.<\/td>\n<\/tr>\n<tr style=\"height: 48px\">\n<td style=\"padding: 5px;border: 1px solid #cccccc;height: 48px;width: 312px\"><b>long<\/b><\/td>\n<td style=\"padding: 5px;border: 1px solid #cccccc;height: 48px;width: 312px\">Jobs requiring more than 144 hours walltime<\/td>\n<td style=\"padding: 5px;border: 1px solid #cccccc;height: 48px;width: 312px\">2 weeks<\/td>\n<\/tr>\n<tr style=\"height: 24px\">\n<td style=\"padding: 5px;border: 1px solid #cccccc;height: 24px;width: 312px\"><b>gpu<\/b><b><\/b><\/td>\n<td style=\"padding: 5px;border: 1px solid #cccccc;height: 24px;width: 312px\">Jobs running on GPU nodes<\/td>\n<td style=\"padding: 5px;border: 1px solid #cccccc;height: 24px;width: 312px\">144 hrs.<\/td>\n<\/tr>\n<tr style=\"height: 24px\">\n<td style=\"padding: 5px;border: 1px solid #cccccc;height: 24px;width: 312px\"><b>gpuexpress<\/b><\/td>\n<td style=\"padding: 5px;border: 1px solid #cccccc;height: 24px;width: 312px\">Jobs runnning on GPU nodes requiring less than 15 hours walltime<\/td>\n<td style=\"padding: 5px;border: 1px solid #cccccc;height: 24px;width: 312px\">15 hrs.<\/td>\n<\/tr>\n<tr style=\"height: 24px\">\n<td style=\"padding: 5px;border: 1px solid #cccccc;height: 24px;width: 312px\"><b>private<\/b><\/td>\n<td style=\"padding: 5px;border: 1px solid #cccccc;height: 24px;width: 312px\">Jobs using dedicated resources<\/td>\n<td style=\"padding: 5px;border: 1px solid #cccccc;height: 24px;width: 312px\">unlimited<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>&nbsp;<\/p>\n<hr \/>\n<h1><\/h1>\n<h2>LSF<\/h2>\n<p>Minerva uses LSF for batch submission. bsub is the submission command. Options can be put on the command line or in the submission script. HOWEVER, if the options are placed in the submission script, you must feed the script into the bsub command via stdin for the options to be read: E.g.,<\/p>\n<p>cat MyLSF.script | bsub<br \/>or<br \/>bsub &lt; MyLSF.script<\/p>\n<p>&nbsp;<\/p>\n<p><strong>Some important points of interest:<\/strong><\/p>\n<ul>\n<li>The default disposition for output and logs is for LSF to email the output to you. This piece is not working yet so you must use the &#8220;-o&#8221; option to save the output.<\/li>\n<li>In general, the shortest quantum of time in LSF is 1 minute. Wall time is expressed as HHH:MM &#8212; There are no seconds. Durations are generally in minutes.<\/li>\n<li>System level checkpoints are supported by LSF. There are some &#8220;gotchas&#8221; ( E.g., the default method does not work on our system) so check with the SC staff if you need\/want to do checkpointing.<\/li>\n<\/ul>\n<p>Some useful commands:<\/p>\n<p>bjobs &#8211; shows all your jobs in the queue<br \/>bpeek &#8211; peek at your output before the job ends<br \/>bqueues &#8211; what queues are available<br \/>bkill &#8211; kill a job<\/p>\n<p><a href=\"https:\/\/labs.icahn.mssm.edu\/minervalab\/documentation\/\">Click here<\/a> for additional Minerva documentation.<\/p>\n<p>&nbsp;<\/p>\n<hr \/>\n<h2>Acknowledging Mount Sinai in Your Work<\/h2>\n<p>Utilizing S10 BODE and CATS partitions requires acknowledgements of support by NIH in your publications. To assist, we have provided exact wording of acknowledgements required by NIH for your use. <a href=\"https:\/\/labs.icahn.mssm.edu\/minervalab\/mount-sinai-data-warehouse-msdw\/acknowledge-scientific-computing-at-mount-sinai\/\">Click here for acknowledgements<\/a>.<\/p>\n<p>Supported by grant UL1TR004419 from the National Center for Advancing Translational Sciences, National Institutes of Health.<\/p>\n<p>[\/et_pb_text][\/et_pb_column][\/et_pb_row][\/et_pb_section]<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Scientific Computing and Data \/ High Performance Computing \/ Minerva Quick StartMinerva Quick Start GuidePartitions The Chimera partition: 286 compute nodes &#8211; 48 Intel 8168 cores (2.7GHz) and 192 GB memory 4x high memory nodes &#8211; 48 Intel 8168 cores (2.7GHz) and 1.5 TB memory 48 V100 GPUs in 12 nodes &#8211; 32 Intel 6142 [&hellip;]<\/p>\n","protected":false},"author":415,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_et_pb_use_builder":"on","_et_pb_old_content":"<h2><\/h2>\r\n\u00a0\r\n<h2><img class=\"alignnone size-full wp-image-1180\" src=\"https:\/\/labs.icahn.mssm.edu\/minervalab\/wp-content\/uploads\/sites\/342\/2020\/03\/Untitled-9.png\" alt=\"\" width=\"722\" height=\"157\" \/><\/h2>\r\n<h2><span style=\"color: #221f72\">Partitions<\/span><\/h2>\r\n<strong>The Chimera partition:<\/strong>\r\n<ul>\r\n \t<li>286 compute nodes - 48 Intel 8168 cores (2.7GHz) and 192 GB memory<\/li>\r\n \t<li>4x high memory nodes - 48 Intel 8168 cores (2.7GHz) and 1.5 TB memory<\/li>\r\n \t<li>48 V100 GPUs in 12 nodes - 32 Intel 6142 cores (2.6GHz) and 384 GB memory - 4x V100-16 GB GPU<\/li>\r\n<\/ul>\r\n<strong>The BODE2 partition:<\/strong>\r\n<ul>\r\n \t<li>78 compute nodes \u2013 48 Intel 8268 cores (2.9GHz) and 192 GB memory<\/li>\r\n \t<li>Only BODE-enabled users have access to the BODE2 partition<\/li>\r\n<\/ul>\r\n\r\n<hr \/>\r\n\r\n<h2><span style=\"color: #221f72\">Connecting to Minerva<\/span><\/h2>\r\nFor security, Minerva uses the Secure Shell (ssh) protocol and Two Factor authentication. Unix systems typically have an ssh client already installed. Windows systems can download one of several ssh clients that are available for free such as <a href=\"http:\/\/www.chiark.greenend.org.uk\/~sgtatham\/putty\/\">PuTTY<\/a>.\r\n\r\nTwo Factor authentication requires you to enter a password that is the combination of your Sinai password and a generated token (Either Software or Hardware token).\r\nSoftware Token:\r\nOn an Android and\/or iPhone, the application is called \"VIP Access\" and is published by Symantec. Blackberry, Windows Mobile, etc are also supported.\r\n\r\nHardware Token:\r\nYou can obtain a Hardware Token from the IT Helpdesk. We don't have Hardware Token available now.\r\nTo setup two factor authentication visit the <a href=\"https:\/\/itsecurity.mssm.edu\/vip-two-factor-setup\/\">ASCIT website<\/a>\r\n\r\n<b>From on-site and off-site<\/b>\r\n\r\nAll users can login to Minerva cluster via ssh to minerva.hpc.mssm.edu. As a part of our HIPAA compliance activities, we need to shut down the external gateway access to Minerva. The High Performance Computing team has made adjustments so that all users can connect to internal login nodes, thus<strong>\u00a0all users<\/strong>\u00a0will need a VPN account\u00a0<strong>for off-campus login.<\/strong> Please refer to<a href=\"https:\/\/labs.icahn.mssm.edu\/minervalab\/internal-users-minerva-action-required-external-login-nodes-being-removed-on-april-15\/\"> here<\/a> for details.\r\n\r\nFor example:\r\n\r\n> ssh your_userid@minerva.hpc.mssm.edu\r\nPassword: > your_Sinai_password123456\r\n\r\n( the > sign indicates what you would type in; 123456 represents the numeric sequence obtained from your token)\r\n\r\nFor more information on logging in, please visit\u00a0<a href=\"https:\/\/labs.icahn.mssm.edu\/minervalab\/logging-in\/\">Logging In<\/a>\r\n\r\n<hr \/>\r\n\r\n<h2><span style=\"color: #221f72\">File System<\/span><\/h2>\r\n<table style=\"border: 1px solid #cccccc\">\r\n<tbody>\r\n<tr>\r\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\"><span style=\"color: #339966\">\/hpc\/users\/<\/span><userid><\/td>\r\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\">User HOME directories.\u00a0 20GB quota.\u00a0 <strong>It is NOT purged and is backed up.<\/strong>\u00a0 Generally used for all the \u2018rc\u2019 and configuration files for various programs.<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\"><span style=\"color: #339966\">\/sc\/arion\/work\/<\/span><userid><\/td>\r\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\">A WORK directory for each user.\u00a0 100GB quota.\u00a0 <strong>It is NOT purged and it is NOT backed up<\/strong>. To be used for whatever purpose the user desires.<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\"><span style=\"color: #339966\">\/sc\/arion\/scratch\/<\/span><userid><\/td>\r\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\">A folder for each user inside the \/sc\/arion\/scratch directory.\r\n\r\n\/sc\/arion\/scratch has a 100TB quota and it is shared by all users.\u00a0 This should be used in lieu of \/tmp for temporary files as well as short term storage up to a maximum of 14 days.\u00a0 <strong>Files older than 14 days are purged automatically by the system.<\/strong><\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\"><span style=\"color: #339966\">\/sc\/arion\/projects\/<\/span><projectid><\/td>\r\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\">PI\u2019s can request project storage. Need to submit an allocation request and renew annual at <a href=\"https:\/\/labs.icahn.mssm.edu\/minervalab\/event\/allocation-renewal-period-20-21\/\">https:\/\/labs.icahn.mssm.edu\/minervalab\/event\/allocation-renewal-period-20-21\/<\/a>\r\n\r\nA directory for each approved project.\u00a0 The quota is set to the approved allocation for the project.\u00a0 <strong>It is NOT purged but it is NOT backed up.\u00a0 <\/strong><\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n\u00a0\r\n\r\n<hr \/>\r\n\r\n<h2><span style=\"color: #221f72\">Queues<\/span><\/h2>\r\nThe queues that are available are:\r\nDefault memory per core is set as 3000MB for all the queues.\r\n<table style=\"border: 1px solid #cccccc\">\r\n<tbody>\r\n<tr>\r\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\"><b>Queue<\/b><\/td>\r\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\"><b>Description<\/b><\/td>\r\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\"><b>Max Walltime<\/b><\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\"><b>Premium<\/b><\/td>\r\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\">Jobs requesting high priority with APS doubled as 200. Charged at 150% of alloc rate<\/td>\r\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\">144 hrs.<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\"><b>express<\/b><\/td>\r\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\">Jobs requiring less than 12 hours walltime<\/td>\r\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\">12 hrs.<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\"><b>interactive<\/b><\/td>\r\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\">Jobs running in interactive mode<\/td>\r\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\">12 hrs.<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\"><b>long<\/b><\/td>\r\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\">Jobs requiring more than 144 hours walltime<\/td>\r\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\">2 weeks<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\"><b>gpu<\/b><\/td>\r\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\">Jobs running on GPU nodes<\/td>\r\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\">144 hrs.<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\"><b>private<\/b><\/td>\r\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\">Jobs using dedicated resources<\/td>\r\n<td style=\"padding: 5px;border: 1px solid #cccccc\" width=\"312\">unlimited<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n\u00a0\r\n\r\n<hr \/>\r\n\r\n<h2><span style=\"color: #221f72\">LSF<\/span><\/h2>\r\nMinerva uses LSF for batch submission. bsub is the submission command. Options can be put on the command line or in the submission script. HOWEVER, if the options are placed in the submission script, you must feed the script into the bsub command via stdin for the options to be read: E.g.,\r\n\r\ncat MyLSF.script | bsub\r\nor\r\nbsub < MyLSF.script\r\n\r\n\u00a0\r\n\r\n<strong>Some important points of interest:<\/strong>\r\n\r\n\u00a0\r\n<ul>\r\n \t<li>The default disposition for output and logs is for LSF to email the output to you. This piece is not working yet so you must use the \"-o\" option to save the output.<\/li>\r\n \t<li>In general, the shortest quantum of time in LSF is 1 minute. Wall time is expressed as HHH:MM -- There are no seconds. Durations are generally in minutes.<\/li>\r\n \t<li>System level checkpoints are supported by LSF. There are some \"gotchas\" ( E.g., the default method does not work on our system) so check with the SC staff if you need\/want to do checkpointing.<\/li>\r\n<\/ul>\r\nSome useful commands:\r\n\r\nbjobs - shows all your jobs in the queue\r\nbpeek - peek at your output before the job ends\r\nbqueues - what queues are available\r\nbkill - kill a job\r\n\r\n<span style=\"color: red\">Check out the main pages for all options<\/span>.","_et_gb_content_width":"","footnotes":""},"class_list":["post-965","page","type-page","status-publish","hentry"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/labs.icahn.mssm.edu\/minervalab\/wp-json\/wp\/v2\/pages\/965","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/labs.icahn.mssm.edu\/minervalab\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/labs.icahn.mssm.edu\/minervalab\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/labs.icahn.mssm.edu\/minervalab\/wp-json\/wp\/v2\/users\/415"}],"replies":[{"embeddable":true,"href":"https:\/\/labs.icahn.mssm.edu\/minervalab\/wp-json\/wp\/v2\/comments?post=965"}],"version-history":[{"count":34,"href":"https:\/\/labs.icahn.mssm.edu\/minervalab\/wp-json\/wp\/v2\/pages\/965\/revisions"}],"predecessor-version":[{"id":12942,"href":"https:\/\/labs.icahn.mssm.edu\/minervalab\/wp-json\/wp\/v2\/pages\/965\/revisions\/12942"}],"wp:attachment":[{"href":"https:\/\/labs.icahn.mssm.edu\/minervalab\/wp-json\/wp\/v2\/media?parent=965"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}