All use of Scientific Computing and Data resources in publications and presentations must acknowledge Mount Sinai’s CTSA award. Click here

Scientific Computing and Data / High Performance Computing / Minerva Quick Start

Minerva Quick Start Guide

Welcome to Minerva

Minerva is Mount Sinai’s high-performance computing (HPC) environment. It provides researchers with access to powerful computing resources for large-scale data analysis, simulations, artificial intelligence (AI), machine learning, and other computationally intensive research.

This guide introduces the basic concepts you need to get started, including:

  • Partitions – groups of computing resources designed for different types of workloads.
  • File systems – locations where you store programs, data, and results.
  • Jobs – programs or analyses submitted to Minerva for processing.
  • Queues – scheduling options that determine where and how your job runs.
  • Load Sharing Facility (LSF) – the job scheduling system used to submit and manage jobs.

Partitions

A partition is a collection of computing resources with specific hardware, capabilities, or access requirements. When you submit a job, you can specify the partition that best matches your computational needs. Think of partitions as different sections of a large computing facility. One may be optimized for general-purpose computing, another for GPU workloads, and another for projects with specific access requirements.

Minerva currently provides the following partitions:

The Chimera partition:

The Chimera partition provides general-purpose CPU computing, high-memory computing, and GPU resources.

  • 286 compute nodes – 48 Intel 8168 cores (2.7GHz) and 192 GB memory
  • 4x high memory nodes – 48 Intel 8168 cores (2.7GHz) and 1.5 TB memory
  • 48 V100 GPUs in 12 nodes – 32 Intel 6142 cores (2.6GHz) and 384 GB memory – 4x V100-16 GB GPU

 

The BODE2 partition:

The BODE2 partition provides additional CPU computing resources for users participating in the BODE program.

  • 78 compute nodes – 48 Intel 8268 cores (2.9GHz) and 192 GB memory
  • Only BODE-enabled users have access to the BODE2 partition

 

The CATS partition:

The CATS partition provides large-scale computing resources for eligible NIH-funded research projects.

  • 3,520 64-core 2.6 GHz Intel IceLake processors in 55 nodes
  • 1.5 TB of memory per node
  • 82.5 TB memory (collectively)
  • Eligible for all NIH-funded projects

Connecting to Minerva

For security, Minerva requires the Secure Shell (ssh) protocol and Microsoft Azure Multi-Factor Authentication (MFA).

Multi-Factor Authentication (MFA)

Minerva has transitioned from Symantec VIP to Microsoft Azure MFA. You can authenticate using one of the following methods:

  • App Push: A notification sent to the Microsoft Authenticator app on your mobile device.

  • Phone Text/Call: A one-time passcode sent via SMS or an automated phone call.

  • App Passcode: A rolling six-digit code generated within the Authenticator app.

Note: If you have not yet configured your MFA preferences, please visit the Microsoft MFA Setup Portal.

Connection Steps

  1. VPN: If you are off-campus, you must first connect to the Mount Sinai VPN.

  2. SSH Command: Open your terminal or an SSH client like PuTTY.

  3. Authentication: * Username: Use your standard Sinai credentials.

    • Password: Enter your Sinai password.

    • MFA Challenge: Complete the second-step verification using your chosen Azure MFA method.

SSH command: ssh your_userid@minerva.hpc.mssm.edu

For detailed instructions and troubleshooting, please refer to the official Logging In documentation.

 

 

File System

Minerva provides several different locations for storing files. Choosing the appropriate location is important because each file system has different storage limits, backup policies, and retention periods.

/hpc/users/<userid> User HOME directories. 30 GB quota.  It is NOT purged and is backed up.  Generally used for all the ‘rc’ and configuration files for various programs.
/sc/arion/work/<userid> A WORK directory for each user. 100 GB quota.  It is NOT purged and it is NOT backed up. To be used for whatever purpose the user desires.
/sc/arion/scratch/<userid>

A folder for each user inside the /sc/arion/scratch directory.

/sc/arion/scratch has a 100TB quota and it is shared by all users.  This should be used in lieu of /tmp for temporary files as well as short term storage up to a maximum of 14 days.  Files older than 14 days are purged automatically by the system.
/sc/arion/projects/<projectid>

PI’s can request project storage. Click here to submit an allocation request and renew annually.

A directory for each approved project. The quota is set to the approved allocation for the project.  It is NOT purged but it is NOT backed up. 

 

 

Queues

When you submit a job to Minerva, you are asking the system to run your computational task using available resources. A queue determines how your job is scheduled and what type of resources it can use. For example, a short analysis might use the express queue, while a GPU-based machine-learning job would use the gpu queue. Choosing the appropriate queue helps your job run efficiently and prevents you from requesting resources you do not need.

The queues that are available are: Default memory per core is set as 3000MB for all the queues.

Queue Description Max Walltime (maximum amount of time that a job is allowed to run.)
Premium Jobs requesting high priority with APS doubled as 200. Charged at 150% of alloc rate 144 hrs.
express Jobs requiring less than 12 hours walltime 12 hrs.
interactive Jobs running in interactive mode for CPU or GPU computing 12 hrs.
long Jobs requiring more than 144 hours walltime 2 weeks
gpu Jobs running on GPU nodes 144 hrs.
gpuexpress Jobs runnning on GPU nodes requiring less than 15 hours walltime 15 hrs.
private Jobs using dedicated resources unlimited

 

 

LSF

Minerva uses LSF to schedule and manage computational jobs. Instead of running a large analysis directly on the login node, you submit it to LSF. LSF determines when and where the job will run based on the resources you requested and the availability of the system.

Minerva uses LSF for batch submission. bsub is the submission command. Options can be put on the command line or in the submission script. HOWEVER, if the options are placed in the submission script, you must feed the script into the bsub command via stdin for the options to be read: E.g.,

cat MyLSF.script | bsub
or
bsub < MyLSF.script

 

Some important points of interest:

  • The default disposition for output and logs is for LSF to email the output to you. This piece is not working yet so you must use the “-o” option to save the output.
  • In general, the shortest quantum of time in LSF is 1 minute. Wall time is expressed as HHH:MM — There are no seconds. Durations are generally in minutes.
  • System level checkpoints are supported by LSF. There are some “gotchas” (E.g., the default method does not work on our system) so check with the SC staff if you need/want to do checkpointing.

Some useful commands:

bjobs – shows all your jobs in the queue
bpeek – peek at your output before the job ends
bqueues – what queues are available
bkill – kill a job

Click here for additional Minerva documentation.

 

Acknowledging Mount Sinai in Your Work

Utilizing S10 BODE and CATS partitions requires acknowledgements of support by NIH in your publications. To assist, we have provided exact wording of acknowledgements required by NIH for your use. Click here for acknowledgements.

Supported by grant UL1TR004419 from the National Center for Advancing Translational Sciences, National Institutes of Health.