All use of Scientific Computing and Data resources in publications and presentations must acknowledge Mount Sinai’s CTSA funding award. Click here

Scientific Computing and Data / High Performance Computing / Documentation / Minerva Restricted Cluster

Minerva Restricted Cluster

The Minerva Restricted Cluster  (Minerva-Res) is a hardened, high-performance computing (HPC) cluster and shared-use resource for all qualified researchers at Mount Sinai. It is designed and operated in accordance with NIST SP 800-171 Rev. 3 security requirements. It is dedicated for genomic data coming from the NIH Controlled-Access repository. NIH does not consider controlled access genomic data to be CUI but is using the NIST 800-171 security controls as a best practice for data protection. We reference NIH supported projects involving genomic data sharing as CUI-like data protections.  It is an encrypted, tightly controlled compute environment that includes a variety of standard compute nodes and an encrypted shared filesystem.

Access the Cluster

Projects

Access to Minerva-Res is granted strictly on a per-project basis. Principal Investigators (PIs) may manage multiple projects. User accounts can only be created after a project has been formally requested and approved. Continued access to any project requires an active Data Use Agreement (DUA) managed by the institution GCO.

Accounts

All accounts must be associated with an active, approved project. Users may only access data that is specifically required for your approved research.

Please Note: We don’t support external collaboration accounts currently due to security requirements.

Accounts will be deactivated when any of the following occurs:

  • The account is no longer associated with an active project
  • Employment or affiliation with the institution ends
  • A policy or DUA violation occurs
  • Project completion

Log In

Note: Access to Minerva-Res requires connection from the campus network or VPN (virtual private network).

The login nodes are connected to the campus network allowing access only on campus or via tunneling over Mount Sinai’s VPN if off campus.

There are currently two login nodes available for general use, ci06e01 and ci06e02. You may connect to one of them through one of two round-robin Domain Name Server (DNS) load balancing names or you may specify one of them explicitly, if you prefer one over the other. For example, if you have a disconnected screen session running on one of the nodes, you will want to log onto that particular node if you want to reconnect.

The addresses of the nodes are:

  • minerva-res.hpc.mssm.edu – round-robin redirect (recommended)
  • ci06e01.hpc.mssm.edu – specific login node
  • ci06e02.hpc.mssm.edu – specific login node

You can login to the Minerva-Res cluster via ssh to minerva-res.hpc.mssm.edu.

ssh yourUserID@minerva-res.hpc.mssm.edu
(yourUserID@minerva.hpc.mssm.edu) Password: <Enter your Sinai School Password Only>
Enter Your Microsoft verification code<609017>

Authenticate using your institutional (school or hospital) credentials and complete Microsoft Azure MFA (multi-factor authentication) on your mobile device.

Transfer Data

Data may only be transferred to or from Minerva-Res using an approved method. Standard SFTP and routine SSH file transfers are disabled by default. All transfers are logged, and users remain responsible for following restrictions applicable to their data.

Approved Transfers

To transfer data into or out of the cluster, a completed and approved DUA is required. Once approved, specific source addresses or subnets will be permitted. Downloading, copying, or transferring data to unauthorized systems is a policy violation.

Derived Data

Any results or data users generate from source data must be handled with the same security standards as the original source. Treat all derived outputs as sensitive.

 

Software Environment

Software is available through the read-only module system at /hpc/packages. Use the module command to load available applications. 

module load <software_name>

For complete details, please visit our documentation on Software Environment: Lmod (https://labs.icahn.mssm.edu/minervalab/documentation/software-environment-lmod/).

To request additional software, send in an HPC ticket hpchelp@hpc.mssm.edu.

Running Jobs (LSF Scheduler)

Workloads are managed by the IBM Spectrum LSF scheduler. Do not run computationally intensive tasks directly on the login nodes — all jobs must be submitted to the scheduler.

Available queues: night, short, normal, interactive

For interactive work or exploratory analyses, use the interactive queue. For most batch jobs, use normal or short depending on expected runtime.

For additional information, please visit our documentation on Load Sharing Facility (LSF) Job Scheduler (https://labs.icahn.mssm.edu/minervalab/documentation/lsf-job-scheduler/).

 

Open OnDemand

A dedicated Open OnDemand portal (https://ondemand-res.hpc.mssm.edu/) provides browser-based access to applications including Desktop, SAS, Stata, Jupyter and RStudio.

Note: CMS projects are currently unavailable through File Explorer and will be made available as soon as possible. Data transfer (upload or download) is disabled in Open OnDemand Restricted environment.

For additional information, please visit our documentation on Load Sharing Facility (LSF) Job Scheduler (https://labs.icahn.mssm.edu/minervalab/documentation/open-ondemand/).

Storage

Minerva-Res uses an encrypted-at-rest filesystem (GPFS) for all user data. Scratch and work directories available on standard clusters are not available in this environment.

You can check your current usage and quota by running:

df -h /arionencrypt/projects/<projectid>

or for CMS data,

df -h /arionencrypt/cmsdata/<projectid>

Storage Type Path Quota Notes
Home Directory /arionencrypt/users/<userid> 30 GiB, 1M files Encrypted-at-rest (GPFS)
Project Data /arionencrypt/projects/<projectid> Per project allocation Encrypted-at-rest (GPFS)
CMS Project Data /arionencrypt/cmsdata/<projectid> Per project allocation Encrypted-at-rest (GPFS)
Local Scratch /tmp or /local Variable LUKS-encrypted on compute nodes

 

Long-Term Data Archive: TSM (Tivoli Storage Manager)

TSM clients are available on the login nodes. 

For complete details, please visit our documentation on Long-Term Data Archive: TSM (https://labs.icahn.mssm.edu/minervalab/documentation/access-tsm-with-command-line/).

 

Network & Internet Access

Compute nodes (currently CPU only) have no general campus or internet connectivity. If internet access is required for a workflow, it must be routed through a pre-approved proxy to a whitelisted destination. Circumventing network controls is strictly prohibited.

Compute nodes communicate over a private InfiniBand network for high-speed inter-node data transfer.

Unsupported Services

The following services are not supported on Minerva-res. Please plan your workflows accordingly:

  • Databases
  • Globus
  • Posit Connect
  • Web servers
  • On-the-fly scripts (Jupyter, RStudio IDE, Ollama)

Security & Compliance

Minerva-Res is a strictly monitored environment designed to protect sensitive data and maintain NIST SP 800-171 compliance. The environment enforces the following restrictions and monitoring protocols beyond a standard HPC environment:

  • Users must only access data specifically required for the approved research (minimum necessary principle).
  • Bypassing security controls, creating unauthorized network tunnels, or using unapproved proxies is forbidden.
  • Sessions that remain idle will be automatically terminated after 12 hours (Session Monitoring).
  • All command activity is logged to central security platforms (Graylog/SIEM) for security auditing and system health monitoring (Command Auditing).

Users are expected to maintain compliance at all times. Violations may result in immediate access revocation and escalation per institutional policy.