Minerva Restricted Cluster

1. Submit the Minerva Restricted Cluster: Project Resource Allocation Request Form.
2. Review and acknowledge the Annual Minerva Restricted Supercomputer Policies and Procedures. Once logged in, navigate to Minerva-Res Forms and select Minerva Restricted Supercomputer Policies and Procedures.

• Existing Access: You must hold an active Minerva account. If you do not have one, please submit a request here.
• Policy Compliance: You must review and formally agree to the Annual Minerva Restricted Supercomputer Policies and Procedures. Once logged in, navigate to Minerva-Res Forms and select Minerva Restricted Supercomputer Policies and Procedures.
• PI Authorization: Your request will generate an email notification to your Principal Investigator, who must explicitly approve your access.

Check Quota Commands

df -h /arionencrypt/projects/<projectid>

# Or for CMS data:
df -h /arionencrypt/cmsdata/<projectid>

Storage Types

There are currently two login nodes available for general use: ci06e01 and ci06e02. You may connect to one of them through one of two round-robin Domain Name Server (DNS) load-balancing names, or you may specify one explicitly if you prefer (e.g., to reconnect to a disconnected screen session).

Node Addresses

• Recommended (Round-Robin): minerva-res.hpc.mssm.edu
• Specific Node 1: ci06e01.hpc.mssm.edu
• Specific Node 2: ci06e02.hpc.mssm.edu

Login Command Example

ssh yourUserID@minerva-res.hpc.mssm.edu

(yourUserID@minerva.hpc.mssm.edu) Password: <Enter your Sinai School Password Only>

Enter Your Microsoft verification code<609017>

• Password: Enter your Sinai School Password only.
• MFA: Complete your Microsoft Azure multi-factor authentication when prompted on your mobile device.

For complete details, please visit the Minerva’s Authentication.

Software is available through the read-only module system at /hpc/packages. Use the module command to load available applications:

module load <software_name>

For complete details, please visit the Software Environment: Lmod Documentation. To request additional software, send a ticket to hpchelp@hpc.mssm.edu.

Workloads are managed by the IBM Spectrum LSF scheduler. Do not run computationally intensive tasks directly on the login nodes—all jobs must be submitted to the scheduler.

Important Differences from Standard Minerva

Unlike the standard Minerva cluster, the LSF configuration on Minerva-Res is restricted to maintain security and compliance:

• Available Queues: Only the express and premium queues are available. Interactive and long queues, which are available on the standard Minerva cluster, are not available on Minerva-Res.
• GPU Resources: There are no GPU queues or GPU-accelerated resources available on Minerva-Res.

For standard non-controlled workloads that require interactive queues, long runtimes, or GPU access, please use the standard Minerva cluster.

For complete details, please visit the Load Sharing Facility (LSF) Job Scheduler Documentation.

A dedicated portal at ondemand-res.hpc.mssm.edu provides browser-based access to applications including Desktop, SAS, Stata, Jupyter, and RStudio.

For complete details, please visit Open OnDemand Documentation.

All data transfer to and from Minerva-Res must use an approved method.

• Standard SFTP/SSH and general internet access are disabled by default
• Only whitelisted proxy sites are permitted, for defined periods
• All transfers are logged
• Users remain responsible for DUA-mandated restrictions on their data

Approval requirements

Non-Controlled Data

Reviewed by the PI and confirmed free of NIH/CMS controlled-access or participant-level genomic data.

• In:
  • Stage files in /sc/arion/projects/<project>/from-arion-to-arionencrypt/<userid> on Arion
  • Submit a data transfer request
  • HPC transfers to Minerva-Res staging
  • Move files into your working directory
• Out:
  • Stage files in /arionencrypt/projects/<project_res>/from-arionencrypt-to-arion
  • Submit a data transfer request
  • PI approves
  • HPC transfers to Arion staging
  • Move files into your working directory

Controlled Data

• In: Direct download from approved repositories only.
  • Requires External Connectivity Request Form (justification + source IP/subnet)
  • Security review: 2–3 weeks
  • Approved IPs are allow-listed for the transfer window only, then removed
  • Access is one-time only — notify HPC once no longer needed; re-request for future transfers
• Out: Prohibited unless expressly authorized under the applicable DUA and institutional review (contact hpchelp@hpc.mssm.edu).

For complete details, please visit the Data Transfer Guide for Minerva-Res.

Login and compute nodes (currently CPU only) have no general campus or internet connectivity. If internet access is required for a workflow, it must be routed through a whitelisted destination for a designated timeframe. Circumventing network controls is strictly prohibited.

• External network access: If external connectivity is required, please follow these steps:
  • Complete the External Connectivity Request Form and provide the required information, including the justification for the request and the external IP address or subnet that needs access.
  • The HPC team will review the request and evaluate the research justification. Additional information may be requested if needed. All requests are subject to security review and approval.
  • Following approval, the requested IP source will be whitelisted for the designated timeframe.
• Access Types & Durations:
  • One-time access: Access will be removed 7 days after it is granted. Users will be notified of the scheduled removal.
  • Weekly or monthly access: Access will remain in place until the next quarterly internal review by the Minerva team.
  • Annual access: Access will be removed 7 days after the approved activity is completed.
  • Other use cases: Access duration will be determined on a case-by-case basis.
• Whitelisted URLs: To access whitelisted URLs, use the command module load res_proxies.
• Inter-node Communication: Compute nodes communicate over a private InfiniBand network for high-speed data transfer.

TSM clients can be accessed directly on Minerva-Res login nodes. For complete details, please visit Access TSM with Command Line Documentation.

The following services are not supported on Minerva-Res:

• Databases
• Globus
• Posit Connect
• Web servers
• On-the-fly scripts (Jupyter, RStudio IDE, Ollama)

Minerva-Res is a strictly monitored environment designed to protect sensitive data and maintain NIST SP 800-171 compliance. The environment enforces strict restrictions and monitoring protocols beyond standard HPC settings:

• Minimum Necessary Principle: Users must only access data specifically required for their approved research.
• Network Tunnels: Bypassing security controls, creating unauthorized network tunnels, or using unapproved proxies is forbidden.
• Session Monitoring: Sessions that remain idle will be automatically terminated after 12 hours.
• Command Auditing: All command activity is logged to central security platforms (Graylog/SIEM) for security auditing and system health monitoring.

Violations of compliance may result in immediate access revocation and escalation per institutional policy.