All use of Scientific Computing and Data resources in our publications and presentations must acknowledge CTSA. Click here

Scientific Computing and Data / High Performance Computing / Hardware and Technical Specs

Hardware and Technical Specs

  • The primary asset for Scientific Computing is the supercomputer, Minerva.
  • The HPC resource, Minerva, was created in 2012 and has been upgraded several times, most recently in Nov. 2024.
  • It utilizes 24,912 Intel Platinum in different generations including 8568Y+ 2.3GHz, 8358 2.6 GHz, and 8268 2.9 GHz compute cores (96 cores or 64 cores or 48 cores per node with two sockets in each node) with 1.5TB of memory per node, 196 H100 GPUs, 32 L40S, 40 A100 GPUs, 48 V100 GPUs, 440 terabytes of total memory, 32 petabytes of spinning storage accessed via IBM’s Spectrum Scale/General Parallel File System (GPFS) for a total of > 2 petaflops of CPU compute power and ~ 8 petaflops for GPU compute power.
  • Minerva has contributed to over 1,700 peer-reviewed publications since 2012.
  • Minerva cluster design is driven by the research demand performed by Minerva users (i.e. the number of nodes, the amount of memory per node, and the amount of disk space for storage).

 

The following diagram shows the overall Minerva configuration.

Compute Nodes

Chimera Partition

Added in Nov. 2024. Nodes purchased prior to 2024 and integrated to new NDR network via HDR 100Gb/s.
  • 4 login nodes – Intel Emerald Rapids 8568Y+, 2.3GHz – 96 cores with 512 GB memory per node.
  • 146 compute nodes* – Intel Emerald Rapids 8568Y+, 2.3GHz– 96 cores with 1.5 TB memory per node.
    • 14,016 cores in total.
  • 188 H100 in 47 nodes – Intel ER 8568Y+, 2.3GHz– 96 cores with 1.5 TB memory per node.
    • 4 x H100-80GB(SXM5) NVLinked GPUs per node.
  • 32 L40s GPUs in 4 nodes – AMD Genoa 9334 2.7GHz – 64 cores with 1.5TB memory per node.
    • 8x L40s-48GB GPUs per node. L40s doesn’t support FP64.
  • 3.84 TB Local NVME SSD (3.5TB usable) per node.
    • It can deliver a sustained read-write speed of 3.5 GB/s in contrast with SATA SSDs that limit at 600 MB/s.
  • NDR InfiniBand fat tree fabric networking (400Gb/s).
    • 6 service nodes.
    • 295.5 TB memory in total.
    • Direct water-cooling solution.
    • New NFS storage (for users’ home directories) –140 TB usable.
  • 33 high memory nodes – Intel 8268 24C, 2.9GHZ – 1.5 TB memory.
  • 48 V100 GPUs in 12 node – Intel 6142 16C, 2.6GHz – 384 GB memory – 4x V100-16 GB GPU.
  • 32 A100 GPUs in 8 nodes – Intel 8268 24C, 2.9GHz – 384 GB memory – 4x A100-40 GB GPU.  1.92TB SSD (1.8 TB usable) per node.
  • 8 A100 GPUs in 2 nodes – Intel 8358 32C, 2.6GHz – 2 TB memory – 4x A100-80 GB GPU.
    • A100 is connected via NVLink.
    • 7.68 TB NVMe SSD (7.0TB usable) per node.
  • 8 H100 GPUs in 2 nodes – Intel 8358 32C, 2.6 GHz – 0.5 TB memory – 4xH100-80GB GPU.
    • 3.84 TB NVMe SSD (3.5 TB usable) per node.
  • [Decommissioned on Nov. 5th 2024] 4 login nodes – Intel Xeon(R) Platinum 8168 24C, 2.7GHz – 384 GB memory.
  • [Decommissioned on July 17th and Nov. 5th 2024] 275 compute nodes* – Intel 8168 24C, 2.7GHz – 192 GB memory.
    • 13,152 cores (48 cores per node).
  • *Compute Node —where you run your applications. Users do not have direct access to these machines. Access is managed through the LSF job scheduler.

 

 
  •  

BODE2 Partition

CATS Partition

Private Nodes
[Decommissioned on July 17th and Nov. 5th, 2024].

$2M S10 BODE2 awarded by NIH (Kovatch PI).

  • 3,744 48-core 2.9 GHz Intel Cascade Lake 8268 processors in 78 nodes.

$2M CATS awarded by NIH (Kovatch PI).

  • 3,520 cores in 55 nodes- Intel IceLake 8358, 26GHz – 64 cores with 1.5 TB memory per node.
  • 82.5 TB memory (collectively).
  • Open to eligible NIH funded projects.
  • Purchased by private groups.
  • Hosted on Minerva.

 

 

 

 

Summary

Total system memory (computes + GPU) = 440 TB  Total number of cores (computes + GPU) = 24,912 cores
CPU Peak performance of all nodes = > 2 PFLOPS H100 Peak performance based FP64 Tensor cores = 12.5 PFLOPS.
Max performance from HPL LINPACK run is 7.9 PFLOPS. Private nodes are not counted in the calculation.

 

Acknowledging Mount Sinai in Your Work

Utilizing S10 BODE and CATS partitions requires acknowledgements of support by NIH in your publications. To assist, we have provided exact wording of acknowledgements required by NIH for your use. Click here for acknowledgements.

 

File System Storage

For Minerva, we focused on parallel file systems because NFS and other file systems simply cannot scale to the number of nodes or provide performance for the sheer number of files that the genomics workload entails. Specifically, Minerva is using IBM’s General Parallel File System (GPFS) because it has advantages that are specifically useful for this workload such as parallel metadata, tiered storage, and sub-block allocation. Metadata is the information about the data in the file system. The flash storage is utilized to hold the metadata and tiny files for fast access.

Currently we have one parallel file system on Minerva, Arion, which users can access at /sc/arion. The Hydra file system was retired at the end of 2020.

GPFS Name Lifetime Storage Type Raw PB Usable PB
Arion 2019 – Lenovo DSS 14 9.6
Arion 2019 – Lenovo G201 flash 0.12 0.12
Arion 2020 – Lenovo DSS 16 11.2
Arion 2021 – Lenovo DSS 16 11.2
    Total 46 32

Supported by grant UL1TR004419 from the National Center for Advancing Translational Sciences, National Institutes of Health.