[\/et_pb_text][\/et_pb_column][\/et_pb_row][\/et_pb_section][et_pb_section fb_built=”1″ _builder_version=”3.22″][et_pb_row _builder_version=”3.25″ background_size=”initial” background_position=”top_left” background_repeat=”repeat”][et_pb_column type=”4_4″ _builder_version=”3.25″ custom_padding=”|||” custom_padding__hover=”|||”][et_pb_text admin_label=”Hardware and Specs” _builder_version=”4.9.0″ header_font=”|600|||||||” header_text_color=”#221f72″ header_2_text_color=”#221f72″ header_2_font_size=”24px” background_size=”initial” background_position=”top_left” background_repeat=”repeat” hover_enabled=”0″ sticky_enabled=”0″]<\/p>\n
Hardware and Technical Specs<\/h1>\n
<\/p>\n
The Minerva supercomputer is maintained by High Performance Computing (HPC). Minerva was created in 2012 and has been upgraded several times (most recently in Nov. 2024) and has over 11 petaflops of compute power. It consists of 24,912 Intel Platinum processors in different generations including 2.3 GHz, 2.6 GHz, and 2.9 GHz compute cores (96 cores or 64 cores or 48 cores per node with two sockets in each node) with 1.5 terabytes (TB) of memory per node, 356 graphics processing units (GPUs), including 236 Nvidia H100 GPUs, 32 Nvidia L40S servers, 40 Nvidia A100 GPUs, 48 Nvidia V100 GPUs, 440 TB of total memory, and 32 petabytes of spinning storage accessed via IBM\u2019s Spectrum Scale\/General Parallel File System (GPFS). Minerva has contributed to over 1,900 peer-reviewed publications since 2012. The Minerva cluster design is driven by the research demand performed by Minerva users (i.e. the number of nodes, the amount of memory per node, and the amount of disk space for storage).<\/p>\n
<\/p>\n
The following diagram shows the overall Minerva configuration.<\/p>\n
<\/p>\n
![\"\"](\"https:\/\/labs.icahn.mssm.edu\/minervalab\/wp-content\/uploads\/sites\/342\/2022\/06\/minerva-diagram-2022.png\")
<\/p>\n
<\/p>\n
Compute Nodes<\/h2>\nChimera Partition<\/strong><\/span><\/h4>\n\n- 4 login nodes <\/b>\u2013 Intel Xeon(R) Platinum 8168 24C, 2.7GHz \u2013 384 GB memory<\/li>\n
- 275 compute nodes* <\/b>\u2013 Intel 8168 24C, 2.7GHz \u2013 192 GB memory\n
\n- 13,152 cores (48 per node (2 sockets\/node))<\/li>\n<\/ul>\n<\/li>\n
- 37 high memory nodes<\/b> \u2013 Intel 8168\/8268 24C, 2.7GHz\/2.9GHZ \u2013 1.5 TB memory<\/li>\n
- 48 V100 GPUs in 12 nodes<\/b> \u2013 Intel 6142 16C, 2.6GHz \u2013 384 GB memory \u2013 4x V100-16 GB GPU<\/li>\n<\/ul>\n
\n- 32 A100 GPUs in 8 nodes <\/strong>\u2013 Intel 8268 24C, 2.9GHz \u2013 384 GB memory \u2013 4x A100-40 GB GPU\n
\n- 1.92TB SSD (1.8 TB usable) per node<\/li>\n<\/ul>\n<\/li>\n
- 8 A100 GPUs in 2 nodes<\/strong>\u00a0\u2013 Intel 8358 32C, 2.6GHz \u2013 2 TB memory \u2013 4x A100-80 GB GPU\n
\n- 7.68<\/b>\u00a0<\/b>TB NVMe PCIe SSD\u00a0<\/b>\u00a0(7.0TB usable) per node, which can deliver a sustained read-write speed of 3.5 GB\/s in contrast with SATA SSDs that limit at 600 MB\/s<\/li>\n
- The A100 is connected via NVLink<\/li>\n<\/ul>\n<\/li>\n
- 10 gateway nodes<\/li>\n
- New NFS storage<\/b> (for users home directories) \u2013 192 TB raw \/ 160 TB usable RAID6<\/li>\n
- Mellanox EDR InfiniBand<\/b> fat tree fabric (100Gb\/s)<\/li>\n<\/ul>\n
*Compute Node<\/em>\u00a0\u2014where you run your applications. Users do not have direct access to these machines. Access is managed through the LSF job scheduler.<\/p>\nBODE2 Partition<\/strong><\/span><\/h4>\n$2M S10 BODE2 awarded by NIH (Kovatch PI)<\/p>\n
\n- 3,744 48-core 2.9 GHz Intel Cascade Lake 8268 processors in 78 nodes<\/li>\n
- 192 GB of memory per node<\/li>\n
- 240 GB of SSDs per node<\/li>\n
- 15 TB memory (collectively)<\/li>\n
- Open to all NIH funded projects<\/li>\n<\/ul>\n
CATS Partition<\/strong><\/span><\/h4>\n$2M CATS awarded by NIH (Kovatch PI)<\/p>\n
\n- 3,520 64-core 2.6 GHz Intel IceLake 8358 processors in 55 nodes<\/li>\n
- 1.5 TB<\/strong> of memory per node<\/li>\n
- 82.5 TB memory (collectively)<\/li>\n
- Open to eligible NIH funded projects<\/li>\n<\/ul>\n
Private Nodes<\/strong><\/span><\/h4>\nPurchased by private groups and hosted on Minerva.<\/p>\n
In summary,<\/p>\n
Total system memory <\/b>(computes + GPU + high mem) = <\/span>210\u00a0TB<\/strong><\/p>\nTotal number of cores <\/b>(computes + GPU + high mem) = <\/span>24,214\u00a0cores<\/strong><\/p>\nPeak performance <\/b>(computes + GPU + high mem, CPU only) = <\/span>2\u00a0PFLOPS<\/strong><\/p>\nFile System Storage<\/h2>\n
For Minerva, we focused on parallel file systems because NFS and other file systems simply cannot scale to the number of nodes or provide performance for the sheer number of files that the genomics workload entails. Specifically, Minerva is using IBM’s General Parallel File System (GPFS) because it has advantages that are specifically useful for this workload such as parallel metadata, tiered storage, and sub-block allocation. Metadata is the information about the data in the file system. The flash storage is utilized to hold the metadata and tiny files for fast access. <\/span><\/span><\/p>\nCurrently we have one parallel file system on Minerva, Arion, which users can access at \/sc\/arion. The Hydra file system was retired at the end of 2020.<\/span><\/p>\n\n\n \n\nGPFS Name<\/span><\/strong><\/td>\nLifetime<\/span><\/strong><\/td>\nStorage Type<\/span><\/strong><\/td>\nRaw PB<\/span><\/strong><\/td>\nUsable PB<\/span><\/strong><\/td>\n<\/tr>\n\nArion<\/td>\n 2019 –<\/td>\n Lenovo DSS<\/td>\n 14<\/td>\n 9.6<\/td>\n<\/tr>\n \nArion<\/td>\n 2019 –<\/td>\n Lenovo G201 flash<\/td>\n 0.12<\/td>\n 0.12<\/td>\n<\/tr>\n \nArion<\/td>\n 2020 –<\/td>\n Lenovo DSS<\/td>\n 16<\/td>\n 11.2<\/td>\n<\/tr>\n \nArion<\/td>\n 2021 –<\/td>\n Lenovo DSS<\/td>\n 16<\/td>\n 11.2<\/td>\n<\/tr>\n \n\u00a0<\/td>\n \u00a0<\/td>\n Total<\/strong><\/td>\n46<\/td>\n 32<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\nSupported by grant UL1TR004419 from the National Center for Advancing Translational Sciences, National Institutes of Health.<\/p>\n
Acknowledging Mount Sinai in Your Work<\/h2>\n
- \n
- 4 login nodes <\/b>\u2013 Intel Xeon(R) Platinum 8168 24C, 2.7GHz \u2013 384 GB memory<\/li>\n
- 275 compute nodes* <\/b>\u2013 Intel 8168 24C, 2.7GHz \u2013 192 GB memory\n
- \n
- 13,152 cores (48 per node (2 sockets\/node))<\/li>\n<\/ul>\n<\/li>\n
- 37 high memory nodes<\/b> \u2013 Intel 8168\/8268 24C, 2.7GHz\/2.9GHZ \u2013 1.5 TB memory<\/li>\n
- 48 V100 GPUs in 12 nodes<\/b> \u2013 Intel 6142 16C, 2.6GHz \u2013 384 GB memory \u2013 4x V100-16 GB GPU<\/li>\n<\/ul>\n
- \n
- 32 A100 GPUs in 8 nodes <\/strong>\u2013 Intel 8268 24C, 2.9GHz \u2013 384 GB memory \u2013 4x A100-40 GB GPU\n
- \n
- 1.92TB SSD (1.8 TB usable) per node<\/li>\n<\/ul>\n<\/li>\n
- 8 A100 GPUs in 2 nodes<\/strong>\u00a0\u2013 Intel 8358 32C, 2.6GHz \u2013 2 TB memory \u2013 4x A100-80 GB GPU\n
- \n
- 7.68<\/b>\u00a0<\/b>TB NVMe PCIe SSD\u00a0<\/b>\u00a0(7.0TB usable) per node, which can deliver a sustained read-write speed of 3.5 GB\/s in contrast with SATA SSDs that limit at 600 MB\/s<\/li>\n
- The A100 is connected via NVLink<\/li>\n<\/ul>\n<\/li>\n
- 10 gateway nodes<\/li>\n
- New NFS storage<\/b> (for users home directories) \u2013 192 TB raw \/ 160 TB usable RAID6<\/li>\n
- Mellanox EDR InfiniBand<\/b> fat tree fabric (100Gb\/s)<\/li>\n<\/ul>\n
*Compute Node<\/em>\u00a0\u2014where you run your applications. Users do not have direct access to these machines. Access is managed through the LSF job scheduler.<\/p>\n
BODE2 Partition<\/strong><\/span><\/h4>\n
$2M S10 BODE2 awarded by NIH (Kovatch PI)<\/p>\n
- \n
- 3,744 48-core 2.9 GHz Intel Cascade Lake 8268 processors in 78 nodes<\/li>\n
- 192 GB of memory per node<\/li>\n
- 240 GB of SSDs per node<\/li>\n
- 15 TB memory (collectively)<\/li>\n
- Open to all NIH funded projects<\/li>\n<\/ul>\n
CATS Partition<\/strong><\/span><\/h4>\n
$2M CATS awarded by NIH (Kovatch PI)<\/p>\n
- \n
- 3,520 64-core 2.6 GHz Intel IceLake 8358 processors in 55 nodes<\/li>\n
- 1.5 TB<\/strong> of memory per node<\/li>\n
- 82.5 TB memory (collectively)<\/li>\n
- Open to eligible NIH funded projects<\/li>\n<\/ul>\n
Private Nodes<\/strong><\/span><\/h4>\n
Purchased by private groups and hosted on Minerva.<\/p>\n
In summary,<\/p>\n
Total system memory <\/b>(computes + GPU + high mem) = <\/span>210\u00a0TB<\/strong><\/p>\n
Total number of cores <\/b>(computes + GPU + high mem) = <\/span>24,214\u00a0cores<\/strong><\/p>\n
Peak performance <\/b>(computes + GPU + high mem, CPU only) = <\/span>2\u00a0PFLOPS<\/strong><\/p>\n
File System Storage<\/h2>\n
For Minerva, we focused on parallel file systems because NFS and other file systems simply cannot scale to the number of nodes or provide performance for the sheer number of files that the genomics workload entails. Specifically, Minerva is using IBM’s General Parallel File System (GPFS) because it has advantages that are specifically useful for this workload such as parallel metadata, tiered storage, and sub-block allocation. Metadata is the information about the data in the file system. The flash storage is utilized to hold the metadata and tiny files for fast access. <\/span><\/span><\/p>\n
Currently we have one parallel file system on Minerva, Arion, which users can access at \/sc\/arion. The Hydra file system was retired at the end of 2020.<\/span><\/p>\n
\n
\n \n<\/colgroup>\n \n GPFS Name<\/span><\/strong><\/td>\n Lifetime<\/span><\/strong><\/td>\n Storage Type<\/span><\/strong><\/td>\n Raw PB<\/span><\/strong><\/td>\n Usable PB<\/span><\/strong><\/td>\n<\/tr>\n \n Arion<\/td>\n 2019 –<\/td>\n Lenovo DSS<\/td>\n 14<\/td>\n 9.6<\/td>\n<\/tr>\n \n Arion<\/td>\n 2019 –<\/td>\n Lenovo G201 flash<\/td>\n 0.12<\/td>\n 0.12<\/td>\n<\/tr>\n \n Arion<\/td>\n 2020 –<\/td>\n Lenovo DSS<\/td>\n 16<\/td>\n 11.2<\/td>\n<\/tr>\n \n Arion<\/td>\n 2021 –<\/td>\n Lenovo DSS<\/td>\n 16<\/td>\n 11.2<\/td>\n<\/tr>\n \n \u00a0<\/td>\n \u00a0<\/td>\n Total<\/strong><\/td>\n 46<\/td>\n 32<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n Supported by grant UL1TR004419 from the National Center for Advancing Translational Sciences, National Institutes of Health.<\/p>\n
Acknowledging Mount Sinai in Your Work<\/h2>\n
- 7.68<\/b>\u00a0<\/b>TB NVMe PCIe SSD\u00a0<\/b>\u00a0(7.0TB usable) per node, which can deliver a sustained read-write speed of 3.5 GB\/s in contrast with SATA SSDs that limit at 600 MB\/s<\/li>\n
- 48 V100 GPUs in 12 nodes<\/b> \u2013 Intel 6142 16C, 2.6GHz \u2013 384 GB memory \u2013 4x V100-16 GB GPU<\/li>\n<\/ul>\n
- 275 compute nodes* <\/b>\u2013 Intel 8168 24C, 2.7GHz \u2013 192 GB memory\n
![\"\"](\"https:\/\/labs.icahn.mssm.edu\/minervalab\/wp-content\/uploads\/sites\/342\/2021\/08\/Minerva-Configuration-08-2021-scaled.gif\")
<\/p>