[\/et_pb_text][\/et_pb_column][\/et_pb_row][\/et_pb_section][et_pb_section bb_built=”1″ _builder_version=”3.22″ prev_background_color=”#000000″][et_pb_row _builder_version=”3.25″ background_size=”initial” background_position=”top_left” background_repeat=”repeat”][et_pb_column type=”4_4″ _builder_version=”3.25″ custom_padding=”|||” custom_padding__hover=”|||”][et_pb_text admin_label=”Hardware and Specs” _builder_version=”4.27.4″ header_font=”|600|||||||” header_text_color=”#221f72″ header_2_text_color=”#221f72″ header_2_font_size=”24px” background_size=”initial” background_position=”top_left” background_repeat=”repeat” hover_enabled=”0″ sticky_enabled=”0″ background_pattern_color=”rgba(0,0,0,0.2)” background_mask_color=”#ffffff” text_text_shadow_horizontal_length=”text_text_shadow_style,%91object Object%93″ text_text_shadow_horizontal_length_tablet=”0px” text_text_shadow_vertical_length=”text_text_shadow_style,%91object Object%93″ text_text_shadow_vertical_length_tablet=”0px” text_text_shadow_blur_strength=”text_text_shadow_style,%91object Object%93″ text_text_shadow_blur_strength_tablet=”1px” link_text_shadow_horizontal_length=”link_text_shadow_style,%91object Object%93″ link_text_shadow_horizontal_length_tablet=”0px” link_text_shadow_vertical_length=”link_text_shadow_style,%91object Object%93″ link_text_shadow_vertical_length_tablet=”0px” link_text_shadow_blur_strength=”link_text_shadow_style,%91object Object%93″ link_text_shadow_blur_strength_tablet=”1px” ul_text_shadow_horizontal_length=”ul_text_shadow_style,%91object Object%93″ ul_text_shadow_horizontal_length_tablet=”0px” ul_text_shadow_vertical_length=”ul_text_shadow_style,%91object Object%93″ ul_text_shadow_vertical_length_tablet=”0px” ul_text_shadow_blur_strength=”ul_text_shadow_style,%91object Object%93″ ul_text_shadow_blur_strength_tablet=”1px” ol_text_shadow_horizontal_length=”ol_text_shadow_style,%91object Object%93″ ol_text_shadow_horizontal_length_tablet=”0px” ol_text_shadow_vertical_length=”ol_text_shadow_style,%91object Object%93″ ol_text_shadow_vertical_length_tablet=”0px” ol_text_shadow_blur_strength=”ol_text_shadow_style,%91object Object%93″ ol_text_shadow_blur_strength_tablet=”1px” quote_text_shadow_horizontal_length=”quote_text_shadow_style,%91object Object%93″ quote_text_shadow_horizontal_length_tablet=”0px” quote_text_shadow_vertical_length=”quote_text_shadow_style,%91object Object%93″ quote_text_shadow_vertical_length_tablet=”0px” quote_text_shadow_blur_strength=”quote_text_shadow_style,%91object Object%93″ quote_text_shadow_blur_strength_tablet=”1px” header_text_shadow_horizontal_length=”header_text_shadow_style,%91object Object%93″ header_text_shadow_horizontal_length_tablet=”0px” header_text_shadow_vertical_length=”header_text_shadow_style,%91object Object%93″ header_text_shadow_vertical_length_tablet=”0px” header_text_shadow_blur_strength=”header_text_shadow_style,%91object Object%93″ header_text_shadow_blur_strength_tablet=”1px” header_2_text_shadow_horizontal_length=”header_2_text_shadow_style,%91object Object%93″ header_2_text_shadow_horizontal_length_tablet=”0px” header_2_text_shadow_vertical_length=”header_2_text_shadow_style,%91object Object%93″ header_2_text_shadow_vertical_length_tablet=”0px” header_2_text_shadow_blur_strength=”header_2_text_shadow_style,%91object Object%93″ header_2_text_shadow_blur_strength_tablet=”1px” header_3_text_shadow_horizontal_length=”header_3_text_shadow_style,%91object Object%93″ header_3_text_shadow_horizontal_length_tablet=”0px” header_3_text_shadow_vertical_length=”header_3_text_shadow_style,%91object Object%93″ header_3_text_shadow_vertical_length_tablet=”0px” header_3_text_shadow_blur_strength=”header_3_text_shadow_style,%91object Object%93″ header_3_text_shadow_blur_strength_tablet=”1px” header_4_text_shadow_horizontal_length=”header_4_text_shadow_style,%91object Object%93″ header_4_text_shadow_horizontal_length_tablet=”0px” header_4_text_shadow_vertical_length=”header_4_text_shadow_style,%91object Object%93″ header_4_text_shadow_vertical_length_tablet=”0px” header_4_text_shadow_blur_strength=”header_4_text_shadow_style,%91object Object%93″ header_4_text_shadow_blur_strength_tablet=”1px” header_5_text_shadow_horizontal_length=”header_5_text_shadow_style,%91object Object%93″ header_5_text_shadow_horizontal_length_tablet=”0px” header_5_text_shadow_vertical_length=”header_5_text_shadow_style,%91object Object%93″ header_5_text_shadow_vertical_length_tablet=”0px” header_5_text_shadow_blur_strength=”header_5_text_shadow_style,%91object Object%93″ header_5_text_shadow_blur_strength_tablet=”1px” header_6_text_shadow_horizontal_length=”header_6_text_shadow_style,%91object Object%93″ header_6_text_shadow_horizontal_length_tablet=”0px” header_6_text_shadow_vertical_length=”header_6_text_shadow_style,%91object Object%93″ header_6_text_shadow_vertical_length_tablet=”0px” header_6_text_shadow_blur_strength=”header_6_text_shadow_style,%91object Object%93″ header_6_text_shadow_blur_strength_tablet=”1px” box_shadow_horizontal_tablet=”0px” box_shadow_vertical_tablet=”0px” box_shadow_blur_tablet=”40px” box_shadow_spread_tablet=”0px” vertical_offset_tablet=”0″ horizontal_offset_tablet=”0″ z_index_tablet=”0″]<\/p>\n
Hardware and Technical Specs<\/h1>\n
\u00a0<\/p>\n
- \n
- The Minerva supercomputer is maintained by Scientific Computing and Data (SCD) at the Icahn School of Medicine, Mount Sinai.<\/li>\n
- Minerva was created in 2012 and has been upgraded several times (most recently in Nov. 2024 and Feb. 2026) and has over 20 petaflops of compute power.<\/li>\n
- It consists of 25,584 Intel Platinum processors in different generations including 2.1 GHz, 2.3GHz, 2.6 GHz, and 2.9 GHz computing cores (either 48, 64, 96, or 112 cores per nodes with two sockets in each node) with 1.5 or 2 terabytes (TB) of memory per node, 408 Nvidia graphics processing units (GPUs), including 48 B200s, 236 H100s, 32 L40S, 44 A100s, and 48 V100s, together with 452 TB of total memory, and 46 petabytes of raw spinning storage (32 petabytes of usable storage) accessed via IBM\u2019s Spectrum Scale\/General Parallel File System (GPFS).<\/li>\n
- Minerva has contributed to over 2,100 peer-reviewed publications since 2012.<\/li>\n<\/ul>\n
The following diagram shows the overall Minerva configuration:<\/p>\n
![](\"https:\/\/scpublic.dmz.hpc.mssm.edu\/HPC\/Minerva_Arch\/Minerva_Arch.jpg\")
<\/p>\n\u00a0<\/p>\n
\u00a0<\/p>\n
\n\nCompute Nodes<\/h2>\n
Chimera Partition<\/strong><\/h4>\n
\n\n
\n Added in Nov. 2024.<\/i><\/u><\/td>\n Nodes purchased prior to 2024 and integrated to new NDR network via HDR 100Gb\/s.<\/i><\/u><\/td>\n<\/tr>\n \n \n - \n
- 4 login nodes <\/b>\u2013 Intel Emerald Rapids 8568Y+, 2.3GHz \u2013 96 cores with 512 GB memory per node.<\/li>\n
- 146 compute nodes* <\/b>\u2013 Intel Emerald Rapids 8568Y+, 2.3GHz\u2013 96 cores with 1.5 TB memory per node.\n
- \n
- 14,016 cores in total.<\/li>\n<\/ul>\n<\/li>\n
- 188 H100 in 47 nodes<\/b> \u2013 Intel ER 8568Y+, 2.3GHz\u2013 96 cores with 1.5 TB memory per node.\n
- \n
- 4 x H100-80GB(SXM5) NVLinked GPUs per node.<\/li>\n<\/ul>\n<\/li>\n
- 32 L40s GPUs in 4 nodes<\/b> \u2013 AMD Genoa 9334 2.7GHz \u2013 64 cores with 1.5TB memory per node.\n
- \n
- 8x L40s-48GB GPUs per node. L40s doesn\u2019t support FP64.<\/li>\n<\/ul>\n<\/li>\n
- 3.84 TB Local NVME SSD (3.5TB usable) per node.<\/strong>\n
- \n
- It can deliver a sustained read-write speed of 3.5 GB\/s in contrast with SATA SSDs that limit at 600 MB\/s.<\/li>\n<\/ul>\n<\/li>\n
- NDR InfiniBand<\/strong> fat tree fabric networking (400Gb\/s).\n
- \n
- 6 service nodes.<\/li>\n
- 295.5 TB memory in total.<\/li>\n
- Direct water-cooling solution.<\/li>\n
- New NFS storage (for users\u2019 home directories) \u2013140 TB usable.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/td>\n
\n - \n
- 33 high memory nodes<\/b> \u2013 Intel 8268 24C, 2.9GHZ \u2013 1.5 TB memory.<\/li>\n
- 48 V100 GPUs in 12 node <\/b>\u2013 Intel 6142 16C, 2.6GHz \u2013 384 GB memory \u2013 4x V100-16 GB GPU.<\/li>\n
- 32 A100 GPUs in 8 nodes<\/b> \u2013 Intel 8268 24C, 2.9GHz \u2013 384 GB memory \u2013 4x A100-40 GB GPU.\u00a0 1.92TB SSD (1.8 TB usable) per node.<\/li>\n
- 8 A100 GPUs in 2 nodes<\/b> \u2013 Intel 8358 32C, 2.6GHz \u2013 2 TB memory \u2013 4x A100-80 GB GPU.\n
- \n
- A100 is connected via NVLink.<\/li>\n
- 7.68 TB NVMe SSD (7.0TB usable) per node.<\/li>\n<\/ul>\n<\/li>\n
- 8 H100 GPUs in 2 nodes<\/b> \u2013 Intel 8358 32C, 2.6 GHz \u2013 0.5 TB memory \u2013 4xH100-80GB GPU.\n
- \n
- 3.84 TB NVMe SSD (3.5 TB usable) per node.<\/li>\n<\/ul>\n<\/li>\n
- 3,520 cores in 55 nodes<\/strong>\u2013 Intel IceLake 8358 \u2013 1.5 TB memory.<\/li>\n
- [Decommissioned on Nov. 5th 2024] 4 login nodes<\/strong> \u2013 Intel Xeon(R) Platinum 8168 24C, 2.7GHz \u2013 384 GB memory.<\/li>\n
- [Decommissioned on July 17th and Nov. 5th 2024] 275 compute nodes*<\/strong> \u2013 Intel 8168 24C, 2.7GHz \u2013 192 GB memory.\n
- \n
- 13,152 cores (48 cores per node).<\/li>\n<\/ul>\n<\/li>\n
- *Compute Node<\/em>\u00a0\u2014where you run your applications. Users do not have direct access to these machines. Access is managed through the LSF job scheduler.<\/li>\n<\/ul>\n
\u00a0<\/p>\n
\u00a0<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
AIMS (AI Mount Sinai) Partition<\/strong><\/h4>\n
\n\n
\n $2M AIMS awarded by NIH (Kovatch PI<\/i><\/u>), <\/i>Open to eligible NIH funded GPU projects, Added in Feb. 2026.<\/i><\/u><\/td>\n \u00a0<\/b><\/td>\n<\/tr>\n \n \n - \n
- 48 B200 in 6 nodes<\/b> \u2013 Lenovo SR780a V3 DGX.\n
- \n
- 8x NVLinked B200 GPUs \u2013 192 GB memory per GPU, 9 TB GPU memory in total.<\/li>\n
- 112 Intel Xeon Platinum 8570 2.1GHz cores and 2 TB memory per node, 672 cores and 12 TB memory in total.<\/li>\n
- 25 TB NVMe SSD local storage per node.<\/li>\n
- FP4 (4-bit floating point) format, enabling nearly an exaflop with FP4 for AI inference.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/td>\n
\u00a0<\/b><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n \n\nSummary<\/strong><\/h4>\n
\n\n
\n Total system memory\u00a0<\/b>(computes + GPU) =\u00a0452 TB\u00a0<\/strong><\/td>\n Total number of cores\u00a0<\/b>(computes + GPU) =\u00a025,584 cores<\/strong><\/td>\n<\/tr>\n \n CPU Peak performance\u00a0<\/b>of all nodes = 1.9 PFLOPS<\/strong><\/td>\n GPU Peak performance<\/b> based FP64 Tensor cores = 18.6 PFLOPS.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n \u00a0<\/p>\n<\/div>\n<\/div>\n
\n\nFile System Storage<\/h2>\n
Minerva uses IBM\u2019s General Parallel File System (GPFS) because it has advantages that are specifically useful for informatics workflows that involve high speed metadata access, tiered storage, and sub-block allocation. Metadata is the information about the data in the file system, and it is stored in flash memory for fast access. A parallel file system was used for Minerva because NFS and other file systems cannot scale to the number of nodes or provide performance for the large number of files involved in typical genomics workflows.<\/p>\n
Currently we have one parallel file system on Minerva, Arion, which users can access at \/sc\/arion. The Hydra file system was retired at the end of 2020.<\/p>\n
\n
\n \n<\/colgroup>\n \n GPFS Name<\/strong><\/td>\n Lifetime<\/strong><\/td>\n Storage Type<\/strong><\/td>\n Raw PB<\/strong><\/td>\n Usable PB<\/strong><\/td>\n<\/tr>\n \n Arion<\/td>\n 2019 \u2013<\/td>\n Lenovo DSS<\/td>\n 14<\/td>\n 9.6<\/td>\n<\/tr>\n \n Arion<\/td>\n 2019 \u2013<\/td>\n Lenovo G201 flash<\/td>\n 0.12<\/td>\n 0.12<\/td>\n<\/tr>\n \n Arion<\/td>\n 2020 \u2013<\/td>\n Lenovo DSS<\/td>\n 16<\/td>\n 11.2<\/td>\n<\/tr>\n \n Arion<\/td>\n 2021 \u2013<\/td>\n Lenovo DSS<\/td>\n 16<\/td>\n 11.2<\/td>\n<\/tr>\n \n \u00a0<\/td>\n \u00a0<\/td>\n Total<\/strong><\/td>\n 46<\/strong><\/td>\n 32<\/strong><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n We setup Minerva Restricted Cluster in July 2026 with a dedicated encrypted GPFs file system, called ArionEncrypt.<\/p>\n
\n
\n \n<\/colgroup>\n \n GPFS Name<\/strong><\/td>\n Lifetime<\/strong><\/td>\n Storage Type<\/strong><\/td>\n Raw PB<\/strong><\/td>\n Usable PB<\/strong><\/td>\n<\/tr>\n \n ArionEncrypt<\/td>\n 2026 \u2013<\/td>\n IBM<\/td>\n 10<\/td>\n 6.9<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n \u00a0<\/p>\n<\/div>\n<\/div>\n
- 48 B200 in 6 nodes<\/b> \u2013 Lenovo SR780a V3 DGX.\n
- [Decommissioned on Nov. 5th 2024] 4 login nodes<\/strong> \u2013 Intel Xeon(R) Platinum 8168 24C, 2.7GHz \u2013 384 GB memory.<\/li>\n
- 48 V100 GPUs in 12 node <\/b>\u2013 Intel 6142 16C, 2.6GHz \u2013 384 GB memory \u2013 4x V100-16 GB GPU.<\/li>\n
- 33 high memory nodes<\/b> \u2013 Intel 8268 24C, 2.9GHZ \u2013 1.5 TB memory.<\/li>\n
- 146 compute nodes* <\/b>\u2013 Intel Emerald Rapids 8568Y+, 2.3GHz\u2013 96 cores with 1.5 TB memory per node.\n
- 4 login nodes <\/b>\u2013 Intel Emerald Rapids 8568Y+, 2.3GHz \u2013 96 cores with 512 GB memory per node.<\/li>\n
![\"\"](\"https:\/\/labs.icahn.mssm.edu\/minervalab\/wp-content\/uploads\/sites\/342\/2021\/08\/Minerva-Configuration-08-2021-scaled.gif\")
<\/p>