The San Diego Supercomputer Center at the University of California, San Diego deployed the Gordon data-intensive supercomputer during 2011. After extensive testing, SDSC operated Gordon for allocated access by academic researchers, educators, and students through NSF’s XSEDE project from March 2012 to March 2017. Some NSF-funded projects continue to use Gordon, with its operation currently funded by the Simons Foundation. During its 61 months of operation as an XSEDE resource, it ran 2.3 million jobs; it provided 450 million core hours for more than 9,000 users, most of whom gained access via a gateway rather than via the command line; and it enabled publication of hundreds of scientific papers.

Gordon has a peak speed of 341 Tflop/s delivered by 16,384 cores in 1,024 compute nodes, each with 64 GB of DRAM. Its signature feature is 300 TB of flash memory served by 64 I/O nodes, with 300 GB accessible to each compute node in the normal configuration.

Gordon was designed by SDSC in collaboration with vendor partners Appro, Intel, Mellanox, and ScaleMP. The design incorporated three significant technology innovations:

• Extensive use of flash memory via SSDs to bridge the latency gap in the memory hierarchy between DRAM and disk, thereby greatly improving performance for analyses that generate and reuse large amounts of intermediate data.
• Virtual shared-memory via the vSMP software, which enables large-memory analyses in a cost-effective manner by aggregating up to 1 TB of DRAM from 16 separate compute nodes.
• A dual-rail, 3D torus interconnect with 4x4x4 topology that uses separate rails for message passing and I/O to minimize interference between them.

With faster processors and more DRAM per node than on most other XSEDE resources when operation of Gordon began, it was very attractive to users from a broad swath of science and engineering disciplines. The large amount of DRAM was used to obtain a better understanding of the structure of viruses in two studies. Researchers from Cornell and TSRI reconstructed the HK97 virus from microscopy data, while researchers from UCSD modeled the surface proteins of avian flu virus.

Flash memory was widely used as an alternative to disk for I/O of intermediate data generated during data-intensive analyses. Researchers doing quantum chemistry and materials science analyses made extensive use of this capability to store and retrieve the large numbers of electron integrals that arise during such analyses. A research group at Harvard screened millions of candidate semiconductors to find better photovoltaic materials for solar cells. Each candidate was analyzed in a separate job, with hundreds of jobs running simultaneously, all using flash memory to improve performance and minimize the impact on other users that disk I/O would entail.

Flash memory was also used as persistent storage for several projects that received dedicated allocations of an I/O node, and use by the following two projects continues. The IODA project, operated by CAIDA at SDSC, monitors Internet traffic data to detect hacker activity or connectivity disruptions. Deploying the IODA databases on the flash memory of an I/O node resulted in an 18x speedup compared to using disk. The OpenTopography gateway, another project hosted at SDSC, provides persistent access to topography data from high-resolution LiDAR and to tools for analyzing those data.

Virtual shared memory enabled by the vSMP software was used by researchers who needed even more DRAM than in a single compute node or who could benefit from using more than 16 cores in a shared-memory analysis. Researchers from Cornell studied the stress response of a vertebra using a high-resolution, finite-element model. Storing the resulting 750-GB stiffness matrix in DRAM aggregated across multiple compute nodes allowed the analysis to complete more than 10x faster than splitting the matrix between 64 GB of DRAM in a single compute node and 700 GB of disk. Researchers from UC Irvine and UCSD analyzed a large network in mathematical anthropology using 256 cores in a shared-memory model, while other researchers at UCSD post-processed large cosmological simulations using both 256 cores and 256 GB of shared memory.

Science gateways, especially the CIPRES gateway for phylogenetic research, accounted for an increasing percentage of users and enabled many significant results during the operation of Gordon as an XSEDE resource. A research team led from UC Berkeley generated a new tree of life that revealed the existence of a massive new superphylum of bacteria.

Tens of outreach activities exposed thousands of researchers, educators, and students to the benefits of data-intensive computing, especially those enabled by Gordon. SDSC staff hosted tutorials at scientific meetings, workshops at SDSC and on other university campuses, and annual summer institutes. These activities targeted potential users of Gordon in varied disciplines, especially fields such as finance and ecology that had previously made little use of high-performance computing. University faculty also used Gordon for classes in parallel processing and data science.

Last Modified: 06/28/2017
Modified by: Michael L Norman