This NSF-funded infrastructure project built a dynamically configurable cluster called MYSTIC to study system re-configurability across the entire computing stack, from the processor to memory, storage, and the network. The MYSTIC project aimed to significantly improve programmable infrastructure in the computer science department at the Illinois Institute of Technology (IIT). The primary goal of this work was to fill the existing void in delivering an open testbed for experimenting with reconfigurable communication and I/O subsystems to perform low-level systems research. 

The PIs designed, procured, and deployed a dynamically configurable system MYSTIC to study system reconfigurability across the entire stack. This work aimed to remove network, memory, and storage performance bottlenecks by dynamically reconfiguring them to the needs of an application. MYSTIC is composed of exotic hardware that is not available in any of the leading systems testbeds, and it combines programmable networks-on-chip meshes, programmable NICs, various processor architectures, and a variety of accelerators. Although the PIs have access to many of the fastest supercomputers and clouds in the world, but none of them provide the type of hardware needed for programmability at the lowest levels with such a variety of hardware under one system. This testbed is likely to have far-reaching impact by broadening the research possible on real reconfigurable systems. 

MYSTIC software stack is organized in three distinct layers, each layer depending on the next layer: Overcloud, Undercloud, and Protocloud. The Overcloud runs the OpenStack cloud platform for resource management. The Undercloud uses MaaS to deploy basic services such as LDAP, Ceph, and Grafana. The Protocloud is used on infrastructure nodes serving as gateways, firewalls, DHCP, and DNS services. Experimentation on MYSTIC uses both traditional software stacks (e.g. Linux) as well as specialized software stacks (e.g. Nautilus Aerokernel, Palacios VMM), through either a virtual machines or bare metal provisioning.

MYSTIC is physically located in the Ocient Computation Center at IIT. It is composed of three 42U racks and contains 54 heterogenous nodes that can consume up to 60kWh of electricity. These nodes contain a total of 1396-cores of general-purpose cores across all major architectures and manufacturers (Intel Xeon Scalable Processors, Broadwell, Haswell, Phi x200, AMD Epyc, IBM Power9, Cavium ThunderX & ThunderX2, and ARMv8 A72). These nodes have 4.3TB RAM, 41TB NVMe storage, 70TB SATA SSDs, and 384TB HDD storage. MYSTIC houses 40 accelerators (8x Intel Xeon Phi x100, 8x NVIDIA Tesla K40 GPUs & 16x V100 GPUs, and 8x Intel Altera FPGAs). The system has an internal 100GbE fat-tree network and an external 10GbE network. The largest individual system has 192-cores and 768GB of memory, making it an ideal many-core platform to evaluate future processor architectures that won?t be available for years to come. The system also has a distributed file system (Ceph) with 150TB persistent storage. 

Cutting edge research has been conducted from computing systems, to operating systems, to applications in machine learning ? research that could not have been conducted on other platforms with the same level of efficiency or results. Some of the major systems research projects this work has enabled are: 1) XTASK - eXTreme fine-grAined concurrent taSK invocation runtime; 2) XSearch - Distributed Indexing and Search in Large-Scale File Systems; 3) Nautilus Aerokernel; and 4) Computer Audition. The MYSTIC system has enabled research that produced 17 papers in peer-reviewed workshops, conferences, and journals. 

The intellectual merit of the MYSTIC system lies in its focus on enabling new low-level systems research, primarily for exploring techniques applicable to programmable and reconfigurable communication and I/O substrates. This includes research in system-level adaptation and configuration of storage and memory systems, exploration of OS abstractions built atop programmable NoCs, and investigations tying inter-node communication with intra-node communication.  These directions have resulted in research systems that increase performance across a wide array of applications, while also enabling new directions in OS and system software design and implementation for reconfigurable and programmable hardware. The PIs have systems expertise spanning the entire stack from computer architecture, operating systems, memory hierarchies, storage systems, and networking.

MYSTIC?s broader impacts can be expressed in the far reach it has had in reaching over one hundred faculty and students who have used MYSTIC, spanning undergraduate and graduate students, postdocs, researchers, and faculty members ? across 19 different institutions. The users of MYSTIC work in computer systems, such as parallel computing, real-time systems, and distributed databases, as well as big data applications such as machine learning, natural language processing, scientific computing, security, networking, and theory. There were 7 students who have been trained directly on the MYSTIC system or have carried out research projects with MYSTIC funds; these students range from four undergraduate students, two master students, and one PhD student. One student was female, and six students were domestic students. 

More information on MYSTIC can be found at http://mystic.cs.iit.edu; MYSTIC can be accessed at https://openstack1.mystic.cs.iit.edu/horizon/auth/login/.

Last Modified: 01/05/2022
Modified by: Ioan Raicu