Today's large-scale "cloud" services are incredibly power hungry, using energy sufficient to power every household in New York City twice over, and that percentage is expected to double in the next 10 years[1].  According to the NRDC, "In 2013, U.S. data centers consumed an estimated 91 billion kilowatt-hours of electricity, equivalent to the annual output of 34 large (500-megawatt) coal-fired power plants. Data center electricity consumption is projected to increase to roughly 140 billion kilowatt-hours annually by 2020, the equivalent annual output of 50 power plants, costing American businesses $13 billion annually in electricity bills and emitting nearly 100 million metric tons of carbon pollution per year.[2]"  The enormous energy demands of these datacenters limits their growth potential and results in unnecessary operational expense.  The good news is that there are many improvements that can be made in datacenters to raise their efficiency.  In some cases, commercial providers run their systems at only 10 or 20% utilization due in large part to the inability for the underlying network to scale to meet bandwidth requirements.

Fundamentally, the packet-switching technology underlying current data-center interconnects limits their ability to scale: implementing control- and data-planes necessary to forward packets individually is costly at present, and will rapidly cease to be feasible as link data rates continue to increase.

The overall goal of this project has been to leverage microsecond optical circuit-switch technology to develop a hybrid switching paradigm that spans the gap between traditional circuit switching and full-fledged packet switching, achieving a level of performance and scale not previously attainable.  We have designed and built a hybrid switch whose optical switching capacity is orders of magnitude larger than the electrical packet switch, yet whose performance from an end-to-end perspective is largely indistinguishable from a giant (electrical) packet switch.  A key aspect of this project has been demonstrating a system-level control plane for hybrid networks capable of leveraging both circuit- and packet-switching.

Our work has resulted in a characterization of commercial datacenter workloads, in particular multiple clusters at Facebook.  Our findings have been used by our project as well as other projects to design an interconnect supporting commercial workloads.  We have explored circuit switching in the context of a new, novel, non-crossbar “Selector Switch” architecture.  A Selector Switch switches entire groups of input ports between entirely disjoint network matchings, rather than switching traffic from one input port to a single output port.  We have studied the fundamental scaling limitations of MEMS-based optical switches, and found that they scale as a function of resolvable states, rather than ports.  This means that by selecting among matchings, rather than individual input/output port matchings, we can build a Selector Switch that scales to 1000s of ports with existing technology, without requiring expensive optical amplification or telecom-grade transceivers.  We have designed and build a 61-port Selector Switch prototype.

Paired with a Selector Switch-based architecture, we have demonstrated how to build a Top-of-Rack switch that supports hybrid packet/circuit networks.  The resulting prototype, REACToR, relies on a trio of mechanisms to support hybrid networks.  First, it explicitly “pulls” packets from attached servers based on foreknowledge of which circuits will be established at a given time.  This eliminates the need to maintain large packet buffers in the ToR.  Second, REACToR classifies packets based on a centralized circuit schedule, directing low-latency packets to a separate packet-switched network to reduce overall latency.  Finally, REACToR plays a key role in ensuring datacenter-wide synchronization.  We have designed and build a 16-port FPGA-based REACToR prototype.

We have designed two centralized circuit-switch scheduling algorithms, Solstice and Albedo.  Solstice operates by processing an input demand matrix, representing a snapshot of the overall datacenter workload demand.  The Solstice algorithm is a computationally tractable heuristic for maximizing the bandwidth through the network, operating closely to non-tractable optimal solutions.  We next designed Albedo, which extends the assumptions of Solstice by assuming that traffic could be indirected, or sent through intermediate nodes.  By allowing some traffic to travel over what would otherwise be unused paths, Albedo is able to increase the total amount of traffic sent through the network as compared to Solstice.  We have implemented both Solstice and Albedo in software.

The net result of this project is a new strategy for supporting high bandwidth networks in the future, sidestepping the impending limitations of all-packet-switched networks.  By switching some datacenter traffic optically, future datacenters will be able to support higher bandwidths, which will result in faster computations, fewer servers waiting for the data they need (and thus less server energy usage), and higher overall efficiency.  The result of higher datacenter efficiency is a dramatic reduction in overall US energy usage (since datacenters account for 2.5% of US energy usage), and the removal of a barrier that companies encounter today, namely growing their networks to meet their customer demands.

[1] https://energy.gov/eere/buildings/data-centers-and-servers

[2] https://www.nrdc.org/resources/americas-data-centers-consuming-and-wasting-growing-amounts-energy

Last Modified: 04/23/2017
Modified by: George Porter