Traffic measurement is central to network operation, management, and security. Our overall project objective is to design optimal learning algorithms to improve the measurement performance (in terms of both accuracy and delay) for a wide variety of traffic measurement, network inference, and anomaly detection problems.

We have developed stochastic online learning strategies for traffic matrix estimation based on theories such as multi-arm bandit (MAB), heavy hitter detection based on group testing, and active learning theories under known and unknown traffic models. We successfully developed the optimal test plan that identifies all the heavy hitters with a minimum number of tests/measurements. We established the optimal test-plan in closed form and shows that it is order optimal among all test plans as the population size grows to infinity. We show via simulation examples orders of magnitude improvement of the group testing approach over two prevailing sampling-based approaches in detection accuracy and counter consumption.

 With a reconfigurable control plan and a global view of the network, SDN architecture allows SDN controller to optimally and dynamically allocate measurement resources to support a variety of monitoring applications. We leverage this flexibility to develop adaptive flow measurement strategies that are tightly coupled with network inference engine to address a variety of network tomography problems, including traffic matrix estimation, link (performance) identification, and network anomaly detection. First, we developed a framework called iSTAMP (Intelligent SDN-based Traffic (de)Aggregation and Measurement Paradigm), which uses the flexibility of SDN to partition TCAM entries of switches/routers into two parts to: 1) optimally aggregate flows to meet routing constraints while providing best observations for network inference, and 2) directly measure K most informative flows. We have built and a prototype of iSTAMP on GENI platform and delivered demos/posters in three GEC conferences. We proceeded to address practical challenges of deploying iSTAMP in multi-node environment, including (1) how to design pseudo-optimal aggregation matrix in real time while meeting the routing and longest-prefix match constraints (i.e., make the rules implementable), (2) how to learn the K most important flows in a scalable manner with minimal “learning/exploration” phase, and (3) how to allocate flow measurement rules among distributed SDN switches spread across the networks.

Looking beyond passive traffic measurement, our team has also developed online learning-based framework for passive and active network performance measurement. For instance, our system can intelligently measure a small sub-set of the precisely determined entries of the matrix of per-flow measurements (known as Optimal Observation Matrix (OOM)) which leads to the best possible estimation accuracy via applying matrix completion techniques

Through collaboration with our industry partners, we have applied our measurement and analysis techniques on a variety of data sets, ranging from traffic from backbone ISPs and data centers. This project has graduated one PhD and five MS students. It also provided training for two other PhD and one other MS students. In total, this project has recruited one female PhD and three female MS students (two of which have graduated). Four undergraduate students have conducted independent research studies under this project.

Last Modified: 10/27/2016
Modified by: Chen-Nee Chuah