The collaborative AMIS (Advanced Measurement Instrument and Services) project goal was to provide network monitoring, measurement, and analysis services for network traffic at points of international network exchange, while preserving the privacy of the underlying network data.  Our teams focus areas included distributed device, data, and task management through the research and development of a platform to support collaborative modular operations, such as data acquisition (measurement tasks), functional services (privacy, analysis, complex event processing, correlation, etc.), and data management (aggregation, distribution, storage, etc).  Developing a multi-tenant monitoring and measurement system based on geographically distributed high-rate network interfaces comes with its own set of challenges.  Specifically, one must be able to provide methods to describe a measurement topology, such as where measurements should take place and what tools to be used, execute measurement tasks across devices, manage resources within and between devices and backend services, provide methods to manage resulting measurement data, and provide methods for downstream analysis or other functional modules.  We present the five main output components of our work:   

1) Agent-based Edge Computing Platform:  Edge and fog computing is often thought of the process of moving computation (function) in proximity to sources of data generation.  It is not feasible to transmit the data used for high-rate network measurements to remote locations, which makes the task a good candidate for edge computing techniques.  We made use of and extended an existing agent-based edge computing framework (http://cresco.io/) developed by the project team to coordinate device and data operations as shown in Figure 1, between distributed devices and backend services.  An example measurement pipeline with related framework services is shown in Figure 2.  The following components describe additional methods in which underlying framework components were used.       

2) Measurement Topology Language:  In order to perform distributed network measurement you must have compatible (versions) tools at each measurement point, configurations to inform measurement tasks, and methods to execute task and manage resulting data.  We developed a declarative graph-based language, as shown in Figure 5, to describe the network measurement topology, including measurement locations, points of data aggregation, and downstream functional units.  The underlying agent-based framework distributes management tools based on requested versions and configurations to device and provisions requested ancillary services on backend (cloud) resources, such as analytic or storage services.  In addition, a simplified RESTful API for common measurement functions was provided.  In addition, we have developed a web-based GUI to be used to construct measurement, data flow, and downstream functional module topologies, as shown in Figure 5.       

3) Resource Management and Scheduling:  In mutli-tenent (multi-user, multi-task, etc.) operations great care must be taken to ensure devices with finite resources, such as measurement devices are not overwhelmed with request.  Likewise, we want to make sure that duplicate concurrent measurement request do not result in duplicate resource utilizations.  Making use of the underlying agent-based framework we collected the state of agents, relationship between agents, and the resource utilization of manage tasks, as shown in Figure 3.  On submission, the entire measurement topology is decomposed into node and edge components, measurement topologies with explicit (measurement location) assignments as assigned if resources are available, unassigned function services like CEP, filtering, or aggregation services are scheduled where resources are available and topology allows, and finally upstream and downstream configuration is added to all component configurations composing the measurement pipeline.  Constrain satisfaction, resource, and service discovery techniques are used in scheduling optimizations.         

4) Analysis Services:  As we previously described, source data might come directly from a measurement device or some other upstream source, such as an aggregation of filtered or enriched data.  In addition to task-based measurement tools developed by our collaborators, we deployed continuous IPFIX/NetFlow generation services, with ASN flow injection.  Our team developed a number of modules to be used for complex event processing, flow correlation, filtering, and new stream composition services.  These modules can be chained with other functional modules allowing very complex measurement and alerting services to be deployed.  A few example measurement-focused queries based on the EQL (http://esper.espertech.com/release-7.1.0/esper-reference/html/epl-views.html) and SPARQL (https://www.w3.org/TR/sparql11-overview/) language are shown in Figure 4.       

The underlying system we have developed is not limited to just network measurement and can be used for many types of task execution, data management, and application monitoring tasks.  The system is especially beneficial in the area of edge computing, where data processing is moved toward sources of data generation.  The impacts range from device and data management in healthcare to smart cities and beyond.

Currently, the Github repository specific to the AMIS project is managed by the lead institution in this project (University of Massachusetts Lowell), and has not yet been made public. Until it is made public, please email PI Cody Bumgardner to request access to the repisitory.

The Github repository pertaining the the described agent-based edge framework can be found here: https://github.com/CrescoEdge/quickstart

Last Modified: 06/21/2019
Modified by: Vernon Bumgardner