| NSF Org: |
CNS Division Of Computer and Network Systems |
| Recipient: |
|
| Initial Amendment Date: | July 29, 2018 |
| Latest Amendment Date: | July 29, 2018 |
| Award Number: | 1823357 |
| Award Instrument: | Standard Grant |
| Program Manager: |
Sol Greenspan
sgreensp@nsf.gov (703)292-7841 CNS Division Of Computer and Network Systems CSE Directorate for Computer and Information Science and Engineering |
| Start Date: | August 1, 2018 |
| End Date: | July 31, 2020 (Estimated) |
| Total Intended Award Amount: | $53,454.00 |
| Total Awarded Amount to Date: | $53,454.00 |
| Funds Obligated to Date: |
|
| History of Investigator: |
|
| Recipient Sponsored Research Office: |
1910 UNIVERSITY DR BOISE ID US 83725-0001 (208)426-1574 |
| Sponsor Congressional District: |
|
| Primary Place of Performance: |
ID US 83725-1135 |
| Primary Place of
Performance Congressional District: |
|
| Unique Entity Identifier (UEI): |
|
| Parent UEI: |
|
| NSF Program(s): | CCRI-CISE Cmnty Rsrch Infrstrc |
| Primary Program Source: |
|
| Program Reference Code(s): |
|
| Program Element Code(s): |
|
| Award Agency Code: | 4900 |
| Fund Agency Code: | 4900 |
| Assistance Listing Number(s): | 47.070 |
ABSTRACT
Benchmark programs play an important role in evaluating the advances in program analysis research. Thus, the confidence of the evaluation depends on the quantity and the quality of the benchmark programs. The larger their number and the closer they resemble real-world programs, the higher confidence in the evaluation. Currently, research in the field of program analysis often relies on a few outdated or deficient benchmark programs for evaluation purposes. The difficulty in obtaining sufficient benchmark programs comes from the requirements imposed on the benchmark programs by program analyzers. Usually researchers manually obtain, inspect, and transform potential benchmark programs to create benchmark programs that meet those requirements. Such an unscalable approach often fails to produce adequate benchmark programs. The goal of this research is to automate this process and make it available through an online, public infrastructure. The infrastructure facilitates obtaining, selecting, and transforming open-source software projects for use as benchmark programs. In addition, the infrastructure gives researchers the ability to easily curate benchmark programs for program analysis and share the results of running their analyzers on those benchmarks.
The objectives of this project are to solicit requirements and feedback from the program analysis community on automating benchmark program generation, and to prototype a Program Analysis Collaboratory (PAClab) research infrastructure that implements that process. Using researchers' specifications for adequate benchmark program generation, PAClab locates and obtains potential benchmark programs in open-source repositories; performs necessary program transformations as specified by researchers; and outputs the adequate benchmark programs. In addition, PAClab enables researchers to share their program analyzers using container technology, the transformed benchmark programs, and the results of running their analyzers on those benchmark programs. The intellectual merit of the project lies in investigating techniques for specifying and automating program transformations for targeted program analyzers. The broader impacts of the project stem from its potential to broaden adaptation of program analysis advances through the increased confidence of their evaluations; to lower the entry barrier for new program analysis researchers; and to accelerate the pace in program analysis research. PAClab's educational benefits include the ability for students to easily locate real-world programs and evaluate their implementations of program analyzers.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
PUBLICATIONS PRODUCED AS A RESULT OF THIS RESEARCH
Note:
When clicking on a Digital Object Identifier (DOI) number, you will be taken to an external
site maintained by the publisher. Some full text articles may not yet be available without a
charge during the embargo (administrative interval).
Some links on this page may take you to non-federal websites. Their policies may differ from
this site.
PROJECT OUTCOMES REPORT
Disclaimer
This Project Outcomes Report for the General Public is displayed verbatim as submitted by the Principal Investigator (PI) for this award. Any opinions, findings, and conclusions or recommendations expressed in this Report are those of the PI and do not necessarily reflect the views of the National Science Foundation; NSF has not approved or endorsed its content.
Static program analyses allow software engineers the ability to analyze computer programs without the need to actually run those programs. The ultimate goal of the Program Analysis Collaboratory (PAClab) project is to improve open-source program benchmark availability for static analysis researchers, which will result in stronger empirical evaluation of static analysis tools and techniques thus giving researchers and practitioners more confidence in the claims made by those tool developers. In particular, this project outlines and implements a new strategy for automatically obtaining program benchmarks, that compile without error, from open-source repositories according to user-defined specifications that describe desirable program benchmark features.
The efforts of this planning project resulted in a fully functional web-based prototype tool named PAClab that automatically obtains from the largest repository of open-source programs, GitHub, benchmark programs written in the Java programming language. PAClab provides a web-based and user-friendly interface for entering program selection criteria. The key innovative component of the tool is a sophisticated program transformation toolchain that removes portions of the source code that are not relevant to the analysis at hand and automatically resolves dependencies to make the desired programs compilable outside of their native projects. Using its efficient, scalable backend infrastructure PAClab can process a large number of program benchmarks while support multiple users. PAClab allows researchers to easily share their generated program benchmark sets by providing collaborators with a link to the session. This session page contains selection criteria information as well as the resulting set of benchmark programs that are also available for download. This enables better collaboration within the program analysis research community as well as the ability to more easily replicate prior research results.
This planning project disseminated its results through two peer-reviewed publications, an invited talk, two masters theses, and close collaboration with the program analysis community to integrate PAClab in empirical evaluation of program analysis research. In addition, the project provided support and training for two female graduate students in Computer Science.
Last Modified: 08/31/2020
Modified by: Elena Sherman
Please report errors in award information by writing to: awardsearch@nsf.gov.
