What does it mean for a computer to "understand" a technical domain,such as software? To test understanding for a person, one wouldprobably try and measure performance on a number of different tasks.

One test of knowledge would be to explain how entities from the domainare related.  For example, a "button", a "link", and an "option" are all similar in that they are names of GUI elements.  We looked at ways to infer relationships between software entities from text aboutsoftware.  Specifically, we collected data from the Q&A websiteStackOverflow, and used NLP methods and statistical machine-learningmethods to find a set of relationships that were frequentllymentioned, and internally consistent. The quality of the top-scoringrelationship was good (as much as 90% for certain measures) whenevaluated by domain experts.

As another test, one might ask a person read some computer code and explain in English what it does--i.e., to produce meaningful comments for the code.  We built a system could produce English comments on a new program that it was presented with.  This system was developedu sing machine learning methods: it was trained on actual comments, written by programmers to explain code that they had written.  The learning method we used works by reading the program into a neural memory ("encoding" it), then producing the comment one word at a time, updating the neural memory after each word is produced ("decoding").The best-performing model for comment generation was a novel learning system we called "encoder-reviewer-decoder", which allows some additional neural processing to be performed after the source code isloaded, but before the first word of the comments are produced.  The encoder-reviewer-decoder model is intended to model a person's mentalprocess of thinking about the program after reading it, but before explaining it.  Although this model cannot produce good-enough program comments on its own, it is accurate enough to help a programmersubstantially, filling in enough of a comment automatically to reduce typing by over a third.

Another common way to test understanding for people would be to ask them to answer question about terms from the domain.  We looked at answering "fill-in-the-blank" or "cloze" questions, such as

  Django is a free and open-source web framework, written in ____,  which follows the model-view-template (MVT) architectural pattern.

(The correct answer here is "Python").  From StackOverflow, a web site with questions and answers about computer programming, we extracted thousands of definitions, and constructed over 35,000 cloze questions. These questions are very hard: two experienced, senior programmers answered only 46.8% correctly, on average.

We explored a number of approaches to answering these questions automatically.  The first approach modeled what a human might do.  The system first searches for documents that might contain the answer,then "reads" the documents returned by the search, and finally produces the answer.  To "read" the documents use existing methods called reading comprehension (RC) systems.  These methods performed reasonably well, with the best existing question-answering methods reaching accuracy of 32.1%.  However, further experiments showed than one could do about as well (accuracy of 33.6%) using another approach called a "neural language model" (LM), which simply guesses a plausible word to fill in the blank using general statistics about English sentences in the software domain.

Motivated by this, we explored a new approach to answering cloze questions about software. The insight we had was that often, related software entities (e.g., "Django" and "Python") were used together as tags for StackOverFlow posts. (Every posted question can be tagged by the user with a set of software entities, to make it easier for possible question-answerers to find relevant questions.)  Often entities that are semantically related are used together as tags ofthe same post.  We thus designed a new method specifically designed for answering Cloze (fill-in-the-blank) questions using this sort of text with tagged entities.

Our method is called CASE (Context-Adjusted Syntax Embeddings).  It's a  hybrid of an neural LM and a co-occurrence model, which predicts an answer based on co-occurrence with the entity.  This allows a useful "division of labor" between the two models.  The LM can predict the appropriate “type” of the answer entity based on the question syntax (in the example, a programming language is needed), while the co-occurrence model picks out the entity of that "type" based on co-occurrence with the term defined.  CASE far outperforms the other methods: it achieved an accuracy of 44.9%, which is close to the performance of human experts.

Last Modified: 07/26/2018
Modified by: William Cohen