Over the decades, humans have been learning computer language. Natural Language Processing (NLP) means computers are now learning how humans use language, i.e., the comprehension by computers of the structure and meaning of human languages, allowing users to interact with computers using natural sentences.  This task is not easy. One of the challenges is to "understand" meaning, e.g., draw inferences, derive sentiment relations, understand figurative language.

This project  has been concerned with developing an algorithm for detecting idiomatic expressions. Many idiomatic expressions are ambiguous. Ambiguities in semantics arise when multiple interpretations are possible. For example, hit the roof can be interpreted literally or idiomatically depending on the context: sales hit the roof vs. hit the roof of the car.  

The research team has explored various linguistic properties of idiomatic expressions cross-linguistically. In our earlier work, we approached the problem as outlier detection: literal expressions are semantically related to the rest of the context, while words that constitute an idiom appear inconsistent with the rest of the context. Our technique incorporated the following observations: (1) A sequence with literal meaning has many neighbors, whereas a figurative one has few. (2) Idiomatic expressions should demonstrate low semantic proximity between the words composing them. (3) Idiomatic expressions should demonstrate low semantic proximity between the expression and the preceding and subsequent segments. 

Later we refined our hypothesis and decided to work with text at the topic level structure. Informally, topics are just clusters of similar words. A document usually contains several topics. The idea is that topic words in a given text segment, such as a paragraph, are less likely to be a part of an idiomatic expression. Our additional hypothesis is that contexts in which idioms occur, typically, are more affective and therefore, we incorporate simple sentiment analysis, focusing on the intensity of emotions. This approach can be still viewed as outlier detection. This approach allowed us to differentiate idioms from literals using local semantic contexts with better accuracy.

We further developed our approach and captured the intuition about cohesive ties, idioms and local context by using a vector representation of words, i.e., we created a representation for words that capture their meaning, semantic relationships and the different types of contexts they are used in and used it in our classification task. The performance results have improved significantly. We applied our approach to English and Russian, the two languages from two different language families, Germanic and Slavic, respectively, and whose structural and morphological properties are different enough to test if our method is (relatively) language-independent. Our experimental results were positive. 

The results of this study is a method for idiom detection that does not rely on resource-heavy linguistic technology, such as syntactic parsing, part-of-speech tagging or manual annotation. Our algorithm manages to capture the contextual differences in which literal and idiomatic expressions occur. This technology can be used in many other applications, such as machine translation, language understanding systems, sentiment and emotion analysis, among others. 

A number of undergraduate and graduate students had the opportunity to participate in active research and gain experience in working collaboratively on an interdisciplinary problem. The project involved students from a variety of disciplines: linguistics, computer science, literature, and mathematics. 

Last Modified: 06/01/2018
Modified by: Anna Feldman