This project produced a number of contributions in the semantic representation of images. This is a novel image representation, which replaces the space of image pixels by a semantic space, where each dimension represents a visual concept. A vocabulary of visual concepts (objects, object attributes, scene classes, etc.) is first defined and a classifier is learned to detect the different concepts in the image. The image is finally represented by the vector of probabilities of containing the concepts in the vocabulary. This is known as the semantic multinomial (SMN) descriptor of the image. The process is illustrated in Figure 1.

The project showed that this  representation is superior to the classical representation of the image by pixels, or features derived from those pixels, for a number of image processing operations.

Image retrieval: This addresses the search for images within an image database. The standard paradigm is query-by-visual-example (QBVE), where a vector of features is derived from each image and used to measure image similarity. Database images are then ranked by similarity to a query image. The project has introduced the query-by-semantic –example (QBSE) paradigm, where the feature vector is replaced by the SMN descriptor. This corresponds to measuring similarity between images in the semantic space. Since this space has a higher level of abstraction than that of classical image features (colors, textures, etc.) it enables retrieval systems with higher generalization ability and robustness.  This is illustrated in Figure 2, where we compare the images retrieved  by QBSE and QBVE, for a common image query. For QBSE, we also show the concepts of highest probability for each image.  Note that the classic QBVE approach tends to return images with similar colors and textures and does not work well for complicated scenes.  On the other hand, the QBSE approach now proposed returns images that contain similar concepts, e.g. peoples and  buildings, to those in the query. This allows the retrieval operation to succeed even when these concepts appear in different sizes, positions, colors, etc. In result, the retrieval operation mimics the similarity judgments of people much more closely than under QBVE.

Cross-modal retrieval: One difficult problem in multimedia is to bring together information from multiple modalities. For example, how does one search an image database for the best match to a query text. The semantic representation provides a universal solution to this problem. Since, the representation is based on concept probabilities, not pixels, sounds, or characters, it provides an abstract space where information of multiple modalities can be equally projected. After all, concept classifiers can be built for images, sound, speech, music, or text with nearly equal difficulty. This is illustrated in Figure 3, for an example involving image and text data. Images and texts are initially represented in an image and text space, respectively. A common semantic vocabulary is defined, and a set of concept classifiers built on each of these spaces. This allows the mapping of images and texts  into a common semantic space. In this space, it is trivial to measure similarities from data of different modalities, e.g. find the text article that best matches a query image, or vice-versa.

Action and event recognition: Video classification requires reasoning in terms of semantic events. Figure 4 illustrates this with an example of video from TV coverage of the Olympic games. At the highest level, the video can be broken down into the events “context shot,” “pole vault,” “triple jump,” and “100 m dash.” Each of these concepts is itself defined in terms of semantic properties of lower level. For example, the “long jump” event is visually simila...

Please report errors in award information by writing to: awardsearch@nsf.gov.