The goal of this SHF project was to develop viewer-aware mobile video-memory design techniques by integrating mobile viewer-awareness with hardware adaptation to achieve power optimization. It addressed a number of critical challenges to viewer-aware video hardware design from three aspects: (1) experimentally and mathematically connect viewer experience, ambient illuminance, and memory performance; (2) develop energy-quality adaptive hardware that can adjust memory usage based on ambient luminance so as to reduce power usage without impacting viewer experience; and (3) design a mobile video system for verification.

We conducted psychophysical experiments on viewer experience in different viewing contexts, which illustrated that if a video system is operating in poor light conditions, viewers can tolerate larger amounts of quality degradation, such that more least-significant-bits (LSBs) of video data can be truncated to save energy without impacting perceived video quality. We also assessed the degree to which human observers were able to discriminate between reference videos and bit-truncated versions that were rendered and displayed at various levels of objective quality as a function of the ambient-illumination level. Based on the experimental results, we developed expectation-based mathematical models to connect video quality and memory performance. The results of the numerical studies on embedded memory design showed that the developed models provide a useful and fast tool to enable optimal hardware designs.

We studied viewer-aware hardware-implementation schemes for dynamic energy-quality knobs with minimal effect on viewer experience, such as voltage scaling, bitcell structures, LSB truncations, error correction code (ECC) schemes, device sizing, hardening most significant bits (MSBs), and hardening parity bits. As an example, we developed a novel viewer-aware bit truncation technique that enables better visual experience while maintaining similar power efficiency. We also designed a flexible power-efficient video memory that can dynamically adjust the strength of ECC, thereby enabling power-quality trade-off based on application requirements. We developed a new adaptive ECC technique, which can effectively select three power-quality tradeoff levels for video applications: hamming code-74, hamming code-1511, and no ECC. Using 1,000 randomly selected videos, our results showed that the developed memory enables runtime quality adaptation with significantly reduced overhead and better video quality, as compared to existing techniques.

Additionally, the impact of video content on viewer experience was studied from the psychological perspective. Our research demonstrated the correlation characteristics between “banding distortion” to viewers caused by hardware noise and the areas in frames that exhibit low variance among pixel luminance values, which has the potential to enable content-adaptation opportunities for hardware design. Based on macroblock characteristics analysis and subjective video testing, two models, decision tree and logistic regression, were developed to enable effective connection of the video content to the hardware design process. We further developed a novel viewer-aware bit truncation technique that enables better visual experience while maintaining similar power efficiency. Based on the developed models and viewer-aware bit truncation technique, a content-adaptive video memory design with dynamic energy-quality tradeoff was implemented, which enables up to 33.31% power savings.

We also developed a Region-of-Interest (ROI)-aware video storage technique that takes advantage of deep learning to identify the most important areas to optimize the video output quality while saving power consumption. Based on this, we designed a content-adaptable ROI-aware video system to support general videos, and conducted system testing, including power efficiency, number of truncated bits, output quality, and area overhead, which shows that this proposed memory enables run‐time quality adaptation with significantly reduced pixel bits and further power savings, as compared to existing techniques. 

The outcomes of this project have been widely disseminated to the research community and general public. This project directly trained over ten graduate students and undergraduate students (funded through supplements), including four underrepresented (female and minority) students. Among them, one MS and three PhD students have successfully defended their theses/dissertations. The team published five journal papers at prestigious venues such as IEEE Transactions on Very Large Scale Integration (VLSI) Systems, IEEE Transactions on Sustainable Computing, and IEEE Access. The findings of this project have been integrated into the PIs’ existing undergraduate and graduate courses; and the project further promoted K-12 STEM and computing education community outreach through the team’s organized outreach events and on-going NSF-funded RET award.  

Last Modified: 01/26/2023
Modified by: Na Gong