Perceptual Focus Prediction & Video Transcoding

Acuity Window. Human has only 1-2 degree sharp vision around the center of vision. The Lab is developing a roving acuity window that can keep over 90% of high acuity gazes confined using anatomical mechanics (left). The Blue Angel air show’s movie (top) is tracked and perceptually encoded in real time.

Perceptual Focus & Object Driven Digital Video Transcoding

This research is engaged in a video compressions and transmission scheme with perception engineering. We are developing an intelligent scientific and medical image transmission scheme for interactive high-resolution still/video image transmission which can use viewer’s perceptual feedback to deliver a high compression factor but with greater perceptual quality.

We are using a number of schemas to derive viewer visual perception with high precision. In one schema we have incorporated automatic eye-glance tracing mechanism to observe viewer’s perceptual focus. Then we use the mechanical model of eye dynamics and advanced prediction filters to accurately estimate viewer’s window of focus on the visual plane.

In another schema we have further incorporated real-time moving object analysis inside video stream to further identify the foci of attention. Based on this feedback, blocks in the view frames are assigned variable bit-rate and quality constraints.

MM: Oleg V. Komogortsev, and Javed I. Khan, Eye Movement Prediction by Kalman Filter with Integrated Linear Horizontal Oculomotor Plant Mechanical Model, Proceedings of the Eye Tracking Research & Applications Symposium, ETRA 08, Savanna, GA, March 26-28, 2008, (accepted).

MM: Oleg Komogortsev & Javed I. Khan, Predictive Compression for Real Time Multimedia Communication Using Eye Movement Analysis, ACM Transactions on Multimedia Computing, Communications, and Applications, TOMCCAP, 2007 (accepted, October 2007).

MM: Komogortsev, O., Khan, J., Kalman Filtering in the Design of Eye-Gaze-Guided Computer Interfaces, Springer Lecture Notes in Computer Science, Proceedings of the 12th International Conference on Human-Computer Interaction (HCI 2007), , Springer Berlin / Heidelberg, Beijing, China, July 22-27, 2007, pp.679-689, [SPRINGER] [PDF]

MM: Komogortsev, O., Khan Javed, Perceptual Multimedia Compression Based on the Predictive Kalman Filter Eye Movement Modeling, Proceedings of the 14th SPIE/ACM Annual Multimedia Computing and Networking Conference, MMCN'07, San Jose, California, Part of Electronic Imaging Symposium, Sponsored by SPIE and IS&T In cooperation with ACM Multimedia, Proc. SPIE Vol. 6504, 650406, Multimedia Computing and Networking 2007; Roger Zimmermann, Carsten Griwodz; Eds., Jan 28- Feb 1, 2007,pp.1-12. [SPIE] [PDF]

MM: Javed I. Khan, Komogortsev O., “A Hybrid Scheme for Perceptual Object Window Design with Joint Scene Analysis and Eye-Gaze Tracking for Media Encoding based on Perceptual Attention”, SPIE Journal of Electronic Imaging, SPIE—The International Society for Optical Engineering, Journal of Electronic Imaging 15(02), 023018, April 2006. [SPIE] [KhKo06]

MM: Komogortsev, O., Khan J., “Perceptual Attention Focus Prediction for Multiple Viewers in Case of Multimedia Perceptual Compression with Feedback Delay”, Proceedings of the Eye Tracking Research & Applications Symposium, ETRA 06, San Diego, March 27-29, 2006, pp.101-108. [KoKh06]

MM: Oleg Komogortsev and Javed I. Khan, Predictive Perceptual Compression for Real Time Video Communication , Proceedings of the ACM International Conference on Multimedia, ACM MM04, New York, October 2004, best student paper contender, pp.220-227, (GS(2)). [KoKh04]

MM: Oleg Komogortsev and Javed I. Khan, Contour Approximation for Faster Object Based Transcoding with Higher Perceptual Quality, Proceedings of the International Conference on Computer Graphics and Imaging, CGIM 2004, Kauai, Hawaii, USA - August 2004, pp.441-446. [KoKh04b]

MM: Javed I. Khan and Zhong Guo, Flock-of-Bird Algorithm for Fast Motion based Object Tracking and Transcoding in Video Streaming, Proceedings of the 13th IEEE International Packet Video Workshop 2003, Nantes, France, April 2003. [KhGu03]

MM: Khan Javed I. & D. Yun, Multi Resolution Perceptual Encoding for Interactive Image Sharing in Remote Tele-Diagnostics, Manufacturing Agility and Hybrid Automation -I, Proceedings of the International Conference on Human Aspects of Advanced Manufacturing: Agility & Hybrid Automation, HAAMAHA'96, International Ergonomics Association (IEA) Press, Maui, Hawaii, August 1996, pp183-187. [PDF]

MM: Khan Javed I. & D. Yun, Perceptual Focus Driven Image Transmission for Tele-Diagnostics, Proceedings of the International Conference on Computer Assisted Radiology, CAR'96, Elsevier Press, June 1996, pp579-584. [PDF]

Selected Publications

More Publications >>

The encoder driven by a dynamic control algorithm adopts to these specifications in real-time with redistribution of spatial resolution and the bit allocation. It already saves 20-40% bits over conventional MPEG.

The research has contributed in improving the challenging short term prediction problem of eye-glance and has applicability in other areas such as human computer interaction. Indeed we have demonstrated how predictive eye-glance can be used for non-verbal communication in the playing of video games.

The result can radically accelerate remote projection of high definition and wide aperture video and graphics systems. We have extended the technique for multi-viewer case.

The work has been supported by DARPA. The paper on the topic has been best paper contender in ACM Multimedia 2004.