Quanfu Fan, Computer Science Department

Matching Slides to Presentation Videos

Video streaming is becoming a major channel for distance learning (or e-learning). A tremendous number of videos for educational purpose are captured and archived in various e-learning systems today throughout schools, corporations and over the Internet. However, making information searchable and browsable and presenting results optimally for a wide range of users and systems, thereby enriching learning experience remains a challenge.

In this talk I will present two core algorithms developed to support effective browsing and searching of educational videos. The first is a fully automatic and robust approach for matching slides to video with high accuracy. Built upon SIFT (scale invariant feature transformation) keypoint matching using RANSAC (random sample consensus), the approach is independent of capture systems and can handle a variety of videos with different styles and plentiful ambiguities. In particular, we propose a multi-phase matching pipeline that incrementally identifies slides from the easy ones to the difficult ones. We achieve further robustness by using the matching confidence as part of a dynamic Hidden Markov model (HMM) that integrates temporal information, taking camera operations into account as well.

The second algorithm is a non-linear optimization method (bundle adjustment) for accurately computing the projective transformations (homographies) between slides and video frames. Different from estimating homography from a single image, our method solves a set of homographies jointly in a frame sequence that is related to a single slide.

During the talk, I will explain the two algorithms in detail and demonstrate their usability in the SLIC (Semantically Linking Instructional Content) system.