Space-time video completion

Abstract

We present a method for space-time completion of large space-time "holes" in video sequences of complex dy- namic scenes. The missing portions are filled-in by sam- pling spatio-temporal patches from the available parts of the video, while enforcing global spatio-temporal consistency between all patches in and around the hole. This is ob- tained by posing the task of video completion and synthesis as a global optimization problem with a well-defined ob- jective function. The consistent completion of static scene parts simultaneously with dynamic behaviors leads to real- istic looking video sequences. Space-time video completion is useful for a variety of tasks, including, but not limited to: (i) Sophisticated video removal (of undesired static or dynamic objects) by com- pleting the appropriate static or dynamic background infor- mation, (ii) Correction of missing/corrupted video frames in old movies, and (iii) Synthesis of new video frames to add a visual story, modify it, or generate a new one. Some examples of these are shown in the paper. We present a method for space-time completion of large space-time "holes" in video sequences of complex dynamic scenes. We follow the spirit of (10) and use non-parametric sampling, while extending it to handle static and dynamic information simultaneously. The missing video portions are filled-in by sampling spatio-temporal patches from other video portions, while enforcing global spatio-temporal con- sistency between all patches in and around the hole. Global consistency is obtained by posing the problem of video completion/synthesis as a global optimization problem with a well-defined objective function and solving it appropri- ately. The objective function states that the resulting com- pletion should satisfy the two following constraints: (i) Ev- ery local space-time patch of the video sequence should