Never ending match cuts
Context:
Match cuts, being the art of transitioning from one scene to another while continuing the context in some form, are a common trope in filmmaking. Keeping the context using a match cut may be archived by keeping a similar frame-composition, motion-path or audio backdrop. Match cuts may be spatial, temporal or metaphorical as long as continuity may be perceived.
From cinematic classics like Stanley Kubrick's 2001 A Space Odyssey to modern music videos like Radiohead’s Daydreaming match cuts create transitions without stopping the general flow of an audio-visual impression.
Concept:
Match cuts are usually crafted by hand with care and attention to detail. In contrast to that I will try to create match cuts programmatically and in the spirit of “Me and the Machine” use an algorithm to find fitting cuts in found footage videos to create an endless stream of matching videos. The idea is to use the endless stream of audio visual content of the modern day world to create something new and unique that looks and feels continuous and might resemble something handcrafted with context. Due to technical constraints the techniques used might vary, but if possible, I would like to use all forms of match cuts from simple audio matches to complex contextual or even metaphorical matches.
To further give the work meaning, I might limit the pool of videos to choose from or possibly even include interaction by the observer, while keeping the idea of an endless audio-visual stream.
Approach:
To achieve this endless stream continuous image and audio analysis is needed. Both of theses tasks are of high computational cost and no easily done in a live-context. Visual match cuts might be programmatically solved by comparing the last frame of a chosen sequence with frames of another video using different approaches.
The first approach might be as easy as comparing histograms, which is fast, but is only really feasible for colored imagery and might fail to guarantee continuity. Feature matching might be possible for finding similar image compositions and or things of similar shape in two frames. This method might be able to recreate metaphorical and spatial match cuts.
Deep Ranking might give best results in finding similar images, creating the feeling of continuous imagery, but feasible solutions are scarce and training a neural network takes time and a lot of prepared data. This approach might still be possible using an existing framework or image-comparison API.
Using audio feature extraction, classification and segmentation one might be able to find similar audio clips in videos thus creating audio match cut.