Cool Idea #9764 - Recreating Audio With Video Clips to Produce a Video Collage

Recreating Audio With Video Clips to Produce a Video Collage

First let me explain the concept. Say "hello" out loud. Go ahead, people around you won't think you're weird. Maybe. An audio clip of your "hello" can be recorded digitally. It can be matched as a loose similarity to an audio clip of me saying "hello". You can experience audio recognition in technology like an automated telephone response system. Call UPS if you've never interacted with one. This "loose" relativity is useful, but my ideas are rarely useful. Only neat. What about "tight" relativity? What if your "hello" clip was digitally altered to sound like mine? What if I took 1,000 audio clips of people saying "hello" and cut tiny bits and pieces out of each one to make a best fit to approximate a recreation of your clip? This is the basis of my concept.

Here is a clip from Live Free or Die Hard that's an example of a video collage cut together to recreate a written speech.


Cool, eh? The writers made a script for the message to be broadcast, then cut bits and pieces of audio from presidential speeches and edited them together to create a video collage to tell the message. This is a perfect example of where I'm coming from. Only I want to take it much further.

In the movie, the process started with text and some editing crew manually searched for video clips that used the words from the speech. If this were to be automated, a computer would either:
- have to know how to sound out the original text in order to search audio clips for matches
- or, have some text document of all the words represented in the library of audio clips with which to match the search parameters.

Forget all that, let's do something easier. Let's just take in audio as a source and match it to audio clips. Take text out of the equation. As input, say our sample audio is you saying the words "don't fly through the air on a banana". Then, we give a computer access to 1,000 feature-length movies. The algorithm could search for each complete word in the audio of its library and put together clips that match at some forgiving percentage. An unoptimized algorithm would probably take a few days of computing time, but that ain't too bad. At the end of it, you would have a totally cool video of compiled clips where the audio is the words (or close to them) that you originally recorded. Much like the clip from Live Free or Die Hard.

I'm not done yet.

What about exact (well, >90%) replication of your audio input? What if we even wanted the output to SOUND like you? Instead of returning entire words that are loosely similar to what you recorded, what if the algorithm took tiny snippets of length 1/30 of a second and did a best fit? It would take far, far longer to complete, but the result would be a visual assault of video collage. Scenes of movies would be absolutely whizzing past your head, but the audio you hear would be scarily similar to your own voice! This would be very exciting to experience.

Another neat example would be to pass in audio of an explosion. The resulting clip would likely find millions of tiny clips of explosions from movies, all strung together to make a visual representation of the input.

I don't even want to think about the processing time for the precise reproducing algorithm, but what do you guys think about the loose fit algorithm? An application that could accept audio as an input and produce a collage of video clips matching the words?

RyanCast

Watch live video from GameTap Office on Justin.tv

interactive posters

Netflix Recommendations

Netflix is offering a million dollars to anyone who can supply a recommendation system that's 10% better than their currently deployed CineMatch algorithm. Example of why they are paying big $$$ for such an improvement:


They need all the help they can get.

It's an interesting problem to tackle. How do you identify the traits of a movie? Genre is the first that comes to mind, but certainly more is required to recommend a movie. Simple classification such as "Comedy" doesn't necessarily mean I will like Superbad if I enjoyed Tenacious D: The Pick of Destiny.

Based on the Netflix website, you can see each movie profile page has several specific categories, or sub-genres. I imagine the current CineMatch system uses these as a tag recommendation algorithm. For example, the page for There Will Be Blood lists these genres:

Drama
Indie Dramas
20th Century Period Pieces
Dramas Based on Classic Literature
Period Pieces
Dramas Based on the Book

As you can see, each sub-genre has its own landing page. If two given movies have a similar sub-set of genre tags and I liked one of them, it stands that I might enjoy the other. Let's go back to the image example up top...

Orange County	BSG: Season 1	Walk Hard
Comedy	Television	Comedy
Teen Comedies	TV Sci-Fi and Fantasy	Spoofs and Satire
--	TV Dramas	Contemporary Movie Musicals
--	--	Blu-Ray

Other than the "Comedy" link, do YOU see any other similarities? I don't.

OK, what else can movies be related on? What about cast and crew? Ah-HA! Orange County and Walk Hard have the same director and both feature actor Jack Black. Are those truly related criteria, though? Jack Kasdan also directed indie crime thriller Zero Effect. If Jack Black was in Zero Effect, would CineMatch have recommended it, too?

Again, movie recommendation is an interesting problem. I wonder if it could ever work like music recommendation engine Pandora. Check out what I mean by reading this page at How Stuff Works. Could we identify such specific things as plot elements as seen on IMDB.com pages? If I liked a movie with a memorable "peacefully gazing at the night sky" scene, could be there be any recommendation significance to a movie with a similar scene?

Anyone want to go for the million bucks with me? =)

p.s. If anyone can find one single similarity between BSG Season 1 and Walk Hard/OC, please comment!

I can't wait... /sarcasm

here's at least one good reason to have a kid.

HATE ATLANTA TRAFFIC

but it did inspire me to make interesting use of my phone's panorama feature...

Update City

I'm alive, I promise! Ever since Brawl came out, my need to blog has decreased. But no longer! The initial clutches of that warm temptress beckon me no more. I don't know if the game is not as good as Melee, my need to play hours upon hours of Smash has faded with time, or if living outside the college atmosphere demands less devotion to the sport. Whatever it is, after the first two weeks of Brawl, I don't feel addicted to it (this is good.)

And now for random updates:

- I spent last weekend with Lisa and others at a cabin in northern Georgia. It was great fun and I hope we make a tradition of the experience. Hot tubs are swell, the view was phenomenal, and there was much rejoicing. I'll never forget challenging myself to Seven-Eleven-Doubles... and losing.

- Saw No Country For Old Men with Jason. Overall, it was a good film. I was on the edge of my seat for 3/4, but then the movie took a sharp turn at Albuquerque and we were left with WTF? faces. If anyone would like to talk about the moral of the story, please leave a comment. If I had to take a guess, it would be that the inevitable self-destruction of society is a difficult thing for the oldest living generation to accept.

- Started watching a BBC mini-series titled "Jekyll". It stars James Nesbitt as a modern descendant of THE Dr. Jekyll. The guy can act, that's for damn sure. Highly recommend at least watching the first two episodes to see true character transformation on camera. If you're not at least intrigued by the first ten minutes of episode one, you're a boring person.

- Bought and finished God of War: Chains of Olympus. It's a good game, no doubt, but it lacks some of the grandeur of the PS2 titles. The puzzles and locales don't measure up to the kinds found in GOW and GOW2. The combat is spot on. The controls feel natural and the spells & weapons fit right in. If you have a PSP, I'd recommend renting it only because of its brevity.

- StarCraft is still awesome. I've been playing some games lately and it's crazy how I'm still improving, even after being out for ten years. That is certainly the hallmark of a solid game. If I can find a simple way to post some replay files, I'll link to them here. I am bouncing off the walls in excitement for the sequel. Here's hoping we get a release date or beta soon. I expect SOMETHING along those lines to be announced at WWI in Paris this June.

- I filmed another set of interviews for GameTap Watch this week. Look for my re-appearance there soon! I haven't heard anything back yet about the interview I did for the PBS show.

- Speaking of GameTap, there is a brilliant game coming out next Tuesday called Captain McGrandpa: Memory of the Forgotten. It's a GameTap Originals title that I highly recommend playing!

- Lisa is the best girlfriend ever. I'm keeping her. <3 <3 <3

- Rock Band! Boston track pack! Six Boston songs to play! Get it!!!