SSBM + Machine Learning

BACKGROUND

With COVID forcing me to spend more time at home, I had a whole lot more time on my hands and I wanted to keep my mind occupied during quarantine. One project that I had seen on GitHub was Adam Spannbauer's ssbm_fox_detector, which uses machine learning to track a specific video game character on a tournament recording. I felt like the project's focus was narrow, and wanted to expand it to include all of the characters in the video game, not just one!

PROBLEM

To accomplish my goal, I needed to think BIG. I needed to accomplish three things:

1. Get my hands on as many decent-quality tournament videos as I could find.
2. Draw rectangles around each character visible on the screen on each video, and label each rectangle with the character's name
3. Feed all of my data into a program, and teach the program what each character looked like.

SOLUTION

PART I: GETTING THE DATA

I went through a lot of trial and error figuring out how to get my hands on as many videos of Super Smash Bros Melee tournament matches as I could find. Ultimately, the approach I settled on was downloading those videos from YouTube, Twitch, and Vimeo.

Luckily for me, passionate fans of the Super Smash Bros. franchise had already built vods.co, a “comprehensive video archive of competitive gaming matches”.

This site kept track of:

• The names of the people competing in a match
• The characters being played in the match
• The link to a video of the match

I needed to go through every match on the site, find the corresponding video, and download it onto my desktop computer. It'd take too long to do that by hand (there's tournament matches dating back to 2004), so I wrote a simple script that did that for me! All the program did was pretend to be a regular person using their computer:

• Click on each page of matches
• Click on each individual match on the page
• Extract the necessary data from the page
• Download the tournament video, keeping track of everything in a big spreadsheet.

After I had the videos downloaded, I split each video into as many pictures as I could fit on my computer. Now all I had to do was draw rectangles around the Smash Bros. character in each picture!

PART II: PLEASE, NO MORE RECTANGLES

I downloaded hours and hours of videos, which transformed into hundreds of thousands of pictures. For each picture, rectangles needed to be drawn around all of the characters present on-screen. This was a lot of rectangles, too many for me to draw on my own by hand. I needed some help.

Luckily, I had some awesome friends who told me about a process that would help me speed up my rectangle-drawing. The process is called bootstrapping, and to simplify it as much as possible, here's how it works:

1. I needed to draw roughly 1,000 rectangles around characters by hand.
2. I then needed to feed the rectangles into the program that I wanted to draw perfect rectangles.
3. I pretend like the program knows what it's talking about, and give it some pictures it hasn't seen before, and ask it to draw rectangles for me. Since the program doesn't know what the characters look like, it makes some pretty bad guesses. But it can draw bad rectangles really fast, which is better than nothing!
4. I go through the program's bad rectangles, and refine them to an acceptable level.
5. I combine the new rectangles with the rectangles created thus far, and repeat.

As time goes on, my program gets better at drawing rectangles, which means I have to draw fewer rectangles by hand. The process looks a lot like this:

Now I just rinse and repeat, over and over, until I have enough rectangles!

PART III: THE EASY PART

I'm not a machine learning master, by any means. Rather than spend a ton of time learning about machine learning and get really in-depth about the subject, I tried to keep my knowledge shallow enough to move fast. So I went online, found a tutorial, and ripped out the parts I needed to get the job done. All that remained was to see how the results shook out!