With COVID forcing me to spend more time at home, I had a whole lot more time on my hands and I wanted to keep my mind occupied during quarantine. One project that I had seen on GitHub was Adam Spannbauer’s ssbm_fox_detector, which uses machine learning to track a specific video game character on a tournament recording. I felt like the project’s focus was narrow, and wanted to expand it to include all of the characters in the video game, not just one!
To accomplish my goal, I needed to think BIG. I needed to accomplish three things:
I went through a lot of trial and error figuring out how to get my hands on as many videos of Super Smash Bros Melee tournament matches as I could find. Ultimately, the approach I settled on was downloading those videos from YouTube, Twitch, and Vimeo.
Luckily for me, passionate fans of the Super Smash Bros. franchise had already built vods.co, a “comprehensive video archive of competitive gaming matches”.
This site kept track of:
I needed to go through every match on the site, find the corresponding video, and download it onto my desktop computer. It’d take too long to do that by hand (there’s tournament matches dating back to 2004), so I wrote a simple script that did that for me! All the program did was pretend to be a regular person using their computer:
After I had the videos downloaded, I split each video into as many pictures as I could fit on my computer. Now all I had to do was draw rectangles around the Smash Bros. character in each picture!
I downloaded hours and hours of videos, which transformed into hundreds of thousands of pictures. For each picture, rectangles needed to be drawn around all of the characters present on-screen. This was a lot of rectangles, too many for me to draw on my own by hand. I needed some help.
Luckily, I had some awesome friends who told me about a process that would help me speed up my rectangle-drawing. The process is called bootstrapping, and to simplify it as much as possible, here’s how it works:
Now I just rinse and repeat, over and over, until I have enough rectangles!
I’m not a machine learning master, by any means. Rather than spend a ton of time learning about machine learning and get really in-depth about the subject, I tried to keep my knowledge shallow enough to move fast. So I went online, found a tutorial, and ripped out the parts I needed to get the job done. All that remained was to see how the results shook out!