There's a lot of info here, but if you just want cool stuff, you can scroll down to the Web Site and Commentaries section at the bottom.

After playing Soul Calibur just about every day for over three years, my coworkers and I grew tired of wondering what our stats were in the game.  We needed a system that could track our stats but that wouldn't be annoying to use.  We tossed around a couple ideas but decided to try out a system based on video capture to see if it would be able to analyze the video feed from the GameCube and look for certain key events to occur -- character selection per controller, fight start, end game events (KO, Perfect, Ring Out, etc), health changes, and win changes.  Well it turns out that it worked, and with those pieces of data captured off of the video combined with a game manager that is in charge of the circuit rules (maximum allowable time for play, maximum number of fights, etc) and the rotation (winner stays, loser stays, etc), the computer is able to track player stats in realtime.  Toss in a database for saving the stats, a web front-end for analyzing stats and starting circuits, voice synthesis using AT&T Natural Voices for commentaries, and some cool API's, and you've got yourself a tracking Soul Calibur with live computer-generated commentaries responding in realtime to events happening in the fight.  That's Soul Calimeter.

The key feature of Soul Calimeter is its ability to take the live gameplay video and look for important events.  The system tracks events such as character selections on the character selection screen, fights beginning, health increasing and decreasing, wins increasing, and end game events.  Each of these events is useful in determining various statistics both for comparison of player performance as well as fodder for the computer commentators (Bob and Nancy) to talk about.

The Video Capture system was built on the cheapest piece of crap capture card that I could find -- which turns out to be a $25 WaveWatcher card with a Bt878 chipset.  The most important part was getting a reasonably clean video feed -- the SVideo input on the card proved to be pretty important (roughly 3x more reliable detection of video events as compared to composite input).  The app was originally implemented on Windows using the Java Media Framework's standard VFW-based video capture API's.  The VFW implementation proved too unreliable, so a JMF wrapper was written around Microsoft's DirectShow API's to provide for more dependable capture as well as to allow us to programmatically change the video source on the card (which JMF/VFW doesn't support).

The soul calimeter box is a 1Ghz Athlon with 512M of RAM running Tomcat as the webserver.  The stats database originally was MySQL, but MySQL's current stable releases did not support the complexity of query capabilities needed to run the stats analysis.  It has since been run on Mckoi (a pure Java database that kicks ass) for development and Oracle 9i in "production".

Speech synthensis for commentaries is built with the Java Speech API on top of CloudGarden's TalkingJava SDK.  The voices are AT&T Natural Voices, which sound awesome, but aren't very expensive.

The stats was really the main thing we wanted out of Soul Calimeter.  We wanted to know who's kicking who's ass, how badly, and with which characters.  Once we got the basic system, we realized the amount of analysis is pretty endless.  We can currently run the following stats in realtime:

Duration By Person - show the number of minutes that each person has played

Fights By Person - shows the number of fights each person has played

Fights By Character - shows the number of fights each SC character has played (turns out the Random option really isn't so random)

Win Percentages by Person - shows the win/loss percentages for each person

Win Percentages by Character - shows the win/loss percentages for each SC character

Win Percentages By Person Matchup - shows the win/loss percentages given two people player eachother

Win Percentages By Character Matchup - shows the win/loss percentages given two characters against eachother

The cool part is that we track down to each health change, so there's tons more anaylsis to be done and we are building up a huge database of tracking data, so we can add new stats at any time that can run on the historical data.

Currently, the computer can't identify which people are playing and which controllers they have (this is being worked on), so the computer has to be in charge of handing out the controllers to the appropriate people after each game.  The computer has several different sets of circuits -- a timed circuit, a race-to circuit (like "race to 5"), never-ending circuit, and a circuit that implements some custom rules we play by.  In addition, the computer has a rotation that determines who gets the controller after each game.  The rotation implements rules like Winner Stays, Loser Stays, etc.

This is where things start to get cool.  Here are some screenshots from the Soul Calimeter web interface:

The Main Screen - This is the front screen from Soul Calimeter that has a couple high level stats on it.

Circuit Setup - When you want to create a new circuit of games, this is the screen you use.  This allows you to pick the people that will be playing during the circuit and the ruleset you want to use for the games.

Most Recent Game - This shows the most recently played game, or if there is a current game, this shows realtime data from the game as you reload.  These screenshots are from my local version, so the data shown is from one of my testcases rather than real data, but you get the point. Oh yeah -- "controler" is spelled that way because it sounded better when the voice says that spelling.

Stats - And of course, where the rubber meets the road.  These are the stats.  There's more data analysis to be done than time in a day.  We actually have more stats implemented than are shown here, but they're currently only used by the commentators and haven't been hooked up to the stats page.

Random? - And when you choose "Random", if you've ever felt like it's not really random then you're probably right.  Here's the data we've collected on it.  You can see it's pretty terribly distributed.  We'll just have to see if it levels out over time.

OK -- I think the commentaries are the coolest thing, so I've kept them till last.  The API for Soul Calimeter was designed so that it would be really easy to add in new commentaries.  Currently we only have two AT&T Natural Voices (Mike and Crystal, which we have dubbed Bob and Nancy), so we only have the witty banter of the two of them, but it's designed to support an arbitrary number of "commentators".  The commentators can respond to any of the trackable events -- fights starting, games ending, KO's, Perfects, health changing, etc.  In addition, several commentaries are time-based, or triggered by an impending scenario.  A commentary can queue up a single comment by a particular commentator, or it can queue up a discussion between the two commentators.  An example commentary that occurs when one player is getting his ass kicked is "1: {fight.winner.person} just beat {fight.loser.person}'s ass.; 2: Just like I did to your mom last night {commentator1}.".  The {..} are variables that are replaced at runtime.  The 1:/2: specifies which commentator should speak (they can be randomly chosen to mix up the voices).

Some of the commentaries we've implemented include a choke alarm (if a player starts losing a huge lead), "finish him" (when a player has a tiny amount of health left), demanding a perfect (if one player has all his/her health and the other player is very low, the commentators can request a perfect and will respond pleasantly or with disappointment depending on the fight outcome), and stats commentaries -- these can be queued up to run at random times so that the commentators can point out that "Schrag with Ivy has never beaten Adam with Kilik", or that "Adam wins 53% of his fights", etc.

Lastly, commentaries can be generated off of JSGF grammars, which can be built up to generate incredibly complex commentary scenarios.  The grammars can include variable replacement like the example above.

Here are some sample recordings of commentaries.  These particular recordings are from my test cases as well, so they have some debug speaking at the front of the clips setting up the scenario.

Ass Kicking - This is a commentary that is triggered by someone getting his ass kicked

Choke Alarm - This is a commentary that is triggered by someone losing a huge lead (60% lead to a 10% loss)

Block Fest - This is a commentary that is triggered by at least 10 seconds without a hit on either side.  This one is kind of cool because the test case sets up an entire fight, so you hear a random stat at the beginning as well as some trash talk -- this is more like what a real fight sounds like (albeit compressed into a very short time period).

The system is pretty darn reliable.  We only know of about 10 out of our over 300 fights that have had the wrong character detected (it sometimes detects the character as "Random" if we click really fast through the character selection screen, because it doesn't have time to process the switch from the Random box to the randomly selected character).  As far as in-fight data, we very occasionally (even less than the 3% misdetection of characters) get a missed Fight Start event, so it thinks the second fight is the first fight, but it generally recognizes the win count properly anyway and in the worst case will just clean itself up when it ends up back at the character selection screen.  It tries to throw out game data in that case, since it is suspect.  There are a couple steps that we can take to clean up these last few accuracy problems -- specifically the occasional first start problem.

The biggest thing for us at this point is adding in new rulesets, adding commentaries, and adding new stats to track.  Some of the newer commentaries will actually choose discussions based on stats (for instance, it might make predictions based on the players, etc) rather than to just look at in-fight events or raw stats speaking.  We've also talked about having the commentators choose favorites and talk trash about the other player.  I already tell Bob to fuck off when he tells me that "to be the man, you have to beat the man", so the commentaries are pretty effective.  It's even more infuriating when you know when Bob or Nancy talks shit to you that it's based on hard facts.  So when Bob says you got your ass kicked, you really did get your ass kicked.

As far as releasing the app, we don't know what direction we want to go.  The video detection has only been tested on the card we have and with a GameCube.  Obviously the detection system will need to be made more resilient to deal with changes in color, noise, scale, etc as you move among cards and game consoles, but the basic system is there and we use it everyday, so we know it works.  Part of putting this page up is to see if we get any response at all from people about using this app for their own SC fights.  If there's no interest at all, we'll just happily track our own stats; but if it looks like other people might be interested, we'll take a look at making it more releasable.

Since it got submitted to Slashdot I figure I should go ahead and write up the latest and greatest info.

First I wanted to point out that Soul Calimeter has been a collaborative effort with my coworkers Adam Rabung and Mel Riffe.  Many thousands of fights have gone into perfecting the system :) (for work people -- just append "during lunch" to every sentence on this page)

The DirectShow/JMF interface (binaries and source) is available at java.net for anyone who is interested ( http://jmds.dev.java.net ).  It can be a little finicky if you don't shutdown everything just right, but it has been working really well for us (granted we don't actually shut it down ever).

The stats and commentaries have been expanded since I wrote this up -- there are about 3 or 4 more reports and an additional 3 or 4 commentaries than are listed here.

The last cool technology addition is that we have a much slicker system for identifying players now.  I ordered a set of iButtons for everyone (the ID-only ones) and a USB iButton reader (which has a really nice Java API -- I was shocked -- I fully expected it to suck).  So now whoever is playing can just put their iButton in the reader when they come in the room and it will lookup their ID in the db and add the person to the circuit (or remove the person if you put your iButton in again during an active circuit).  Before, we had to setup who was in the circuit using the web interface, which always sucked if someone had to leave early.  Now people can just logout when they need to leave and they get removed from the circuit on the fly.

The last piece I'd like to bring into the mix is the Zaurus w/ wireless so it's easier to get stats on-demand.  When you're playing, sometimes a discussion arises about a particular stat, and it would be nice to be able to just run the query right there.  Beyond that, I'm starting to run out of ideas :)