Human Tetris Video Object Tracking

High Level Design

Almost all major gaming consoles are moving to user input systems involving tracking all or part of a player’s body (see Nintendo Wii Remote, PlayStation Move, and XBox Project Natal). Our project setup is similar to Project Natal, which Microsoft claims to be a “controller-free gaming and entertainment experience”, using a single sensor device (with at least camera and microphone) placed by the TV. We have demonstrated the feasibility of a basic motion-controlled gaming system using low-cost parts.

The system is based off of an ATmega644 microcontroller with two peripherals: a high-speed flash analog-to-digital converter, and an onscreen display device for video overlay. With just 4 kilobytes of RAM and a 20MHz clock speed, the MCU needs these peripherals to take some of the load off of its processing capabilities. Since we wanted to provide a real-time camera feed on the display (like a “mirror” for the player), and since the ATmega644 is not fast enough to generate a full color NTSC signal, we decided early on that we would use a color video camera module which outputs an NTSC signal directly.

The system accepts a color NTSC video signal, filters out the DC component, and samples the video stream at real-time speed. This sample is stored as a grayscale image in the MCU’s memory which can be processed by the application. The onscreen display overlays shape information and game data for user interaction.

High-level block diagram

Based on this system, we have implemented a video arcade game called Human Tetris, based on a Japanese game show event. In the game, contestants are faced with an oncoming solid wall with a shape cutout. The players must fit through the shape, or else they are thrown back into a pool of slime. A few clips from the show are shown in the YouTube video.

About half way through the design stage, we realized that a custom PCB could be very beneficial to our project. After getting approval from Bruce Land, we decided that we would attempt to design and build a custom PCB after we had finalized our hardware design, budget and time permitting.

After we received and soldered components to the PCB, we found one small connection problem on the NTSC input circuit, which destroyed the signal. The resistor’s lead in the RC circuit had to be moved to the other side of the capacitor [corrected PCB design file in schematics]. After a bit more debugging, the PCB worked perfectly. It essentially eliminated all of the noise and other issues we were having with our hardware. There were no longer any bad reads from the ADC or bad writes to the OSD. Everything behaved as it was designed, despite the high frequency signals we were sending between components.

Software

Our software consists of several major sections, corresponding to various hardware pieces and the various tasks required to achieve our goal. The first thing we developed was the driver for the on-screen display device, since that would be our only real output. Then we worked on getting video signal input. With input and output functioning, we moved on to data processing. Once we had a handle on how many resources we had left (in terms of memory and cycles), we moved onto writing the logic for program flow. With a simple shape-tracking library now at hand, we fleshed out the test code into Human Tetris and added Brick Breaker as a quick demonstration of the shape-tracking code’s potential.

SPI Driver for On-screen Display

Talking to the MAX7456 On-Screen Display through the ATmega644’s Serial Peripheral Interface proved to be an interesting challenge. Thankfully, someone had written an open-source library for doing so based on the ATmega162 architecture. Porting that to run on the ATmega644 was just a matter of reading data sheets and swapping register values for macros. However, the open-source library was based on bit-banging. Since Bruce Land covered using the ATmega644’s hardware SPI in lecture, we upgraded our driver to use that. Writing to the SPI works very well, and works as fast or slow as we can set it. Unfortunately, we never got SPI reads to work and are not sure why. But since we only needed the MAX7456 for output and it has such a well-designed interface, this was not a problem. One way reading could have helped would have been to confirm writes, since noise was a major problem prior to using the PCB. To combat the noise, we just reset all configuration options right before every frame, and hoped for the best on transmitting the frame data.

Our first surprise lesson in writing all of the datasheet came when we tried flashing a complete set of new characters to the OSD’s EEPROM character memory. Hardware constraints limit this action to only 8 characters at once (per on-off cycle). Since our output goals involved using the OSD to display much more than text, we needed a better option than programming and booting it 32 times over every time we altered our OSD images. Hooking up the OSD’s reset line to an output that we could toggle allowed us to use one run of the microcontroller to program the entire character memory.

Custom characters themselves were a cinch thanks to the same person that wrote the open source SPI driver. There exists a web page (see references) that will take a PNG image of all 256 characters and return a C header file of character data ready to be used by the open source driver. Our only issue here was that for some reason having an array of PROGMEM array pointers caused all the data to land in the Data memory, which was way above our 4kB limit. Using a rather long function to load the character data on a per-character basis solved the problem.

Part of our need for custom characters came from one of the OSD’s limitations, it only displays 13 rows of 30 characters each in NTSC mode. Also each character pixel is either black, white, or transparent. But each character is 18×12, so NTSC “pixel” level display is still possible. We split each 18×12 character into a 3×2 grid of 6×6 sub-characters, giving us a 39 x 60 symbol screen resolution. 6 sub characters means 64 possible combinations of them, so we wrote makedots.m in MATLAB to take a 6×6 pixel symbol and output all 64 combinations as an image ready to send to the character map conversion website. By keeping them in a clever order in PROGMEM, we could get our desired 18×12 output character by using 6 bit flags per character and then just adding the address of the blank to the sum. To get a 18×12 block from 6 pieces of 6×6 data, we just take the majority vote of the 6 pixels, with ties going to blanks.

NTSC Sampling

As one might imagine, getting a video signal into the ATmega644 proved to be quite a challenge. Talking to our flash ADC was rather trivial, since its only input from the micro controller was a clock. Things got less trivial when we realized the suggested clock minimum frequency was 5MHz, which is 1/4 of our own clock speed. We set up Timer 2 to control an output bit, so it could run independently of the CPU. Prior to using the PCB, we setteled for 1.25MHz as a balance between noise and temporal resolution. Once we had the PCB, running the ADC clock at 5MHz worked best.

NTSC signal behavior.

With the clock set up, we had an 8 bit digital version of NTSC flying at our MCU. If we tried to read it all in, we would be out of RAM in 400 microseconds. Down-sampling was a necessity. 39×60 pixels not only down-sampled nicely to the OSD, it took up just over half our RAM. So we can only hold one “frame” at a time. Loading 39×60 out of a 242×338 signal meant sampling roughly every 6th pixel in every 6th line of an NTSC frame. Lucky for us, our OSD not only output NTSC, it output the horizontal and vertical sync pulses. We connected those directly to the ATmega644’s hardware interrupts, so we could get fairly accurate responses to the start of every frame and the start of every line. The “porches” gave us time to context switch to the ISRs and do the book keeping necessary for down-sampling. 60 samples on a 51.5us line of data meant one sample every 0.85us, which is 17 cycles. To achieve this level of accuracy, we inserted NOPs into our C code until things lined up correctly. We also used microsecond delays and NOPs to center or reads along the visible data part of the horizontal line.

To get the sampling perfect, we needed something better than the imperfect display of our NTSC output. We connected up the UART and wrote a method to dump out a frame of data on to it. However, dumping a frame of data takes almost a minute, so we had to make it due so only after a button push. This made for excellent comparison of test images to read results, which were visualized using MATLAB. We also wrote a MATLAB function to read directly from the UART, which was sometimes more convenient. Viewing our raw data on the computer helped us notice when we had only a few pixels of either porch data or bad reads.

Input Filtering

A few pixels of bad reads would have been okay, but prior to using our PCB we had more than a few bad reads. There was substantial noise all over the circuit, and since we were mostly just moving data around, this was a big problem. We tried out several different approaches to filtering. First, we screened out all bad pixel reads and replaced them with the average of their neighbors. However, bad reads often occurred in clusters, so this did not work. Using a fixed gray value as a replacement worked better. Between down sampling and the noise, our output had a lot of flickering elements. Applying a simplified Gaussian Blur gave smoother results, but that went against our general goals of shape-matching. Averaging two frames of input together would have been great, but that would have required more RAM than we had. We tried out sampling more lines (so higher vertical resolution) and averaging on the fly, but the PCB arrived and squelched noise before we could get the timing right in that code.

Output Filtering

The PCB result was sharp and flawless, but the down-sampling effect was still causing an annoying flicker. Not wanting to have to slow down our game(s) just for the sake of flicker, we came up with a method of filtering our output based on our previous output, a moving average box filter of sorts. We still didn’t have the memory to duplicate even a 13×30 output frame, but by repeating characters across many addresses we could use the address itself to store information. Looking at the last two lines of our character set, you can see we increment the address when there is positive info and decrement it when there is negative info. By capping the ends (so as to not overflow or underflow), we can display a nice running average over time with minimal memory and cycle cost. Our increments and decrements are set to 3, so it takes less than 3 frames for white to black changes and it takes at least 3 positives or negatives in a row to create a noise blip.

To avoid output flicker from frequently updating the OSD, we don’t flush our output buffer to the OSD until the end of our main loop. Keeping this output buffer around is also what lets us do time-averaging of our output to avoid flicker. Another flicker that can appear is a nauseating collection of video signal artifacts when the OSD is displaying characters that are pixel level grids. For this reason our standard background is a loose 1 out of 4 pixels instead of a tight checkerboard pattern.

Timing

In order to run our games, we needed some sort of time base. Since we had a 20Mhz crystal and 20 is not a power of 2, we had to use Timer 1 to get a time base on the order of milliseconds. We settled on a 5ms time base as a balance between ISR executions and accuracy. That time base controls the game state and timers, but the main loop is actually triggered by the NTSC sync pulses. The vertical sync ISR activates the horizontal sync ISR for reading in a frame if a flag is set and the horizontal ISR sets the flag down when reading in a frame is complete. The main loop does the opposite, running when that flag is not set and raising it when finished. This way every time main executes it has fresh input. We run our tracking mode as fast as possible and our games at half that in order to make the OSD flicker less during gameplay.

We have 2 buttons for input, “toggle” and “enter”, which are sufficient for a menu system. We run a de-bouncing flag-raising state machine to get meaningful input from sampling them at around 30Hz. We run a heartbeat LED at one second per cycle, so we can tell if things are working fine or not. In addition to the heartbeat LED, we keep a “Game Timer” at a 5-millisecond time base for game play purposes. We also sample the on-board ADC at 1Hz to get specified black level, which can be set by a trimpot to tune the game(s) to different environments. This live tuning is a bit more useful than just viewing exact values from a UART frame dump and guessing at a good threshold for detection.

Menu and System State

Our menu allows users to choose between the three games and the tracking mode. One button is used as a toggle to go to the next option and the other button is used as enter to select an option. The menu is a simple state machine. Each frame it displays the title, our names, and all the options with the current selection highlighted.

The state of the system is another state machine with states for menu, each of the games, tracking mode, and a win/lose screen. Each “frame” through the main loop activates the code for the appropriate state. The initial warning message is displayed for a few seconds the first time the menu state is activated, which is when the system turns on.

Games

The game state machines are based on the 1 millisecond base of “Game Time”, and the logic is independent of the frame rate. This means we can speed up or slow down our frame rate as needed and not affect the game mechanics. The main state machine simply loads the input data from the video stream then calls out to game logic for each frame. In our game modes, we display the player’s body as a collection of 12×18 character sized blocks for aesthetic value and decreased noise, as opposed to the 6×6 dots used for tracking mode.

Our Human Tetris game does the following. Each shape is displayed for 5 seconds. It is loaded into memory once, then player is displayed each frame using the moving average code (described above) to preserve the shape information. Once the shape and player are in the output buffer, simply looping through that buffer tells us how many “blocks” of the player’s body are outside the shape. If more than a certain tolerance are outside, the player loses. If not, the game advances to the next shape until a victory screen is reached. The number of misses is only checked at the end of a shape’s time.

We implemented a rather nice content pipeline for the Human Tetris shapes. One can place an arbitrary amount of images in a directory, and then run the MATLAB script imgs2progmem to get out a C header file with a single array holding all the shape data in program memory. The only restrictions are the input images must be 39 x 60 and black and white. It should be noted that the shapes are effectively stored as a bit vector (1 bit per block) in the C file. This compresses each shape to a mere 293 bytes, so we could fit a lot in program memory. The artist should keep in mind that the game currently displays shapes and players on a 13×30 grid (via averaging), so care should be taken when drawing the 39×60 sized shapes.

To demonstrate the extensibility of our shape tracking code, we also implemented a few other games: