All of the hardware and software setup details are below, but before I explain how it works, let’s see it in action:
How It Works
This is not a beginner project, so if you don’t have experience doing any Android development, you’ll need to be patient. Just getting your Eclipse development environment set up for Android development with the OpenCV and IOIO libraries took me a couple of hours, and I’ve been using Eclipse for about 10 years.
An Android app running on the phone captures video frames and processes them down to a lower resolution suitable for the 16×32 LED matrix. OpenCV is a powerful computer vision/image processing library, and there’s a version that runs on Android. I used the OpenCV library to convert the video frames to 16×32 pixel resolution to match the LED matrix. I also constrained the color space of the frames to 12 bit color. That is, each pixel has 4 bits each for red, green, and blue. That means that each pixel can have 16 different brightness levels of red/green/blue, yielding 4096 possible colors. In other words, all of the image processing is performed on the phone because it’s much more powerful than the Arduino.
The 16×32 12-bit image uses 1024 bytes of memory on the phone (2 bytes per pixel). The Android then uses the IOIO library to write this data out one of the IOIO board’s serial ports. Each frame starts with a two-byte frame marker 0xF0 0×00, then the bytes for the pixel values are written. The performance bottleneck is between the phone and the IOIO board. I can only write about 4 frames per second, even though the serial interface between the IOIO and Arduino is 115200 baud. Since each pixel really only needs 1.5 bytes instead of 2, I could pack the pixel data tighter to get perhaps one more frame per second, but didn’t think it was worth the trouble.
The green wire in the picture below is a serial connection from the IOIO and Arduino. The Arduino code simply reads the pixel values, using the frame marker to know when a new frame begins. The pixel values are written to the LED matrix panel using the Adafruit library for controlling the panel. Driving this matrix is no small feat for the Arduino, since the matrix panel does not do any PWM on its own; the Arduino needs to generate the PWM. This matrix driver software could have been written for the IOIO to control the matrix directly without an Arduino, but Adafruit had really tuned this library for high-performance and very precise timing, so I thought I’d better stay with the Arduino code for now. The result is video at about 4 frames per second. Not very fast, but the color rendition is pretty good.
Hardware Setup
The RGB matrix panel is wired to the Arduino just as Adafruit’s instructions describe. They have an excellent page that describes how the panel works and how to use it.
The RGB matrix and the Arduino are powered by a 5V regulated power supply that can provide 2A (also from Adafruit). The IOIO board is powered independently by a 9V supply that can provide 1A. It’s important to provide plenty of current to the IOIO board so that the phone can charge, however you can adjust a potentiometer on the IOIO to reduce the charging current. As with any project with multiple power supplies, all the grounds must be connected. A single green wire provides the serial data feed from the IOIO to the Arduino RX pin.
I used a diffuser in front of the display to make it look much better. Without a diffuser, the LEDs are simply blinding and it’s not easy to see any image. My diffuser is a piece of acrylic with paper vellum on it. The diffuser is held about 5mm in front of the LED panel (with a little roll of duct tape as a spacer).
The phone (a Samsung Nexus S) is connected to the IOIO via USB. I mounted above the panel by holding it very gently with a Panavise.
Software Setup
Android + IOIO + OpenCV Software
The hardest part of the software setup is preparing your development environment for Android, IOIO, and OpenCV development. The details of how to do this are beyond the scope of this article, but all of the steps are documented in various places.
Set up your Android development environment: this is documented on the Android SDK website. After you have performed this step, you will be able to write simple Android programs and run them on your phone.
Install the IOIO library: see this great Sparkfun tutorial which describes how to run Android apps that communicate with a connected IOIO board. After you have performed this step, you will be able to upload the HelloIOIO app to your phone and have it communicate with your IOIO board. Don’t move on to the next step until you are sure you have the IOIO working with your phone.
Install the OpenCV library for Android by following these instructions. After successfully doing this, you should be able to run the OpenCV Android examples on your Android phone. Don’t proceed until you have this working successfully.
Now that you have all the supporting libraries in place, download the RGBMatrixDriver Android application project and install it in your Eclipse workspace and open it. With luck, it will compile cleanly. If not, make sure that the project is correctly pointing to the IOIO and OpenCV libraries as in the image below.
You may need to customize the code a bit. I used IOIO pin 7 to send serial data to the Arduino, so you may need to change the pins specified in the call to openUart in RGBMatrixActivity.java. You may also need to change some screen dimensions specified in RGBMatrixView.java to work with your phone — just follow the comments.
Once you have the application running on your phone, this is what it looks like in action. The video image is displayed with the same resolution and colors as the RGB matrix.
Arduino Software
Now that the hard part is done, it’s easy to get the Arduino software installed.
First, you’ll need Adafruit’s library for driving the panel. It’s available here. Install it in your Arduino sketchbook libraries folder just like any other library.
Then download and install the RGBMatrixSerial sketch and install it in your Arduino sketchbook. Compile it and upload it onto your Arduino. The sketch is so simple, I’ll show the whole thing here:
#include "RGBmatrixPanel.h"
#define A A0
#define B A1
#define C A2
#define CLK 8
#define LAT A3
#define OE 9
#define WIDTH 32
int count = 0;
byte currentByte = 0;
byte lastByte = 0;
uint16_t color;
RGBmatrixPanel matrix(A, B, C, CLK, LAT, OE, true);
void setup()
{
Serial.begin(115200);
matrix.begin();
}
void loop() {
int index;
while (Serial.available()) {
lastByte = currentByte;
currentByte = Serial.read();
// Look for the frame marker 0xF000
if ((lastByte == 0xF0) && (currentByte == 0x00)) {
count = 0;
matrix.swapBuffers(false);
} else {
if ((count % 2) == 1) {
color = (lastByte << 8) | currentByte;
index = (count-1)/2;
matrix.drawPixel(index % WIDTH, index / WIDTH, color);
}
count++;
}
}
}
Future Ideas
increase the framerate a bit by packing 2 pixels in 3 bytes of transmitted data (only really need 1.5 bytes per pixel), but need different frame marker detection.
One of the coolest things you can do with the nootropic design Video Experimenter shield for Arduino is decode the closed caption data embedded in NTSC (North American) television broadcasts. I figured out how to do this and documented it in a another project, so if you want to understand all the details of how to capture and decode closed captions, refer to that project. With this project, I take it a step further and show how the spoken dialog embedded in a television show can be visualized on a computer in a “cloud” of words. This is the same type of cloud (often called a “tag cloud”) that you see on blogs, where the frequency of a particular word is reflected in the size of the word. More frequent == larger word.
Hardware Setup
First, here’s how the hardware is set up. It’s really simple. The Video Experimenter needs a composite video feed from a TV tuner like a DVR (e.g. Tivo) or VCR. You can also use a DVD player because DVDs usually have closed captioning data. The USB cable connects to your computer where you run a Processing sketch (program) to visualize the words as they are decoded by the Arduino. The Processing sketch dynamically builds the TV cloud as the words are extracted from the closed caption stream!
Hardware setup
Demo Video
Here’s a video where I show a TV cloud of spoken dialog being created dynamically. I superimposed a video of the television broadcast so you can correlate the broadcast with the Processing application, but note that the Processing application doesn’t acutally display the video. Words spoken with higher frequency are larger.
Example TV Clouds
I always have noticed that whenever I happen to see a US national news broadcast, all the commercials are for drugs. I guess only old people watch the news on TV anymore. Here’s a TV cloud of the commercials shown during NBC Nightly News. Can you guess which drugs are being advertised? Can you guess which maladies they claim to cure? Look at all those nasty side effects!
TV cloud made from drug commercials. Click to enlarge.
Here is a TV cloud made while watching a baseball game. For US readers familiar with baseball, can you guess which teams were playing? Answer is at the end of this post.
TV cloud built from part of a baseball game broadcast. Click to enlarge.
The Processing sketch reads words from the serial line and filters out any word less than 3 letters and some very common words like “the”, “and”, “for”, etc. This application relies on the very nice OpenCloud Java library, so you’ll need to download that and use it in your Processing environment. Create this structure in your Processing sketchbook libraries directory: opencloud/library/opencloud.jar
The nootropic design Video Experimenter shield uses some pretty advanced features of the Arduino’s ATmega328 microcontroller. One downside of this is that you can’t use the Video Experimenter shield on an Arduino Mega. Why? Well, the designers of the Arduino Mega didn’t connect a lot of the ATmega1280/ATmega2560 pins to headers on the board so that you could use them! And, as it turns out, the pins with key features utilized by the Video Experimenter are not connected to anything!
To perform video overlay, the Video Experimenter relies on an input capture pin (to capture the exact time that the pin has changed state). Even though the ATmega1280/ATmega2560 has 4 input capture pins, none of them are connected!
Important pins not even connected!
And to capture video images in the Arduino’s memory, Video Experimenter uses the analog comparator in the chip. But the AIN0 pin for the analog comparator is not connected! What were the Arduino Mega designers thinking?
Fortunately, there is the Seeeduino Mega. The guys at Seeed Studio broke out nearly all the pins on the ATmega1280 so that you can use them. I love the Seeeduino Mega because it provides so many pins on a rather small board.
Simply make 5 connections with jumper wires and you can use the Video Experimenter on the Seeeduino Mega. No code changes necessary!
By connecting 5 jumper wires, you can use the Seeeduino Mega
Here are the connections to make:
11 to 9 (white wire in picture above) 7 to 29 on the Seeeduino Mega (yellow wire) INPUT pin on the Video Experimenter to PE2 on the Seeeduino Mega (green wire) SYNCOUT pin on the Video Experimenter to PD4 on the Seeeduino Mega (gray wire) VSYNC pin on the Video Experimenter to 21 on the Seeeduino Mega (brown wire)
Now you can use the Video Experimenter with an Arduino that has a more powerful processor. It really helps to have 8K of SRAM instead of 2K. Now you can do text and graphics overlay with higher resolutions, like 192×128. Have fun!
Several people in the FPV RC world (that’s First Person View Remote Control for the uninitiated) have contacted me about the possibility of using a Video Experimenter shield as an OSD solution (on screen display). People who fly RC planes and helicopters are increasingly using on-board video cameras to provide a first-person view of the flight, and they like to overlay GPS data onto the video image. Flying radio controlled drones has become a huge hobby (just look how big the DIYDrones and RCGroups communities are!).
I don’t have any RC vehicles, so I’m really on the outside looking in, but I thought I’d experiment to see how useful a Video Experimenter might be in overlaying data onto a video feed. I wrote a simple Arduino program that reads GPS data from the Arduino’s serial RX pin and overlays a heads-up display onto the video input. I think it turned out pretty nice:
Using a Video Experimenter as an OSD
All relevant info is displayed: current latitude/longitude, compass heading, speed (left), altitude (right), and distance home (meters) in the lower right. There’s also a home arrow in the center of the screen pointing toward the hard-coded home location. Note that the ground elevation is also hard coded in the Arduino program so we can calculate altitude.
Circuit Setup
I think there are two ways that a FPV solution could use this approach. If the vehicle is transmitting video and also transmitting GPS telemetry via an XBee radio, then the GPS data can be overlayed onto the video at the ground station. I had to configure the XBee radios to use the same baud rate as my GPS (4800bps).
Circuit for GPS overlay ground station
OR the Arduino and Video Experimenter shield could be onboard the vehicle itself. This way the GPS data is overlayed onto the video before the video is transmitted. In this case, the GPS module is connected directly to the Arduino RX pin, and there’s no XBee radios involved. An Arduino+Video Experimenter shield weigh 2 ounces or 57 grams (I knew you were going to ask that).
I need to be clear that this overlay software can’t run on your ArduPilot hardware. Processing video in an Arduino environment requires nearly all of the ATmega’s SRAM, so there’s no way that this is going to run on the same hardware as your autopilot code.
Again, I’m not in the FPV RC community and don’t have a plane, so I’m not entirely sure how feasible all this is. But a number of RC hobbyists have asked about the Video Experimenter and were enthusiastic about the possibilities. Comments are very welcome!
GPS Configuration
My EM-406 GPS module is configured to only output RMC and GGA sentences, and they are output once per second, so the display only updates at that frequency. It was important to limit the output to only the RMC and GGA sentences, because when using the Arduino video library (TVout), serial communication is accomplished by polling, and it can’t handle data too quickly. To limit the output to RMC and GGA sentences, I wrote these commands to the GPS RX line, with each line followed by a CR and LF character. The Arduino serial terminal works fine for this. Don’t write the comments, of course.
Closed captioning is the technology used to embed text or other information in an NTSC television broadcast (North America, Japan, some of South America). It is typically a transcription of the broadcast audio for the benefit of hearing impaired viewers. No doubt, you’ve all seen closed captions displayed on a TV, but how does it work? This project will explain how closed captioning technology works and then show you how you can decode and display the data using your Arduino and a Video Experimenter shield. There a lot to learn here, so be patient! First, you can take a look at a video showing this capability, then keep reading to learn how it works.
The data that your TV displays is embedded in the broadcast itself in a special format, and in a special location of the video image. When you activate the closed captioning feature on your TV, your TV decodes the information and displays it on the screen. Whether you are displaying it or not, the data is in the broadcast encoded on line 21 of the video frame. This is defined by the standard EIA-608. Here is what the line 21 signal looks like:
Waveform for the closed caption data on line 21 of a TV frame
This shows the voltage of a composite video signal for line 21. The horizontal sync and color burst are just like any other video line, but the section called “clock run-in” is a special sinusoidal wave that allows the TV to synchronize with the closed captioning data which is about to start. The 7-peak run-in is followed by 3 start bits with values of 001. You can see how the voltage rises for the third bit S3. The next 16 bits represent two 8-bit characters of text. That’s right, there are only two characters per video frame, but at 30 frames per second, there is enough bandwidth for closed captions. The last bit of each byte b7 is an odd parity bit. Parity bits are an error detection mechanism. That is, this bit is either on or off in order to keep the total bits in the byte at an odd number. So, if bits b0-b6 have 4 bits on, then the parity bit is on to achieve an odd number of bits (5).
Capturing and Decoding the Data
So, how do we capture and decode this data using the Video Experimenter? We need to use the enhanced TVout library used with all Video Experimenter projects. You may already know from other Video Experimenter projects that we can capture a video image in the TVout frame buffer. For this project, we just want to capture the line 21 data so we can decode it. This is accomplished with the API method: tv.setDataCapture(int line, int dataCaptureStart, char *ccdata) where ‘line’ is the TVout scan line to capture, ‘dataCaptureStart’ is the number of clock cycles on that line to wait before starting to capture, and ‘ccdata’ is a buffer to store the bits in. Typically, we do something like this:
Even though the data is on line 21, I have found it to be on line 13 or 14 as far as TVout is concerned. The value of 310 for dataCaptureStart is the value I have found to work best in order to fit both characters of data in the width of the TVout frame buffer. This will make more sense later when we visually look at the pixels captured. It may take a while to “find” the data by trying different lines and different values for dataCaptureStart to get the right alignment. Just try different values. I have also needed to adjust the small potentiometer near the reset button upward a bit. A resistance of around 710K was required instead of the standard 680K required by the LM1881 chip on the Video Experimenter. You’ll know when you’ve found the data when you see a data line like in the images below. Sometimes you might find data that is not closed captions, but information about the program, like the title, etc. This is called XDS or Extended Data Services. This can be interesting information to decode also!
Once we tell enhanced TVout where to find the data, the buffer ccdata will always contain the pixels of the specified line of the current frame. If we display the captured pixels on the screen we can visually see how it matches up with the line 21 waveform. To produce the picture below, I copied the contents of ccdata to the first line of the TVout frame buffer so we can see the data with our eyes. The data appears as white pixels at the top of the image. It isn’t necessary to display it on the screen in order to decode it and write it to the Serial port. But it makes it easier to find the data visually and see what’s going on.
Closed captioning data line displayed at top of image
On the left side we can see the last 2 peaks of the clock run in sine wave. Then we clearly see the start bits 001. Each bit is about 5 or 6 pixels wide. Then there are 7 zero bits (pixels off) and the parity bit (on). When this picture was taken, no dialog was being spoken, so the characters are all zero bits except for the parity bit. When text data is being broadcast, the bits flash very quickly:
When data is present, the bits flash quickly.
Now that we have found the data in the broadcast, and can display it for inspection, we need to decode this 128-bit wide array of pixels into the two text characters. To do that, we need to note where each bit of the characters starts. Each bit is 5 or 6 pixels wide. The next step I took in my program was to define an array of bit positions that describe the starting pixel of each bit:
These are the bit positions for the two bytes in the data line. By displaying these bit positions just below the data line, we can adjust them if needed by trial and error. Here’s an image with the bit positions displayed below the data line. Since each data bit is nice and wide, they don’t have to line up perfectly to get reliable decoding. These positions have worked well for me for a variety of video sources.
Values of the bit position array displayed to show alignment with data.
OK, we are almost done. Now that we have found the closed caption data line, and have established the starting points for each bit, we can easily decode the bits into characters and write them to the serial port for display on a computer. We can also just print them to the screen if we want. I have taken care of all this code for you, and you can download the complete project code here.
If you have problems finding the data, try different lines for the data (13 or 14), different values for dataCaptureStart, and adjust both potentiometers on the Video Experimenter. Try slowly turning the small pot near the reset button clockwise. If you are patient, you’ll find the data and decode it!
Other project ideas
Instead of writing the data to the serial port, write it to the screen itself with tv.print(s)
Search for keywords in a closed captions and light an LED when the word is found.
