Select Page

Difficulty Level = 8 [What’s this?]

How Does Closed Captioning Work?

Closed captioning is the technology used to embed text or other information in an NTSC television broadcast (North America, Japan, some of South America). It is typically a transcription of the broadcast audio for the benefit of hearing impaired viewers. No doubt, you’ve all seen closed captions displayed on a TV, but how does it work? This project will explain how closed captioning technology works and then show you how you can decode and display the data using your Arduino and a Video Experimenter shield. There a lot to learn here, so be patient! First, you can take a look at a video showing this capability, then keep reading to learn how it works.

The data that your TV displays is embedded in the broadcast itself in a special format, and in a special location of the video image. When you activate the closed captioning feature on your TV, your TV decodes the information and displays it on the screen. Whether you are displaying it or not, the data is in the broadcast encoded on line 21 of the video frame. This is defined by the standard EIA-608. Here is what the line 21 signal looks like:

Waveform for the closed caption data on line 21 of a TV frame

This shows the voltage of a composite video signal for line 21. The horizontal sync and color burst are just like any other video line, but the section called “clock run-in” is a special sinusoidal wave that allows the TV to synchronize with the closed captioning data which is about to start. The 7-peak run-in is followed by 3 start bits with values of 001. You can see how the voltage rises for the third bit S3. The next 16 bits represent two 8-bit characters of text. That’s right, there are only two characters per video frame, but at 30 frames per second, there is enough bandwidth for closed captions. The last bit of each byte b7 is an odd parity bit. Parity bits are an error detection mechanism. That is, this bit is either on or off in order to keep the total bits in the byte at an odd number. So, if bits b0-b6 have 4 bits on, then the parity bit is on to achieve an odd number of bits (5).

Capturing and Decoding the Data

So, how do we capture and decode this data using the Video Experimenter? We need to use the TVout library for Video Experimenter used with all Video Experimenter projects. The code for this project is in the example “ClosedCaptions” in the TVout/examples folder. You may already know from other Video Experimenter projects that we can capture a video image in the TVout frame buffer. For this project, we just want to capture the line 21 data so we can decode it. This is accomplished with the API method: tv.setDataCapture(int line, int dataCaptureStart, char *ccdata) where ‘line’ is the TVout scan line to capture, ‘dataCaptureStart’ is the number of clock cycles on that line to wait before starting to capture, and ‘ccdata’ is a buffer to store the bits in. Typically, we do something like this:

  unsigned char ccdata[16]; // 128 pixels wide is 16 bytes
  tv.setDataCapture(13, 310, ccdata);

Even though the data is on line 21, I have found it to be on line 13 or 14 as far as TVout is concerned. The value of 310 for dataCaptureStart is the value I have found to work best in order to fit both characters of data in the width of the TVout frame buffer. This will make more sense later when we visually look at the pixels captured. It may take a while to “find” the data by trying different lines and different values for dataCaptureStart to get the right alignment. Just try different values. I have also needed to adjust the small potentiometer near the reset button upward a bit. A resistance of around 710K was required instead of the standard 680K required by the LM1881 chip on the Video Experimenter. You’ll know when you’ve found the data when you see a data line like in the images below. Sometimes you might find data that is not closed captions, but information about the program, like the title, etc. This is called XDS or Extended Data Services. This can be interesting information to decode also!

Once we tell enhanced TVout where to find the data, the buffer ccdata will always contain the pixels of the specified line of the current frame. If we display the captured pixels on the screen we can visually see how it matches up with the line 21 waveform. To produce the picture below, I copied the contents of ccdata to the first line of the TVout frame buffer so we can see the data with our eyes. The data appears as white pixels at the top of the image. It isn’t necessary to display it on the screen in order to decode it and write it to the Serial port. But it makes it easier to find the data visually and see what’s going on.

Closed captioning data line displayed at top of image

On the left side we can see the last 2 peaks of the clock run in sine wave. Then we clearly see the start bits 001. Each bit is about 5 or 6 pixels wide. Then there are 7 zero bits (pixels off) and the parity bit (on). When this picture was taken, no dialog was being spoken, so the characters are all zero bits except for the parity bit. When text data is being broadcast, the bits flash very quickly:

When data is present, the bits flash quickly.

Now that we have found the data in the broadcast, and can display it for inspection, we need to decode this 128-bit wide array of pixels into the two text characters. To do that, we need to note where each bit of the characters starts. Each bit is 5 or 6 pixels wide. The next step I took in my program was to define an array of bit positions that describe the starting pixel of each bit:

byte bpos[][8]={{26, 32, 38, 45, 51, 58, 64, 70}, {78, 83, 89, 96, 102, 109, 115, 121}};

These are the bit positions for the two bytes in the data line. By displaying these bit positions just below the data line, we can adjust them if needed by trial and error. Here’s an image with the bit positions displayed below the data line. Since each data bit is nice and wide, they don’t have to line up perfectly to get reliable decoding. These positions have worked well for me for a variety of video sources.

Values of the bit position array displayed to show alignment with data.

OK, we are almost done. Now that we have found the closed caption data line, and have established the starting points for each bit, we can easily decode the bits into characters and write them to the serial port for display on a computer. We can also just print them to the screen if we want. I have taken care of all this code for you, and it is all in the example called “ClosedCaptions” in the TVout library for Video Experimenter.

If you have problems finding the data, try different lines for the data (13 or 14), different values for dataCaptureStart, and adjust both potentiometers on the Video Experimenter. Try slowly turning the small pot near the reset button clockwise. If you are patient, you’ll find the data and decode it!

Other project ideas

  • Instead of writing the data to the serial port, write it to the screen itself with tv.print(s)
  • Search for keywords in a closed captions and light an LED when the word is found.