Tom Igoe Interview

From Processing

This email interview, with questions from Casey Reas, took place from 8 – 27 June 2008.

Tom Igoe is a professor at New York University's Interactive Telecommunications Program (ITP), where he is the area head for physical computing classes. Tom is the author of the excellent Making Things Talk: Practical Methods for Connecting Physical Objects and the co-author (with Dan O'Sullivan) of the definitive Physical Computing: Sensing and Controlling the Physical World with Computers. He's a frequent contributor to the O'Reilly Emerging Technology Conference and he has taught workshops about physical computing around the globe. Tom has also generously contributed his expertise over the years to improving the Processing Serial Library.

When and why did you start using Processing?

I started using it in about 2002, I think I heard about it from you guys, and we were looking for something to replace Director as a tool to teach programming at ITP. A few of the students were already playing around with it (Josh Nimoy and others), and it seemed like just the right thing for us &em; not a lot of interface, but plenty of flexibility, and a very comfortable syntax. Also, it had a serial library right from the start, which is essential for our teaching at ITP. From about the fourth class, we have students using Processing to communicate with microcontrollers. We treat it as a staple skill, like basic animation or A/V control.

The thing that really got me using Processing extensively was when Dan O'Sullivan and I used it to write some examples for our first book, Physical Computing. As I worked on the examples for that book, I realized how simple it was to use, and how fast you could get things done.

How do you use it now?

It's my main desktop prototyping tool. I use it whenever I need to see data incoming from a serially-connected device or a net-connected device, because the serial and net libraries make that dead simple. I have a few live graphing examples that I use all the time just to see the output of sensors attached to microcontrollers. When I want to work out the details of a given network protocol, I'll use Processing in conjunction with PHP to figure out what's going on.

I'm probably atypical as a Processing user, because most of my examples have very little graphics and no sound. For me, it's the easiest tool to turn data into text so I can see how the devices I build are converting physical changes into digital data. Sometimes I get a little more advanced. A recent example I built took motion sensing data off roller derby players from ZigBee radios though the serial port to make sound effects when the players hit each other. There was also a live graph of the sensor data in that sketch, and an interface for saving the data in a file, which I adapted from Ben's Visualizing Data book. And sometimes I use the video library to make video tracking sketches just for my own amusement. But I'd say 75% of my Processing use is to build communications proxies to connect devices to each other.

Interactive Dolls, Noriaki Okada and Soyoung Park, NYU ITP

Cousteau, Terence Arjo, NYU ITP

Please elaborate on the idea of physical computing. What does it mean and what is the potential?

Physical Computing (phys comp) is an approach to learning how humans communicate through computers that starts by considering how humans express themselves physically. A lot of beginning computer interface design instruction takes the computer hardware for given — namely, that there is a keyboard, a screen, perhaps speakers, and a mouse — and concentrates on teaching the software necessary to design within those boundaries. In physical computing, we take the human body as a given, and attempt to design within the limits of its expression.

This means that we have to learn how a computer converts the changes in energy given off by our bodies, in the form of heat, light, sound, and so forth, into changing electronic signals that it can read and interpret. We learn about the sensors that do this, and about very simple computers, called microcontrollers, that read sensors and convert their output into data. Finally, we learn how microcontrollers communicate with other computers.

Physical computing takes a hands-on approach, which means that you spend a lot of time building circuits, soldering, writing programs, building structures to hold sensors and controls, and figuring out how best to make all of these things relate to a person's expression.

As far as its potential, look around you. Computing doesn't just happen on the desktop. It happens on the mobile phone, the museum kiosk, the subway turnstile, the gas pump, the credit card reader, the MIDI keyboard, the stage lighting controller, the car dashboard — most of the technology we interact with on a daily basis includes a computer of some sort, and quite often, is connected to a network. Given that, there is great need for everyone to understand how those physical interfaces work, and how they come into being.

Physical computing is an attempt to teach that without a lot of technical overhead. Processing makes it much simpler. Its physical derivatives, Wiring, Arduino, and Mobile Processing, help with that. It makes it possible for people to dabble without having to be an expert. I don't want everyone who takes a phys comp class to become a hardware hacker, that'd get boring really fast. In fact, some of my favorite past students took my intro class and never made another physical project. Even if you never build another thing after taking a physical computing class, my hope is that you leave the class with a greater understanding of the possibilities and limitations of physical interfaces, an appreciation for the work that goes into making them, and a new ability to negotiate their use.

What are a few sensors that you recommend for beginners to explore?

I'm fond of accelerometers. They give you information about the tilt of a person or object to which they're attached, or the acceleration of the person or object if it's moving. Parallax and Spark Fun make some good accelerometer modules. Distance rangers like the Sharp GP-2Dxx series are also good, they sense a person's distance from the sensor in a range from about 10cm to a meter. Ultrasonic distance rangers are also good, out to about 6m. Maxbotix has some easy ones to use. Photocells and thermistors are long-time physical computing staples, for sensing light and heat. Force-sensing resistors are also quite easy to use to sense how hard someone's pressing against a surface, and they're more popular than ever now that a number of electronics vendors like Spark Fun and Trossen Robotics are carrying force sensors from Interlink and Spectrasymbol. There are so many other sensors out there to use, though. A few years ago I started a class at ITP called Sensor Workshop just to explore different sensors. We've got a wiki, built by the students, at http://itp.nyu.edu/physcomp/sensors, with collected reports from several years of the class.

Atlas Gloves, Dan Phiffer and Mushon Zer-Aviv, NYU ITP

Flight Simulator, Ed Guttman and Ayad Alkadhi, NYU ITP

How do you get the data from a sensor into a Processing program?

The wii has made it dead simple to detect motion, and the OSC-P5 library and a proxy application like DarwiinRemote on the mac make it easy to get accelerometer data into Processing. Tools like Digi's XBee radios make it easy to get just about any sensor data into Processing, thanks to Dan Shiffman and Rob Faludi's Xbee library. I just did an article showing how to do that for Make magazine. What's great about the Wii and the XBees is that they make it possible to read sensor data without needing to learn microcontrollers at all, and for the XBees, you only need a little electronics knowledge.

Most of the time, though, I attach sensors to an Arduino microcontroller module and program it to send serial data to Processing. Arduino and Wiring, both of which have their roots in Processing, make more complex sensor interactions relatively simple, especially when combined with Processing.

Please tell us about your latest book Making Things Talk?

It grew out of a class I teach called Networked Objects. The idea originated from my interest to connect some of the devices I was building to the Internet. The book is a series of examples of how to do that, along with general background knowledge on networks. It covers basic internet connections using TCP and UDP, wireless communications using ZigBee and Bluetooth radios, a couple of email examples, how to make a microcontroller fetch web pages, some RFID, and a little bit more. I assume the reader has a little familiarity with electronics and programming, so it's not for the absolute beginner (for that I recommend Physical Computing by Dan O'Sullivan and myself), but I've had comments from a number of interested amateurs who've picked the book up and found it pretty simple to get going. All the programs in the book are written in Processing, Arduino/Wiring, or PHP. The similarity in syntax between these three environments made it really easy to write the book. I tend to use all three together frequently.

Thank you for answering the questions and for the great pointers to more information. Before we stop, please share a code example with us to demonstrate how to communicate between Arduino and Processing.

One of the most common things you need to know in physical computing is what the behavior of your sensors looks like over time. To do that, I use a simple graphing application. Following are two code samples, one for Arduino/Wiring, to send the values of one or more sensors out serially, and the other for Processing, to receive them and graph them.

This can work with any serial device that's sending in a string of ASCII numerical values separated by commas and terminated by a linefeed. You could do it on a PIC, a BASIC Stamp, an Arduino, a Wiring board, or any device that can send serial data. The code below, however, is for Arduino or Wiring.

Arduino/Wiring Code

/*
  Analog serial reader
  by Tom Igoe
  Language: Arduino/Wiring
 
  Reads several analog inputs and sends their values out.
  This application assumes you have analog sensors attached
  to analog inouts 0 through 5. At the simplest, you can 
  hook up five potentiometers and it'll work fine.
 
  Created 24 May 2006
*/
 
// define the total number of analog sensors
// that you want to read
#define numberOfSensors 6
 
void setup() {
  // initialize the serial port:
  Serial.begin(9600);
}
 
void loop() {
  // loop over the sensors:
  for (int thisSensor = 0; thisSensor < numberOfSensors; thisSensor++) {
    // read each sensor
    int sensorReading = analogRead(thisSensor);
    // print its value out as an ASCII numeric string
    Serial.print(sensorReading, DEC);
    // if this isn't the last sensor to read,
    // then print a comma after it
    if (thisSensor < numberOfSensors -1) {
      Serial.print(",");
    }
  }
  // after all the sensors have been read, 
  // print a newline and carriage return
  Serial.println();
}

Processing Code

/*
  Serial Graphing Sketch
  by Tom Igoe
  Language: Processing
 
  This sketch takes ASCII values from the serial port 
  at 9600 bps and graphs them.
  The values should be comma-delimited, with a newline 
  at the end of every set of values.
  The expected range of the values is between 0 and 1023.
 
  Created 20 April 2005
  Updated 27 June 2008
*/
 
import processing.serial.*;
 
int maxNumberOfSensors = 6;       // Arduino has 6 analog inputs, so I chose 6
boolean fontInitialized = false;  // whether the font's been initialized
Serial myPort;                    // The serial port
 
float[] previousValue = new float[maxNumberOfSensors];  // array of previous values
int xpos = 0;                     // x position of the graph
PFont myFont;                     // font for writing text to the window
 
void setup () {
  // set up the window to whatever size you want:
  size(800, 600);        
  // List all the available serial ports:
  println(Serial.list());
  // I know that the first port in the serial list on my mac
  // is always my  Arduino or Wiring module, so I open Serial.list()[0].
  // Open whatever port is the one you're using.
  String portName = Serial.list()[0];
  myPort = new Serial(this, portName, 9600);
  myPort.clear();
  // don't generate a serialEvent() until you get a newline (\n) byte:
  myPort.bufferUntil('\n');
  // create a font with the fourth font available to the system:
  myFont = createFont(PFont.list()[3], 14);
  textFont(myFont);
  fontInitialized = true;
  // set inital background:
  background(0);
  // turn on antialiasing:
  smooth();
}
 
void draw () {
  // nothing happens in the draw loop, 
  // but it's needed to keep the program running
}
 
void serialEvent (Serial myPort) {
  // get the ASCII string:
  String inString = myPort.readStringUntil('\n');
 
  // if it's not empty:
  if (inString != null) {
    // trim off any whitespace:
    inString = trim(inString);
 
    // convert to an array of ints:
    int incomingValues[] = int(split(inString, ","));
 
    // print out the values
    //  print("length: " + incomingValues.length + " values.\t");
    if (incomingValues.length <= maxNumberOfSensors && incomingValues.length > 0) {
      for (int i = 0; i < incomingValues.length; i++) {
 
        // map the incoming values (0 to  1023) to an appropriate
        // graphing range (0 to window height/number of values):
        float ypos = map(incomingValues[i], 0, 1023, 0, height/incomingValues.length);
 
        // figure out the y position for this particular graph:
        float graphBottom = i * height/incomingValues.length;
        ypos = ypos + graphBottom;
 
        // make a black block to erase the previous text:
        noStroke();
        fill(0);
        rect(10, graphBottom+1, 110, 20);
 
        // print the sensor numbers to the screen:
        fill(255);
        int textPos = int(graphBottom) + 14;
        // sometimes serialEvent() can happen before setup() is done.
        // so you need to make sure the font is initialized before
        // you text():
        if (fontInitialized) {
          text("Sensor " + i + ":" + incomingValues[i], 10, textPos);
        }
        // draw a line at the bottom of each graph:
        stroke(127);
        line(0, graphBottom, width, graphBottom);
        // change colors to draw the graph line:
        stroke(64*i, 32*i, 255);
        line(xpos, previousValue[i], xpos+1, ypos);
        // save the current value to be the next time's previous value:
        previousValue[i] = ypos;
      }
    }
    // if you've drawn to the edge of the window, start at the beginning again:
    if (xpos >= width) {
      xpos = 0;
      background(0);
    } 
    else {
      xpos++;
    }
  }
}