HomeProjectsPeoplePublicationsLinks
SATIN
Download
Documentation
Support
Sample Apps
Research
Publications

More Projects

SATIN Tutorial

Table of Contents

Introducing the SATIN Informal Ink Toolkit

SATIN is a toolkit for making informal ink-based applications. Informal ink-based applications use pens and sketching as the primary interaction technique. Sketching is quick and easy to do with pen and paper, but it's difficult to create computer-based apps that use sketching today. It's often the case that pens are just treated as another pointing device, clicking on buttons and menus.

That's where SATIN comes in. We're trying to make it easier to build informal ink-based apps, and SATIN is our first take at a toolkit to help you do that.


Above are two examples of informal ink-based apps. Click on the images above for enlarged views.


Other SATIN Documentation

In this tutorial, we'll go over lots of details on how SATIN works. If you want to understand the big picture, that is what the problems are and why we built SATIN, then check these out:

  • UIST2000 paper on SATIN
    This is the conference paper on SATIN, describing the toolkit in 10 pages.
  • Presentation on SATIN (ppt)
    This is a presentation of the paper above. The HTML version doesn't do animations, so I'd suggest taking a look at the PowerPoint slides if you can.

If you want to try out a large-scale app built with SATIN, check out DENIM, a sketch-based web site design tool we're working on. It shows a lot of the features in SATIN, like sketching, recognition, zooming, and so on.

Also, all of the Javadoc APIs for SATIN are available online. You can skim through the docs or read them while going thru this tutorial.

Lastly, there is a satin-dev mailing list. Let us know your comments, what problems you run into, any cool apps you make, or new code you want to share with us!

  • Each email to satin-dev should have an unsubscribe email address that you can use.

Setting Up SATIN

If you haven't already, download the latest SATIN zip file. After you've done this, unzip it some location. You'll also want to get the zip file of the tutorial source code. Alternatively, you can just download each individual source file as we get to it.

If you don't have a zip program, you can use jar to unzip. The jar app comes with Java Development Kit, so you should already have it. To unzip, type: 

      jar -xvf [name of zip file]

If you want to try out some SATIN apps, try running SketchySPICE or Brainstorm (there are some batch files for doing so). With SketchySPICE, try drawing AND and OR gates. With Brainstorm, try drawing rectangular shapes about the size of post-it notes, and draw lines between the notes.

Classpath

After you've downloaded SATIN, you'll need to set your CLASSPATH variable to point to the satin jar file. With WinNT and Win2K, go to "Control Panel -> System". There should be some tab that will take you to environment variables. Be sure to point to the absolute directory location of the satin jar file.

With Unix based systems...well, it's either set, setenv, or something like that. It really depends on what shell you're using.

Once you've set the CLASSPATH, you should be able to use the SATIN classes from any directory. One thing to note, though. There's a directory with the SATIN distribution called data. This directory contains the data files for gestures. If you want to use gestures, you'll have to have this directory in the same directory where you run the Java Virtual Machine. We'll remind you of this once we get to gestures.

Part One - Using SATIN Sheets

Immediate gratification can be a good thing, so let's develop our first app, a simple Scribbling program. We'll augment this app as we go along. I'll assume you have a basic knowledge of Java Swing. So here's the code for our first app:

Download code v1
import javax.swing.*;
import edu.berkeley.guir.lib.satin.*;

/**
 * An example of how to use the SATIN toolkit.
 * This app demonstrates how the Sheet works with Java Swing.
 */
public class Scribbler {

   //===========================================================================
   //===   SELF-TESTING MAIN   =================================================

   public static void main(String[] argv) {
      JFrame f = new JFrame();
      Sheet  s = new Sheet();

      //// 1. Make the Sheet the content pane for the frame.
      f.setContentPane(s);

      //// 2. Make the frame visible.
      f.setSize(600, 600);
      f.setVisible(true);
   } // of main

   //===   SELF-TESTING MAIN   =================================================
   //===========================================================================

} // of class

Ok, so besides my quirky programming style, you should notice that we just set the frame's content pane to be the Sheet. The Sheet extends a JPanel, so it's okay. Try compiling this app and running it. You should be able to just scribble things using your mouse. Here's my picture. Keep in mind that it's really hard to draw with a mouse. One of my friends likens drawing with a mouse to drawing with a potato.

Some Definitions

To make sure we're on the same page, we need to define some terms.

  • A stroke is pen (or mouse) input that comes from dragging
  • Ink is a stroke that stays and is displayed on the Sheet

So here are some "hidden" debug features in SATIN. Try them out with the Scribbler app.

  • Hit "Ctrl-Alt-D" to turn on bounding boxes
  • Hit "Ctrl-Alt-W" to turn on the debugging output window (this uses the debugging package we wrote up)
  • Hit "Ctrl-Z" to undo an action
  • Hit "Ctrl-Y" to redo an action
  • Use the arrow keys to pan

Another thing to note is that the Sheet automatically pans if you scribble outside of the window. Click somewhere near an edge, and then move outside of the window to see for yourself.

Let's go on to the next app.

Part Two - Using Interpreters

So scribbling is neat, but we can do more. One feature a lot of these informal ink-based apps have is gestures, that is a stroke that executes a command. For example, you could scribble over a word to erase it. As another example, you could circle all of the objects you wanted to select.


Gestures are strokes that activate a command. In the example above, you can scrub over some text and erase that text.


In SATIN, we process strokes through Interpreters. SATIN already has one called CircleSelectInterpreter, which selects things contained within anything that's sort of circular. We've modified the code from the last example, changes are in bold.

Download code v2

import javax.swing.*;
import edu.berkeley.guir.lib.satin.*;
import edu.berkeley.guir.lib.satin.interpreter.commands.*;

/**
 * An example of how to use the SATIN toolkit.
 * This app demonstrates how the Sheet works with Java Swing.
 */
public class Scribbler {

   //===========================================================================
   //===   SELF-TESTING MAIN   =================================================

   public static void main(String[] argv) {
      JFrame f = new JFrame();
      Sheet  s = new Sheet();

      //// 1. Make the Sheet the content pane for the frame.
      f.setContentPane(s);

      //// 2. Add a Circle Select Interpreter to the Sheet.
      s.setGestureInterpreter(new CircleSelectInterpreter());


      //// 3. Make the frame visible.
      f.setSize(600, 600);
      f.setVisible(true);
   } // of main

   //===   SELF-TESTING MAIN   =================================================
   //===========================================================================

} // of class

Now compile the program and run it. You should still be able to scribble normally. To circle select some items, just draw a circular shape around some objects. Here's an example, going from left to right.



Here are some more features built into the Sheet that you can do to selected objects. These are keys that are standard to most Windows applications.

  • "Ctrl-X" or "Shift-Del" to cut
  • "Ctrl-V" or "Shift-Ins" to paste
  • "Del" to delete

Differentiating Between Gestures and Ink

One potential problem is figuring out whether the stroke should be ink or a gesture. In other words, if you just drew a stroke, how can the computer reliably know whether you just want to draw something or you want to execute a command?

Unfortunately, there really aren't good answers to that yet. It's going to take a lot of good design and lot of advances in recognition technology before it can be done really well. In the meanwhile, one thing that's often done is to just use the right button to mean gestures (as in "right" and "left").

Here's how we can make this work. Interpreters can filter out certain things. All we have to do is to tell it to ignore left and middle buttons. The other thing we should do is to tell the Sheet to not add right-button input as ink. Here's the code below, changes marked in bold.

Download code v3
import javax.swing.*;
import edu.berkeley.guir.lib.satin.*;
import edu.berkeley.guir.lib.satin.interpreter.*;
import edu.berkeley.guir.lib.satin.interpreter.commands.*;

/**
 * An example of how to use the SATIN toolkit.
 * This app demonstrates how the Sheet works with Java Swing.
 */
public class Scribbler {

   //===========================================================================
   //===   SELF-TESTING MAIN   =================================================

   public static void main(String[] argv) {
      JFrame f = new JFrame();
      Sheet  s = new Sheet();

      //// 1. Make the Sheet the content pane for the frame.
      f.setContentPane(s);

      //// 2. Add a Circle Select Interpreter to the Sheet.
      Interpreter intrp = new CircleSelectInterpreter();


      intrp.setAcceptLeftButton(false);
      intrp.setAcceptMiddleButton(false);

      s.setGestureInterpreter(intrp);

      //// 3. Make it so that by default, the Sheet will not add
      ////    right mouse button strokes as ink.
      s.setAddRightButtonStrokes(false);

      //// 4. Make the frame visible.
      f.setSize(600, 600);
      f.setVisible(true);
   } // of main

   //===   SELF-TESTING MAIN   =================================================
   //===========================================================================

} // of class

Ok, one last topic in this section. You've probably noticed that the method to add the CircleSelectInterpreter was setGestureInterpreter(). There's also another parallel method for ink, setInkInterpreter(). The way it's currently defined in SATIN, ink interpreters get called after gesture interpreters. So if the gesture interpreter says its not a gesture, it gets handed off to the ink interpreter.

We actually don't make a strict distinction between gesture and ink interpreters in SATIN. There is no class or interface to distinguish between gesture and ink interpreters. If something is assigned as an ink interpreter, then it's an ink interpreter.

Below are some examples of gesture and ink interpreters.



Here's some more code, changes in bold. We're adding in an ink interpreter that straightens out lines for us. Make the changes, compile, and try it out.

Download code v4
import javax.swing.*;
import edu.berkeley.guir.lib.satin.*;
import edu.berkeley.guir.lib.satin.interpreter.*;
import edu.berkeley.guir.lib.satin.interpreter.commands.*;
import edu.berkeley.guir.lib.satin.interpreter.stroke.*;

/**
 * An example of how to use the SATIN toolkit.
 * This app demonstrates how the Sheet works with Java Swing.
 */
public class Scribbler {

   //===========================================================================
   //===   SELF-TESTING MAIN   =================================================

   public static void main(String[] argv) {
      JFrame f = new JFrame();
      Sheet  s = new Sheet();

      //// 1. Make the Sheet the content pane for the frame.
      f.setContentPane(s);

      //// 2. Add a Circle Select Interpreter to the Sheet.
      Interpreter intrp = new CircleSelectInterpreter();
      intrp.setAcceptLeftButton(false);
      intrp.setAcceptMiddleButton(false);

      s.setGestureInterpreter(intrp);

      //// 3. Add an interpreter that straightens ink to the Sheet.
      s.setInkInterpreter(new LinearizeStrokeInterpreter());


      //// 4. Make it so that by default, the Sheet will not add
      ////    right mouse button strokes as ink.
      s.setAddRightButtonStrokes(false);

      //// 5. Make the frame visible.
      f.setSize(600, 600);
      f.setVisible(true);
   } // of main

   //===   SELF-TESTING MAIN   =================================================
   //===========================================================================

} // of class

Here are two pictures showing how it works. The one on the left shows what I drew, the one on the right shows what it was changed to.



Part Three - Creating Interpreters

So let's create our first interpreter. We'll do something simple, an interpreter that randomly assigns colors to ink. We'll also show how to use the debugging window too. The code for the interpreter is below, commentary follows.


Download code v5

import edu.berkeley.guir.lib.debugging.*;
import edu.berkeley.guir.lib.satin.event.*;
import edu.berkeley.guir.lib.satin.objects.*;
import edu.berkeley.guir.lib.satin.interpreter.*;

public class RandomColorInterpreter
   extends InterpreterImpl {

   //===========================================================================
   //===   CONSTANTS   =========================================================

   private static final Debug debug = new Debug(Debug.ON);

   //===   CONSTANTS   =========================================================
   //===========================================================================



   //===========================================================================
   //===   INTERPRETER METHODS   ===============================================

   public void handleSingleStroke(SingleStrokeEvent evt) {

      debug.println("method called");

   } // of method

   //===   INTERPRETER METHODS   ===============================================
   //===========================================================================



   //===========================================================================
   //===   CLONE   =============================================================

   public Object clone() {
      //// It actually turns out that in Java, you're not supposed to
      //// create new objects in method clone(). See:
      //// http://java.sun.com/docs/books/tutorial/java/javaOO/objectclass.html
      ////
      //// However, SATIN has not been updated to reflect this, so...
      return (new RandomColorInterpreter());
   } // of method

   //===   CLONE   =============================================================
   //===========================================================================

} // of class

The CONSTANTS section shows how to get access to the debugging object. The INTERPRETER METHODS section exposes some of the event handling in SATIN. The CLONE section is just for handling cloning, a requirement for interpreters. Cloning is used when objects are cut and paste.

The interesting part is the method handleSingleStroke(). This method is called for you whenever a person draws a single stroke (ie, button down, drag, button up). There are ways of handling strokes as they are drawn, but that's a little more complex, so we won't touch on it here.

For now, let's just make sure that method handleSingleStroke() is being called correctly. So we'll just add some code to make a call to the Debug object, which will display it on the debugging window.

Below are the modifications to the Scribbler class.


Download code v5
import javax.swing.*;
import edu.berkeley.guir.lib.satin.*;
import edu.berkeley.guir.lib.satin.interpreter.*;
import edu.berkeley.guir.lib.satin.interpreter.commands.*;
import edu.berkeley.guir.lib.satin.interpreter.stroke.*;

/**
 * An example of how to use the SATIN toolkit.
 * This app demonstrates how the Sheet works with Java Swing.
 */
public class Scribbler {

   //===========================================================================
   //===   SELF-TESTING MAIN   =================================================

   public static void main(String[] argv) {
      JFrame f = new JFrame();
      Sheet  s = new Sheet();

      //// 1. Make the Sheet the content pane for the frame.
      f.setContentPane(s);

      //// 2. Add a Circle Select Interpreter to the Sheet.
      Interpreter intrp = new CircleSelectInterpreter();
      intrp.setAcceptLeftButton(false);
      intrp.setAcceptMiddleButton(false);

      s.setGestureInterpreter(intrp);

      //// 3. Add an interpreter that changes ink colors.
      s.setInkInterpreter(new RandomColorInterpreter());

      //// 4. Make it so that by default, the Sheet will not add
      ////    right mouse button strokes as ink.
      s.setAddRightButtonStrokes(false);

      //// 5. Make the frame visible.
      f.setSize(600, 600);
      f.setVisible(true);
   } // of main

   //===   SELF-TESTING MAIN   =================================================
   //===========================================================================

} // of class

So compile the classes and run them. Bring up the debugging window (Ctrl-Alt-W), and then scribble something. You should get something like the following:



The debugging window tells you which class and which method were called, plus what was printed out (through the debug.println() method).

Changing Colors

So let's actually do something interesting with this Interpreter. To set the color for a stroke (actually called a TimedStroke), you get its Style object. Styles group together lots of the visual presentation of objects, like color, line width, and so on.

The other thing changed is the Sheet's color. The reason is that it's hard to see some colors on a gray background, so let's change it to white.

Below is the code for doing this:


Download code v6
import edu.berkeley.guir.lib.debugging.*;
import edu.berkeley.guir.lib.satin.event.*;
import edu.berkeley.guir.lib.satin.objects.*;
import edu.berkeley.guir.lib.satin.interpreter.*;
import edu.berkeley.guir.lib.satin.stroke.*;
import java.awt.Color;
import java.util.Random;

public class RandomColorInterpreter
   extends InterpreterImpl {

   //===========================================================================
   //===   CONSTANTS   =========================================================

   private static final Debug debug = new Debug(Debug.ON);

   //===   CONSTANTS   =========================================================
   //===========================================================================



   //===========================================================================
   //===   NONLOCAL VARIABLES   ================================================

   Random rand = new Random();

   //===   NONLOCAL VARIABLES   ================================================
   //===========================================================================



   //===========================================================================
   //===   INTERPRETER METHODS   ===============================================

   public void handleSingleStroke(SingleStrokeEvent evt) {

      //// 1. Get a reference to the stroke.
      TimedStroke stk = evt.getStroke();

      //// 2. Generate a random color.
      Color       c   = new Color(rand.nextFloat(),
                                  rand.nextFloat(),
                                  rand.nextFloat());

      //// 3. Set the stroke's style to use this color.
      stk.getStyleRef().setDrawColor(c);

   } // of method

   //===   INTERPRETER METHODS   ===============================================
   //===========================================================================



   //===========================================================================
   //===   CLONE   =============================================================


   public Object clone() {
      //// It actually turns out that in Java, you're not supposed to
      //// create new objects in method clone(). See:
      //// http://java.sun.com/docs/books/tutorial/java/javaOO/objectclass.html
      ////
      //// However, SATIN has not been updated to reflect this, so...
      return (new RandomColorInterpreter());
   } // of method

   //===   CLONE   =============================================================
   //===========================================================================

} // of class

Download code v6
import javax.swing.*;
import edu.berkeley.guir.lib.satin.*;
import edu.berkeley.guir.lib.satin.interpreter.*;
import edu.berkeley.guir.lib.satin.interpreter.commands.*;
import edu.berkeley.guir.lib.satin.interpreter.stroke.*;

/**
 * An example of how to use the SATIN toolkit.
 * This app demonstrates how the Sheet works with Java Swing.
 */
public class Scribbler {

   //===========================================================================
   //===   SELF-TESTING MAIN   =================================================

   public static void main(String[] argv) {
      JFrame f = new JFrame();
      Sheet  s = new Sheet();

      //// 1. Make the Sheet the content pane for the frame.
      f.setContentPane(s);
      s.setBackground(java.awt.Color.white);

      //// 2. Add a Circle Select Interpreter to the Sheet.
      Interpreter intrp = new CircleSelectInterpreter();
      intrp.setAcceptLeftButton(false);
      intrp.setAcceptMiddleButton(false);

      s.setGestureInterpreter(intrp);

      //// 3. Add an interpreter that changes ink colors.
      s.setInkInterpreter(new RandomColorInterpreter());

      //// 4. Make it so that by default, the Sheet will not add
      ////    right mouse button strokes as ink.
      s.setAddRightButtonStrokes(false);

      //// 5. Make the frame visible.
      f.setSize(600, 600);
      f.setVisible(true);
   } // of main

   //===   SELF-TESTING MAIN   =================================================
   //===========================================================================

} // of class

Compile the code and run it. Here's a screenshot of what should appear.



Part Four - Multi-Interpreters

Multi-Interpreters are a special kind of Interpreter that lets you combine existing interpreters together. There are other reasons why we have Multi-Interpreters, but we won't discuss them here.

So let's say we want to combine three different interpreters:

  • Tap Select, which lets you tap on objects to select them
  • Circle Select, which lets you select a bunch of objects by circling them
  • Move Selected, which lets you drag selected objects around

Below is the code for combining these Interpreters together. To do this, we create a MultiInterpreter and the Interpreters to it. Then, we set the MultiInterpreter as the gesture interpreter.

When a stroke is created, the Sheet first asks its gesture interpreter to handle the stroke. In this case, the gesture interpreter is the MultiInterpreter. The MultiInterpreter just asks the Interpreters it contains to handle the stroke.

We actually use a DefaultMultiInterpreterImpl as our specific instance of a MultiInterpreter. The Default one just asks each of the Interpreters in the order they were added if it knows how to handle the stroke. If yes, then we're done. If no, then it goes on and asks the next Interpreter. In other words, it chains the Interpreters together.

This is probably a little confusing, but there is an animation of how it works in the powerpoint slides on SATIN. Skip to the part about MultiInterpreters and Stroke Dispatching, and put it into Slide Show mode to see the animations.

One thing to note is that this makes ordering of Interpreters very important. With the DefaultMultiInterpreterImpl, if you want one Interpreter to act before another, you have to add that one first.


Download code v7
import javax.swing.*;
import edu.berkeley.guir.lib.satin.*;
import edu.berkeley.guir.lib.satin.interpreter.*;
import edu.berkeley.guir.lib.satin.interpreter.commands.*;
import edu.berkeley.guir.lib.satin.interpreter.stroke.*;

/**
 * An example of how to use the SATIN toolkit.
 * This app demonstrates how the Sheet works with Java Swing.
 */
public class Scribbler {

   //===========================================================================
   //===   SELF-TESTING MAIN   =================================================

   public static void main(String[] argv) {
      JFrame f = new JFrame();
      Sheet  s = new Sheet();

      //// 1. Make the Sheet the content pane for the frame.
      f.setContentPane(s);
      s.setBackground(java.awt.Color.white);

      //// 2. Set up the gesture interpreters.
      MultiInterpreter mintrp;
      Interpreter      intrp;

      mintrp = new DefaultMultiInterpreterImpl();

      intrp = new TapSelectInterpreter();
      intrp.setAcceptLeftButton(true);
      intrp.setAcceptRightButton(true);
      mintrp.add(intrp);

      intrp = new CircleSelectInterpreter();
      intrp.setAcceptLeftButton(false);
      intrp.setAcceptRightButton(true);
      mintrp.add(intrp);

      intrp = new MoveSelectedInterpreter();
      intrp.setAcceptLeftButton(true);
      intrp.setAcceptRightButton(true);
      mintrp.add(intrp);

      s.setGestureInterpreter(mintrp);

      //// 3. Set up the ink interpreters.
      s.setInkInterpreter(new RandomColorInterpreter());

      //// 4. Make it so that by default, the Sheet will not add
      ////    right mouse button strokes as ink.
      s.setAddRightButtonStrokes(false);

      //// 5. Make the frame visible.
      f.setSize(600, 600);
      f.setVisible(true);
   } // of main

   //===   SELF-TESTING MAIN   =================================================
   //===========================================================================

} // of class

Another interesting thing we can do with MultiInterpreters is to "lie" about whether we handled the stroke event or not. An Interpreter can actually do something with a stroke event but say it didn't, and let another Interpreter in the chain handle the event.

For example, the RandomColorInterpreter does this. It changes the color of the stroke, but doesn't tell the stroke event that it's been handled. We can actually combine the RandomColorInterpreter with the LinearizeStrokeInterpreter and get new effects. (If you did want to tell the stroke event it's been handled, you would call evt.setConsumed(true) where evt is the stroke event).

So here's some code for how to combine the RandomColorInterpreter and the LinearizeStrokeInterpreter.


Download code v8
import javax.swing.*;
import edu.berkeley.guir.lib.satin.*;
import edu.berkeley.guir.lib.satin.interpreter.*;
import edu.berkeley.guir.lib.satin.interpreter.commands.*;
import edu.berkeley.guir.lib.satin.interpreter.stroke.*;

/**
 * An example of how to use the SATIN toolkit.
 * This app demonstrates how the Sheet works with Java Swing.
 */
public class Scribbler {

   //===========================================================================
   //===   SELF-TESTING MAIN   =================================================

   public static void main(String[] argv) {
      JFrame f = new JFrame();
      Sheet  s = new Sheet();

      //// 1. Make the Sheet the content pane for the frame.
      f.setContentPane(s);
      s.setBackground(java.awt.Color.white);

      //// 2. Set up the gesture interpreters.
      MultiInterpreter mintrp;
      Interpreter      intrp;

      mintrp = new DefaultMultiInterpreterImpl();

      intrp = new TapSelectInterpreter();
      intrp.setAcceptLeftButton(true);
      intrp.setAcceptRightButton(true);
      mintrp.add(intrp);

      intrp = new CircleSelectInterpreter();
      intrp.setAcceptLeftButton(false);
      intrp.setAcceptRightButton(true);
      mintrp.add(intrp);

      intrp = new MoveSelectedInterpreter();
      intrp.setAcceptLeftButton(true);
      intrp.setAcceptRightButton(true);
      mintrp.add(intrp);

      s.setGestureInterpreter(mintrp);

      //// 3. Set up the ink interpreters.
      mintrp = new DefaultMultiInterpreterImpl();

      intrp = new RandomColorInterpreter();
      mintrp.add(intrp);

      intrp = new LinearizeStrokeInterpreter();
      mintrp.add(intrp);

      s.setInkInterpreter(mintrp);

      //// 4. Make it so that by default, the Sheet will not add
      ////    right mouse button strokes as ink.
      s.setAddRightButtonStrokes(false);

      //// 5. Make the frame visible.
      f.setSize(600, 600);
      f.setVisible(true);
   } // of main

   //===   SELF-TESTING MAIN   =================================================
   //===========================================================================

} // of class

Here's a screenshot of my attempt at sketching a Mondrian painting.



Part Five - Pie Menus

Even though sketching is cool, not everything you do with pens will be sketching. Sometimes, standard GUI widgets (like buttons and menus) can be more effective. However, these GUI widgets often aren't easy to use with pens. For example, it's impossible to double-tap on anything with a pen.

Another thing we've done with SATIN is create widgets specialized for pens. One example of such are pie menus. It turns out that pie menus are easier to learn and are faster than standard linear popup menus. And they work better for pens and mice, too.



Even though pie menus ship with SATIN, we've actually made them independent of the rest of SATIN. In other words, you can use the pie menus without having to use the sketching functionality.

Here's how to add pie menus to our current app. We create the pie menu instance, and then just add labels to it.


Download code v9
import javax.swing.*;
import edu.berkeley.guir.lib.satin.*;
import edu.berkeley.guir.lib.satin.interpreter.*;
import edu.berkeley.guir.lib.satin.interpreter.commands.*;
import edu.berkeley.guir.lib.satin.interpreter.stroke.*;
import edu.berkeley.guir.lib.satin.widgets.*;

/**
 * An example of how to use the SATIN toolkit.
 * This app demonstrates how the Sheet works with Java Swing.
 */
public class Scribbler {

   //===========================================================================
   //===   PIE MENU SETUP   ====================================================

   public static void setupPieMenu(JComponent comp) {
      PieMenu pm = new PieMenu();

      //// 1. Add the pie menu slices to the pie menu.
      ////    Order starts at north, goes counter clockwise.
      ////    Reason for '\n' is because pie menu is small, so
      ////    words don't fit correctly. Either resize or go
      ////    to next line.
      pm.add("Zoom\nIn");
      pm.add("Rotate\nLeft");
      pm.add("Zoom\nOut");
      pm.add("Rotate\nRight");

      //// 2. Add the pie menu to the component.
      pm.addPieMenuTo(comp);

   } // of method

   //===   PIE MENU SETUP   ====================================================
   //===========================================================================



   //===========================================================================
   //===   SELF-TESTING MAIN   =================================================

   public static void main(String[] argv) {
      JFrame f = new JFrame();
      Sheet  s = new Sheet();

      //// 1. Make the Sheet the content pane for the frame.
      f.setContentPane(s);
      s.setBackground(java.awt.Color.white);

      //// 2. Set up the gesture interpreters.
      MultiInterpreter mintrp;
      Interpreter      intrp;

      mintrp = new DefaultMultiInterpreterImpl();

      intrp = new TapSelectInterpreter();
      intrp.setAcceptLeftButton(true);
      intrp.setAcceptRightButton(true);
      mintrp.add(intrp);

      intrp = new CircleSelectInterpreter();
      intrp.setAcceptLeftButton(false);
      intrp.setAcceptRightButton(true);
      mintrp.add(intrp);

      intrp = new MoveSelectedInterpreter();
      intrp.setAcceptLeftButton(true);
      intrp.setAcceptRightButton(true);
      mintrp.add(intrp);

      s.setGestureInterpreter(mintrp);

      //// 3. Set up the ink interpreters.
      mintrp = new DefaultMultiInterpreterImpl();

      intrp = new RandomColorInterpreter();
      mintrp.add(intrp);

      intrp = new LinearizeStrokeInterpreter();
      mintrp.add(intrp);

      s.setInkInterpreter(mintrp);

      //// 4. Make it so that by default, the Sheet will not add
      ////    right mouse button strokes as ink.
      s.setAddRightButtonStrokes(false);

      //// 5. Set up the pie menu.
      setupPieMenu(s);

      //// 6. Make the frame visible.
      f.setSize(600, 600);
      f.setVisible(true);
   } // of main

   //===   SELF-TESTING MAIN   =================================================
   //===========================================================================

} // of class

By default, the pie menu opens when you right click (right-button down then right-button up). Here's what the app looks like:



Right now, the pie menu doesn't really do anything. The reason is because we haven't set up the callbacks for it. So we're going to add in some functionality that SATIN provides but we haven't discussed yet: rotation and zooming. We won't go into the details of how rotation and zooming works. Instead, we'll just give the code for you to play around with it.

(You may also be wondering why we built zooming and rotation as part of this toolkit, especially if it's for pens and sketching. It turns out that zooming is a generally useful technique for managing information in 2D. Rotation came in for free, but think about how people rotate sheets of paper while drawing in the real world.)

There are a lot of changes in this version. There's going to have to be some hand-waving, because there are many things in here that will take a lot of explaning. Also, the Scribbler class is getting large enough that we should split it into ScribblerSheet and Scribbler (and avoid the use of static class variables). So we'll show the single class version first, and then the multiple class version next. If you want to continue hacking with SATIN, then it's best if you work with the multiple class version.


Download code v10
import javax.swing.*;
import edu.berkeley.guir.lib.satin.*;
import edu.berkeley.guir.lib.satin.interpreter.*;
import edu.berkeley.guir.lib.satin.interpreter.commands.*;
import edu.berkeley.guir.lib.satin.interpreter.stroke.*;
import edu.berkeley.guir.lib.satin.widgets.*;
import edu.berkeley.guir.lib.satin.command.*;
import edu.berkeley.guir.lib.satin.objects.*;
import edu.berkeley.guir.lib.awt.geom.*;
import java.awt.geom.*;
import java.awt.event.*;

/**
 * An example of how to use the SATIN toolkit.
 * This app demonstrates how the Sheet works with Java Swing.
 */
public class Scribbler {

   //===========================================================================
   //===   CONSTANTS   =========================================================

   /**
    * Number of frames to animate when zooming or rotating.
    */
   public static final int NUMFRAMES = 10;

   //===   CONSTANTS   =========================================================
   //===========================================================================



   //===========================================================================
   //===   PIE MENU INNER CLASS CALLBACKS   ====================================

   static class RotateLeftListener implements ActionListener {
      public void actionPerformed(ActionEvent evt) {
         AffineTransform[] txArray = new AffineTransform[NUMFRAMES];

         txArray = AffineTransformLib.animateSlowInSlowOut(
                        AffineTransform.getRotateInstance(-Math.PI/2,

                           SatinConstants.cmdsubsys.getAbsoluteLastXLocation(),
                           SatinConstants.cmdsubsys.getAbsoluteLastYLocation()),
                        NUMFRAMES);
         GraphicalObjectLib.animate(s, txArray);

      } // of method
   } // of inner class

   //-----------------------------------------------------------------

   static class RotateRightListener implements ActionListener {
      public void actionPerformed(ActionEvent evt) {
         AffineTransform[] txArray = new AffineTransform[NUMFRAMES];
         txArray = AffineTransformLib.animateSlowInSlowOut(
                        AffineTransform.getRotateInstance(Math.PI/2,
                           SatinConstants.cmdsubsys.getAbsoluteLastXLocation(),
                           SatinConstants.cmdsubsys.getAbsoluteLastYLocation()),
                        NUMFRAMES);
         GraphicalObjectLib.animate(s, txArray);

      } // of method
   } // of inner class

   //-----------------------------------------------------------------

   static class ZoomInListener implements ActionListener {
      public void actionPerformed(ActionEvent evt) {
         AffineTransform[] txArray = new AffineTransform[NUMFRAMES];
         txArray = AffineTransformLib.animateSlowInSlowOut(
                   AffineTransformLib.scaleAndCenterAt(2,
                           SatinConstants.cmdsubsys.getAbsoluteLastXLocation(),
                           SatinConstants.cmdsubsys.getAbsoluteLastYLocation(),
                           s.getBounds2D(SatinConstants.COORD_ABS)),
                        NUMFRAMES);
         GraphicalObjectLib.animate(s, txArray);
      } // of method
   } // of inner class

   //-----------------------------------------------------------------

   static class ZoomOutListener implements ActionListener {
      public void actionPerformed(ActionEvent evt) {
         AffineTransform[] txArray = new AffineTransform[NUMFRAMES];
         txArray = AffineTransformLib.animateSlowInSlowOut(
                   AffineTransformLib.scaleAndCenterAt(0.5,
                           SatinConstants.cmdsubsys.getAbsoluteLastXLocation(),
                           SatinConstants.cmdsubsys.getAbsoluteLastYLocation(),
                           s.getBounds2D(SatinConstants.COORD_ABS)),
                        NUMFRAMES);
         GraphicalObjectLib.animate(s, txArray);
      } // of method
   } // of inner class

   //===   PIE MENU INNER CLASS CALLBACKS   ====================================
   //===========================================================================



   //===========================================================================
   //===   CLASS VARIABLES   ===================================================

   static JFrame f;
   static Sheet  s = new Sheet();

   //===   CLASS VARIABLES   ===================================================
   //===========================================================================



   //===========================================================================
   //===   PIE MENU SETUP   ====================================================

   public static void setupPieMenu(JComponent comp) {
      PieMenu pm = new PieMenu();

      //// 1. Add the pie menu slices to the pie menu.
      ////    Order starts at north, goes counter clockwise.
      ////    Reason for '\n' is because pie menu is small, so
      ////    words don't fit correctly. Either resize or go
      ////    to next line.
      pm.add("Zoom\nIn").addActionListener(new ZoomInListener());
      pm.add("Rotate\nLeft").addActionListener(new RotateLeftListener());
      pm.add("Zoom\nOut").addActionListener(new ZoomOutListener());
      pm.add("Rotate\nRight").addActionListener(new RotateRightListener());

      //// 2. Add the pie menu to the component.
      pm.addPieMenuTo(comp);

   } // of method

   //===   PIE MENU SETUP   ====================================================
   //===========================================================================



   //===========================================================================
   //===   SELF-TESTING MAIN   =================================================

   public static void main(String[] argv) {
      f = new JFrame();
      s = new Sheet();

      //// 1. Make the Sheet the content pane for the frame.
      f.setContentPane(s);
      s.setBackground(java.awt.Color.white);

      //// 2. Set up the gesture interpreters.
      MultiInterpreter mintrp;
      Interpreter      intrp;

      mintrp = new DefaultMultiInterpreterImpl();

      intrp = new TapSelectInterpreter();
      intrp.setAcceptLeftButton(true);
      intrp.setAcceptRightButton(true);
      mintrp.add(intrp);

      intrp = new CircleSelectInterpreter();
      intrp.setAcceptLeftButton(false);
      intrp.setAcceptRightButton(true);
      mintrp.add(intrp);

      intrp = new MoveSelectedInterpreter();
      intrp.setAcceptLeftButton(true);
      intrp.setAcceptRightButton(true);
      mintrp.add(intrp);

      s.setGestureInterpreter(mintrp);

      //// 3. Set up the ink interpreters.
      mintrp = new DefaultMultiInterpreterImpl();

      intrp = new RandomColorInterpreter();
      mintrp.add(intrp);

      intrp = new LinearizeStrokeInterpreter();
      mintrp.add(intrp);

      s.setInkInterpreter(mintrp);

      //// 4. Make it so that by default, the Sheet will not add
      ////    right mouse button strokes as ink.
      s.setAddRightButtonStrokes(false);

      //// 5. Set up the pie menu.
      setupPieMenu(s);

      //// 6. Make the frame visible.
      f.setSize(600, 600);
      f.setVisible(true);
   } // of main

   //===   SELF-TESTING MAIN   =================================================
   //===========================================================================

} // of class

Here's the multi-class version, the one that is engineered better. Functionally, it's equivalent to the previous version, but it will be easier to extend.


Download code v11
import javax.swing.*;
import edu.berkeley.guir.lib.satin.*;
import edu.berkeley.guir.lib.satin.interpreter.*;
import edu.berkeley.guir.lib.satin.interpreter.commands.*;
import edu.berkeley.guir.lib.satin.interpreter.stroke.*;
import edu.berkeley.guir.lib.satin.widgets.*;

import edu.berkeley.guir.lib.satin.command.*;
import edu.berkeley.guir.lib.satin.objects.*;
import edu.berkeley.guir.lib.awt.geom.*;
import java.awt.geom.*;
import java.awt.event.*;

/**
 * An example of how to use the SATIN toolkit.
 * This app demonstrates how the Sheet works with Java Swing.
 */
public class Scribbler {

   //===========================================================================
   //===   CLASS VARIABLES   ===================================================

   static JFrame f;
   static Sheet  s = new Sheet();

   //===   CLASS VARIABLES   ==================================================
   //===========================================================================



   //===========================================================================
   //===   SELF-TESTING MAIN   =================================================

   public static void main(String[] argv) {
      f = new JFrame();
      s = new ScribblerSheet();

      //// 1. Make the Sheet the content pane for the frame.
      f.setContentPane(s);
      s.setBackground(java.awt.Color.white);

      //// 2. Set up the gesture interpreters.
      MultiInterpreter mintrp;
      Interpreter      intrp;

      mintrp = new DefaultMultiInterpreterImpl();

      intrp = new TapSelectInterpreter();
      intrp.setAcceptLeftButton(true);
      intrp.setAcceptRightButton(true);
      mintrp.add(intrp);

      intrp = new CircleSelectInterpreter();
      intrp.setAcceptLeftButton(false);
      intrp.setAcceptRightButton(true);
      mintrp.add(intrp);

      intrp = new MoveSelectedInterpreter();
      intrp.setAcceptLeftButton(true);
      intrp.setAcceptRightButton(true);
      mintrp.add(intrp);

      s.setGestureInterpreter(mintrp);

      //// 3. Set up the ink interpreters.
      mintrp = new DefaultMultiInterpreterImpl();

      intrp = new RandomColorInterpreter();
      mintrp.add(intrp);

      intrp = new LinearizeStrokeInterpreter();
      mintrp.add(intrp);

      s.setInkInterpreter(mintrp);

      //// 4. Make it so that by default, the Sheet will not add
      ////    right mouse button strokes as ink.
      s.setAddRightButtonStrokes(false);

      //// 5. Make the frame visible.
      f.setSize(600, 600);
      f.setVisible(true);
   } // of main

   //===   SELF-TESTING MAIN   =================================================
   //===========================================================================

} // of class


Download code v11
import javax.swing.*;
import edu.berkeley.guir.lib.satin.*;
import edu.berkeley.guir.lib.satin.interpreter.*;
import edu.berkeley.guir.lib.satin.interpreter.commands.*;
import edu.berkeley.guir.lib.satin.interpreter.stroke.*;
import edu.berkeley.guir.lib.satin.widgets.*;
import edu.berkeley.guir.lib.satin.command.*;
import edu.berkeley.guir.lib.satin.objects.*;
import edu.berkeley.guir.lib.awt.geom.*;
import java.awt.geom.*;
import java.awt.event.*;

/**
 * Sheet used in tutorial.
 */
public class ScribblerSheet
   extends Sheet {

   //===========================================================================
   //===   CONSTANTS   =========================================================

   /**
    * Number of frames to animate when zooming or rotating.
    */
   public static final int NUMFRAMES = 10;

   //===   CONSTANTS   =========================================================
   //===========================================================================



   //===========================================================================
   //===   PIE MENU INNER CLASS CALLBACKS   ====================================

   class RotateLeftListener implements ActionListener {
      public void actionPerformed(ActionEvent evt) {
         AffineTransform[] txArray = new AffineTransform[NUMFRAMES];
         txArray = AffineTransformLib.animateSlowInSlowOut(
                        AffineTransform.getRotateInstance(-Math.PI/2,
                           cmdsubsys.getAbsoluteLastXLocation(),
                           cmdsubsys.getAbsoluteLastYLocation()),
                        NUMFRAMES);
         GraphicalObjectLib.animate(ScribblerSheet.this, txArray);

      } // of method
   } // of inner class

   //-----------------------------------------------------------------

   class RotateRightListener implements ActionListener {
      public void actionPerformed(ActionEvent evt) {
         AffineTransform[] txArray = new AffineTransform[NUMFRAMES];
         txArray = AffineTransformLib.animateSlowInSlowOut(
                        AffineTransform.getRotateInstance(Math.PI/2,
                           cmdsubsys.getAbsoluteLastXLocation(),
                           cmdsubsys.getAbsoluteLastYLocation()),
                        NUMFRAMES);
         GraphicalObjectLib.animate(ScribblerSheet.this, txArray);

      } // of method
   } // of inner class

   //-----------------------------------------------------------------

   class ZoomInListener implements ActionListener {
      public void actionPerformed(ActionEvent evt) {
         AffineTransform[] txArray = new AffineTransform[NUMFRAMES];
         txArray = AffineTransformLib.animateSlowInSlowOut(
                   AffineTransformLib.scaleAndCenterAt(2,
                           cmdsubsys.getAbsoluteLastXLocation(),
                           cmdsubsys.getAbsoluteLastYLocation(),
                           ScribblerSheet.this.getBounds2D(COORD_ABS)),
                        NUMFRAMES);
         GraphicalObjectLib.animate(ScribblerSheet.this, txArray);
      } // of method
   } // of inner class

   //-----------------------------------------------------------------

   class ZoomOutListener implements ActionListener {
      public void actionPerformed(ActionEvent evt) {
         AffineTransform[] txArray = new AffineTransform[NUMFRAMES];
         txArray = AffineTransformLib.animateSlowInSlowOut(
                   AffineTransformLib.scaleAndCenterAt(0.5,
                           cmdsubsys.getAbsoluteLastXLocation(),
                           cmdsubsys.getAbsoluteLastYLocation(),
                           ScribblerSheet.this.getBounds2D(COORD_ABS)),
                        NUMFRAMES);
         GraphicalObjectLib.animate(ScribblerSheet.this, txArray);
      } // of method
   } // of inner class

   //===   PIE MENU INNER CLASS CALLBACKS   ====================================
   //===========================================================================



   //===========================================================================
   //===   CONSTRUCTORS   ======================================================

   public ScribblerSheet() {
      setupPieMenu(this);
   } // of constructor

   //===   CONSTRUCTORS   ======================================================
   //===========================================================================



   //===========================================================================
   //===   PIE MENU SETUP   ====================================================

   protected void setupPieMenu(JComponent comp) {
      PieMenu pm = new PieMenu();

      //// 1. Add the pie menu slices to the pie menu.
      ////    Order starts at north, goes counter clockwise.
      ////    Reason for '\n' is because pie menu is small, so
      ////    words don't fit correctly. Either resize or go
      ////    to next line.
      pm.add("Zoom\nIn").addActionListener(new ZoomInListener());
      pm.add("Rotate\nLeft").addActionListener(new RotateLeftListener());
      pm.add("Zoom\nOut").addActionListener(new ZoomOutListener());
      pm.add("Rotate\nRight").addActionListener(new RotateRightListener());

      //// 2. Add the pie menu to the component.
      pm.addPieMenuTo(comp);

   } // of method

   //===   PIE MENU SETUP   ====================================================
   //===========================================================================

} // of class

Part Six - Recognizers and Interpreters

So all of this is interesting and all, but how do we create new gestures, and do cool things like cut / copy / paste, or even clean up sketches into real shapes?

That's where Recognizers come in. Recognizers are black boxes that, given a stroke, return an n-best sorted list of classifications. That's it. Interpreters act on strokes, whereas Recognizers just classify strokes. See the diagram below for an example.



Interpreters can use recognizers if they want. Look at the diagram below. Here's what's happening:

  1. A "cut" stroke is made
  2. The stroke is eventually dispatched to the Interpreter
  3. The Interpreter asks the Recognizer to classify the stroke
  4. The Recognizer returns an n-best list of what it thinks the stroke is
  5. The Interpreter gets the top-ranked classification. It then executes the code to do a cut.




SATIN ships with Rubine's Recognizer, one that can be trained by example. (It's named after Dean Rubine, who designed and developed it as part of his PhD Dissertation at Carnegie-Mellon). All you have to do is create a name for the gesture category and then scribble a few examples. We've created a tool called Quill to help you do this (sometimes you'll hear us call it GDT or Gesture Design Tool. That was Quill's old name).

So let's show you an example of how gestures work. We'll do this first by adding some gestures to our sample app. There's an Interpreter we've defined called StandardGestureInterpreter. It does gestures that we consider "standard", but you can redefine the gestures if you want.

Note that when you load the app, it takes a little longer before anything happens. This is because the data file is being read in.

If the gestures aren't working, make sure you have the data/ directory copied over to whatever directory you're working in. SATIN assumes that there is a directory called data/interpreters/ in the directory the Java Virtual Machine is started from. This is where the gesture data is stored.


Download code v12
import javax.swing.*;
import edu.berkeley.guir.lib.satin.*;
import edu.berkeley.guir.lib.satin.interpreter.*;
import edu.berkeley.guir.lib.satin.interpreter.commands.*;
import edu.berkeley.guir.lib.satin.interpreter.stroke.*;
import edu.berkeley.guir.lib.satin.widgets.*;

import edu.berkeley.guir.lib.satin.command.*;
import edu.berkeley.guir.lib.satin.objects.*;
import edu.berkeley.guir.lib.awt.geom.*;
import java.awt.geom.*;
import java.awt.event.*;

/**
 * An example of how to use the SATIN toolkit.
 * This app demonstrates how the Sheet works with Java Swing.
 */
public class Scribbler {

   //===========================================================================
   //===   CLASS VARIABLES   ===================================================

   static JFrame f;
   static Sheet  s = new Sheet();

   //===   CLASS VARIABLES   ==================================================
   //===========================================================================



   //===========================================================================
   //===   SELF-TESTING MAIN   =================================================

   public static void main(String[] argv) {
      f = new JFrame();
      s = new ScribblerSheet();

      //// 1. Make the Sheet the content pane for the frame.
      f.setContentPane(s);
      s.setBackground(java.awt.Color.white);

      //// 2. Set up the gesture interpreters.
      MultiInterpreter mintrp;
      Interpreter      intrp;

      mintrp = new DefaultMultiInterpreterImpl();

      intrp = new TapSelectInterpreter();
      intrp.setAcceptLeftButton(true);
      intrp.setAcceptRightButton(true);
      mintrp.add(intrp);

      intrp = new CircleSelectInterpreter();
      intrp.setAcceptLeftButton(false);
      intrp.setAcceptRightButton(true);
      mintrp.add(intrp);

      intrp = new MoveSelectedInterpreter();
      intrp.setAcceptLeftButton(true);
      intrp.setAcceptRightButton(true);
      mintrp.add(intrp);

      intrp = new StandardGestureInterpreter();
      intrp.setAcceptLeftButton(true);
      intrp.setAcceptRightButton(true);
      mintrp.add(intrp);

      s.setGestureInterpreter(mintrp);

      //// 3. Set up the ink interpreters.
      mintrp = new DefaultMultiInterpreterImpl();
      mintrp.setAcceptRightButton(false);

      intrp = new RandomColorInterpreter();
      mintrp.add(intrp);

      intrp = new LinearizeStrokeInterpreter();
      mintrp.add(intrp);

      s.setInkInterpreter(mintrp);

      //// 4. Make it so that by default, the Sheet will not add
      ////    right mouse button strokes as ink.
      s.setAddRightButtonStrokes(false);

      //// 5. Make the frame visible.
      f.setSize(600, 600);
      f.setVisible(true);
   } // of main

   //===   SELF-TESTING MAIN   =================================================
   //===========================================================================

} // of class

Gestures

With this new addition, we should be able to do gestures like cut / copy / paste and so on. Here's a chart of our "standard" gestures. Try scribbling a few things, and try the gestures. If nothing happens, try the gesture again. Sometimes there are recognition errors. In fact, try bringing up the debugging window (Ctrl-Alt-W) to see some of the debugging messages.

The dot in the gesture signifies where you start gesturing.

Pan
Undo
Redo
Cut
Copy
Paste

Quill and the Gestures File

As we said before, Quill is a tool for designing gestures. Here's how to load it up. Go to the directory where the satin and gdt jar files are. Then, either run run-gdt.bat or java -jar gdt.jar. This will bring up Quill.

Go to the file-open menu, and go to data/interpreters/standard.gsa. You should have something that looks like the following:



Double-click on some of the names on the top-left, like "ViewPortLeft" and "CutA". It should show you what the gestures look like. Ok, now we're going to show you the code for how gestures are handled in StandardGestureInterpreter. You don't have to download or compile this code.

The parts to focus on are in bold and italics. Like other interpreters, the StandardGestureInterpreter has a method called handleSingleStroke(). This method asks a RubineInterpreter inner class to classify the stroke and take action. (Technically, given our definition of Recognizer and Interpreter, the RubineInterpreter isn't really an Interpreter since it doesn't take any actions. It's more of a skeleton for an interpreter.) 


//// See bottom of source code for software license

package edu.berkeley.guir.lib.satin.interpreter.commands;

import java.awt.*;
import java.awt.geom.*;
import java.util.*;
import edu.berkeley.guir.lib.awt.geom.*;
import edu.berkeley.guir.lib.debugging.*;
import edu.berkeley.guir.lib.satin.*;
import edu.berkeley.guir.lib.satin.objects.*;
import edu.berkeley.guir.lib.satin.command.*;
import edu.berkeley.guir.lib.satin.recognizer.*;
import edu.berkeley.guir.lib.satin.event.*;
import edu.berkeley.guir.lib.satin.interpreter.*;
import edu.berkeley.guir.lib.satin.interpreter.rubine.*;
import edu.berkeley.guir.lib.satin.stroke.*;

/**
 * Standard gestures, like cut, copy, paste, delete, and pan.
 *
 * <P>
 * This software is distributed under the
 * <A HREF="http://guir.cs.berkeley.edu/projects/COPYRIGHT.txt">
 * Berkeley Software License</A>.
 *
 * <PRE>
 * Revisions:  - SATIN-v1.0-1.0.0, Aug 02 1999, JH
 *               Created class
 *             - SATIN-v2.1-1.0.0, Aug 11 2000, JH
 *               Touched for SATIN release
 * </PRE>
 *
 * @author <A HREF="http://www.cs.berkeley.edu/~jasonh/">Jason Hong</A> (
 * @since   JDK 1.2.2
 * @version SATIN-v2.1-1.0.0, Aug 11 2000
 */
public class StandardGestureInterpreter
   extends GestureCommandInterpreterImpl {

   //===========================================================================
   //===   CONSTANTS   =========================================================

   static final long serialVersionUID = 1118908262546911989L;

   //===   CONSTANTS   =========================================================
   //===========================================================================



   //===========================================================================
   //===   INNER CLASS   =======================================================

   class StandardRubineInterpreter
      extends RubineInterpreter {

      public StandardRubineInterpreter(String str) {
         super(str);
      } // of constructor


      protected void
      handleSingleStroke(SingleStrokeEvent evt, Classification c) {
         String      str = (String) c.getFirstKey();
         TimedStroke stk = evt.getStroke();

         str = str.trim();

         if (stk.getLength2D(COORD_ABS) < 20) {
            return;
         }

         if (str.startsWith("ViewPort")) {
            handleViewPort(str, stk);
            evt.setConsumed();
         }

         if (str.startsWith("Center")) {
            handleCenter(stk);
            evt.setConsumed();
         }

         if (str.startsWith("Copy")) {
            handleCopy(stk);
            evt.setConsumed();
         }

         if (str.startsWith("Cut")) {
            handleCut(stk);
            evt.setConsumed();
         }

         if (str.startsWith("Paste")) {
            handlePaste(stk);
            evt.setConsumed();
         }

         if (str.startsWith("Delete")) {
            handleDelete(stk);
            evt.setConsumed();
         }

         if (str.startsWith("Undo")) {
            handleUndo();
            evt.setConsumed();
         }

         if (str.startsWith("Redo")) {
            handleRedo();
            evt.setConsumed();
         }

      } // of handleSingleStroke

      public Object clone() {
         return (new StandardRubineInterpreter(this.getFileName()));
      } // of method
   } // of class

   //===   INNER CLASS   =======================================================
   //===========================================================================



   //===========================================================================
   //===   CLASS VARIABLES   ===================================================

   private static final Debug debug = new Debug();

   //===   CLASS VARIABLES   ===================================================
   //===========================================================================



   //===========================================================================
   //===   NONLOCAL VARIABLES   ================================================

   protected boolean           flagShallow = true;
   protected RubineInterpreter intrp;

   //===   NONLOCAL VARIABLES   ================================================
   //===========================================================================



   //===========================================================================
   //===   CONSTRUCTORS   ======================================================

   public StandardGestureInterpreter() {
      intrp = new StandardRubineInterpreter("standard.gsa");
      commonInitializations();
   } // of constructor

   //-----------------------------------------------------------------

   private void commonInitializations() {
      setName("Standard Gesture Interpreter");
   } // of commonInitializations

   //===   CONSTRUCTORS   ======================================================
   //===========================================================================



   //===========================================================================
   //===   ACCESSOR / MODIFIER METHODS   =======================================

   /**
    * Set the GraphicalObject retrieval to be shallow (in terms of the data
    * structure, ie one level). This affects how cut, copy, paste, and delete
    * work.
    */
   public void setShallow() {
      flagShallow = true;
   } // of setShallow

   //-----------------------------------------------------------------

   /**
    * Set the GraphicalObject retrieval to be deep (in terms of the data
    * structure, ie one level). This affects how cut, copy, paste, and delete
    * work.
    */
   public void setDeep() {
      flagShallow = false;
   } // of setDeep

   //===   ACCESSOR / MODIFIER METHODS   =======================================
   //===========================================================================



   //===========================================================================
   //===   HANDLE GESTURE METHODS   ============================================

   public void handleSingleStroke(SingleStrokeEvent evt) {
      intrp.handleSingleStroke(evt);
   } // of method

   //-----------------------------------------------------------------

   /**
    * Handle an undo.
    * Override this method if you want.
    */
   protected void handleUndo() {
      try {
         cmdqueue.undo();
         getAttachedGraphicalObject().damage(DAMAGE_LATER);
      }
      catch (Exception e) {
         debug.println(e);
      }
   } // of handleUndo

   //-----------------------------------------------------------------

   /**
    * Handle a redo.
    * Override this method if you want.
    */
   protected void handleRedo() {
      try {
         cmdqueue.redo();
         getAttachedGraphicalObject().damage(DAMAGE_LATER);
      }
      catch (Exception e) {
         debug.println(e);
      }
   } // of handleRedo

   //-----------------------------------------------------------------

   /**
    * Cut whatever intersects the stroke.
    * Override this method if you want.
    */
   protected void handleCut(TimedStroke stk) {
      GraphicalObjectCollection gobcol;
      Iterator                  it;
      GraphicalObject           attach;
      GraphicalObject           gob;
      CutCommand                cmd = new CutCommand();

      gobcol = getGraphicalObjectsTouching(stk, 2);
      it     = gobcol.getForwardIterator();
      attach = getAttachedGraphicalObject();

      while (it.hasNext()) {
         gob = (GraphicalObject) it.next();
         if (gob != attach) {
            cmd.addGraphicalObject(gob);
         }
      }

      cmdqueue.doCommand(cmd);
   } // of method

   //-----------------------------------------------------------------

   /**
    * Copy whatever is in the center of the stroke.
    * Override this method if you want.
    */
   protected void handleCopy(TimedStroke stk) {
      GraphicalObject gob = getGraphicalObjectCenteredAt(stk);
      if (gob != null && gob != getAttachedGraphicalObject()) {
         cmdqueue.doCommand(new CopyCommand (gob));
      }
   } // of handleCopy

   //-----------------------------------------------------------------

   /**
    * Paste to the bottom-center of the stroke.
    * Override this method if you want.
    */
   protected void handlePaste(TimedStroke stk) {
      GraphicalObject gob = getAttachedGraphicalObject();
      Rectangle2D     rect = stk.getBounds2D(COORD_ABS);
      Point2D         pt   = new Point2D.Float(
                               (float) (rect.getX() + rect.getWidth()/2),
                               (float) (rect.getY() + rect.getHeight()));

      GraphicalObjectLib.absoluteToLocal(gob, pt, pt);
      if (gob instanceof GraphicalObjectGroup) {
         cmdqueue.doCommand(new PasteCommand((GraphicalObjectGroup) gob,
                                                   pt.getX(), pt.getY()));
      }
   } // of handlePaste

   //-----------------------------------------------------------------

   /**
    * Delete whatever is in the center of the stroke (except ourself of course).
    * Override this method if you want.
    */
   protected void handleDelete(TimedStroke stk) {
      GraphicalObject gob = getGraphicalObjectCenteredAt(stk);
      if (gob != null && gob != getAttachedGraphicalObject()) {
         cmdqueue.doCommand(new DeleteCommand (gob));
      }
   } // of handleDelete

   //-----------------------------------------------------------------

   /**
    * Center the screen to be at the mark.
    * Override this method if you want.
    */
   protected void handleCenter(TimedStroke stk) {
      GraphicalObject gob    = getAttachedGraphicalObject();
      Rectangle2D     gobbds = gob.getBounds2D();
      Rectangle2D     stkbds = stk.getBounds2D();
      double          x1     = gobbds.getX() + gobbds.getWidth()/2;
      double          y1     = gobbds.getY() + gobbds.getHeight()/2;
      double          x2     = stkbds.getX() + stkbds.getWidth()/2;
      double          y2     = stkbds.getY() + stkbds.getHeight()/2;

      AffineTransform[] txArray;
      txArray = AffineTransformLib.animateSlowInSlowOut(
         AffineTransform.getTranslateInstance(x1 - x2, y1 - y2), 10);
      GraphicalObjectLib.animate(gob, txArray);
   } // of handleCenter

   //-----------------------------------------------------------------

   /**
    * Handle a move view port command.
    * Override this method if you want.
    */
   protected void handleViewPort(String str, TimedStroke stk) {
      AffineTransform tx  = new AffineTransform();
      GraphicalObject gob = getAttachedGraphicalObject();

      double          len = stk.getLength2D(COORD_ABS);

      if (str.equalsIgnoreCase("ViewPortLeft")) {
         tx.translate(+len, 0);
      }
      if (str.equalsIgnoreCase("ViewPortRight")) {
         tx.translate(-len, 0);
      }
      if (str.equalsIgnoreCase("ViewPortUp")) {
         tx.translate(0, +len);
      }
      if (str.equalsIgnoreCase("ViewPortDown")) {
         tx.translate(0, -len);
      }
      if (str.equalsIgnoreCase("ViewPortDiagUpRight")) {
         tx.translate(-len/2, +len/2);
      }
      if (str.equalsIgnoreCase("ViewPortDiagUpLeft")) {
         tx.translate(+len/2, +len/2);
      }
      if (str.equalsIgnoreCase("ViewPortDiagDownRight")) {
         tx.translate(-len/2, -len/2);
      }
      if (str.equalsIgnoreCase("ViewPortDiagDownLeft")) {
         tx.translate(+len/2, -len/2);
      }

      AffineTransform[] txArray;
      txArray = AffineTransformLib.animateSlowInSlowOut(tx, 8);
      GraphicalObjectLib.animate(gob, txArray);
   } // of method

   //===   HANDLE GESTURE METHODS   ============================================
   //===========================================================================



   //===========================================================================
   //===   GET GOB METHODS   ===================================================

   /**
    * Get things that intersect the stroke.
    * Override this method to change the behavior of how objects intersecting
    * a stroke are selected. For example, in DENIM, this method is modified
    * such that DenimPanel objects can be deleted when zoomed out (ie shallow),
    * and strokes and phrases can be deleted when zoomed in (ie deep).
    */
   protected GraphicalObjectCollection
   getGraphicalObjectsTouching(TimedStroke stk, double thresh) {

      //// 0. Don't do anything if we're not a group.
      if (! (getAttachedGraphicalObject() instanceof GraphicalObjectGroup)) {
         return (null);
      }

      GraphicalObject           gob     = getAttachedGraphicalObject();
      GraphicalObject           gobtmp  = null;
      Shape                     s       = stk.getBoundingPoints2D(COORD_ABS);
      GraphicalObjectGroup      gobgrp  = (GraphicalObjectGroup) gob;
      GraphicalObjectCollection gobcol  = new GraphicalObjectCollectionImpl();
      int                       depth;

      //// 1. Set for shallow / deep search.
      if (flagShallow == true) {
         depth = SHALLOW;
      }
      else {
         depth = DEEP;
      }

      //// 2.1. Try searching using the stroke bounds.
      gobcol = gobgrp.getGraphicalObjects(COORD_ABS,
                                          s,
                                          ALL,
                                          depth,
                                          INTERSECTS,
                                          thresh,
                                          gobcol);
      return (gobcol);
   } // of method

   //-----------------------------------------------------------------

   /**
    * Get the GraphicalObject near the center of this TimedStroke.
    */
   protected GraphicalObject
   getGraphicalObjectCenteredAt(TimedStroke stk) {
      return (getGraphicalObjectCenteredAt(stk, DEFAULT_SELECT_THRESHOLD));
   } // of method

   //-----------------------------------------------------------------

   /**
    * Get the GraphicalObject near the center of this TimedStroke, with
    * the specified threshold.
    */
   protected GraphicalObject
   getGraphicalObjectCenteredAt(TimedStroke stk, double thresh) {

      GraphicalObjectCollection gobcol;
      GraphicalObject           gobtmp = null;

      //// 1. Get the graphical objects touching the stroke.
      gobcol = getGraphicalObjectsTouching(stk, thresh);

      //// 2. Return the topmost Graphical Object first.
      if (gobcol.numElements() > 0) {
         gobtmp = GraphicalObjectLib.getTopmostGraphicalObject(gobcol);
      }

      return (gobtmp);
   } // of method

   //===   GET GOB METHODS   ===================================================
   //===========================================================================



   //===========================================================================
   //===   CLONE   =============================================================

   public Object clone() {
      return (new StandardGestureInterpreter());
   } // of clone

   //===   CLONE   =============================================================
   //===========================================================================

} // of class

//==============================================================================

/*
Copyright (c) 2000 Regents of the University of California.
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:

1. Redistributions of source code must retain the above copyright
   notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright
   notice, this list of conditions and the following disclaimer in the
   documentation and/or other materials provided with the distribution.

3. All advertising materials mentioning features or use of this software
   must display the following acknowledgement:

      This product includes software developed by the Group for User
      Interface Research at the University of California at Berkeley.

4. The name of the University may not be used to endorse or promote products
   derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED.  IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.
*/

Using RubineInterpreter to Handle Strokes

Note how the stroke is handled in the inner class. It has code like: 

         if (str.startsWith("Copy")) {
            handleCopy(stk);
            evt.setConsumed();
         }

         if (str.startsWith("Cut")) {
            handleCut(stk);
            evt.setConsumed();
         }

Compare the names in the standard.gsa file in Quill with the names here. You'll see that they are very similar. The RubineInterpreter classifies the stroke and also gives us the gesture category name. Once we have the name of the gesture category, we just act on it.

Another thing to note is that we have lots of strange names for gestures, like "CutA" and "CutB". Why are there multiple Cut gesture categories? It turns out that this is to overcome one problem with Rubine's Recognizer. Rubine's Recognizer doesn't work well with gestures that are different sizes or rotated. To overcome this problem, we just created separate categories for these kinds of gestures, but named them similarly.

Then, in the Interpreter, instead of doing an exact String match (with .equals()) we did a .startsWith() match. Problem solved.

If you have created a gesture file with Quill and are using it in SATIN, you will notice that sometimes you will get "tap" as the recognized gesture even though "tap" is not defined. This indicates an error condition meaning that there aren't enough gestures and/or gesture categories defined (or that they aren't sufficiently different for the recognizer to classify). Adding a few more of each will likely solve the problem.

Part Seven - Graphical Objects

We're going to take a slight detour before coming back to recognizers and interpreters. So we've seen that we can have ink strokes on the Sheet. Can we have anything else? The answer is yes!

If you looked at the code for StandardGestureInterpreter, you'll notice that there are references to something called a GraphicalObject. A GraphicalObject is an object that can be displayed and has some behaviors. It turns out that the Sheet is also a GraphicalObject.

So let's go through an example of how to add some GraphicalObjects to the Sheet. We'll give some code first, and then describe what's going on after. Make the changes, compile, and run. Below is a screenshot of what should appear.


Download code v13
import javax.swing.*;
import edu.berkeley.guir.lib.satin.*;
import edu.berkeley.guir.lib.satin.interpreter.*;
import edu.berkeley.guir.lib.satin.interpreter.commands.*;
import edu.berkeley.guir.lib.satin.interpreter.stroke.*;
import edu.berkeley.guir.lib.satin.widgets.*;

import edu.berkeley.guir.lib.satin.command.*;
import edu.berkeley.guir.lib.satin.objects.*;
import edu.berkeley.guir.lib.awt.geom.*;
import java.awt.geom.*;
import java.awt.event.*;
import java.awt.*;

/**
 * An example of how to use the SATIN toolkit.
 * This app demonstrates how the Sheet works with Java Swing.
 */
public class Scribbler {

   //===========================================================================
   //===   CLASS VARIABLES   ===================================================

   static JFrame f;
   static Sheet  s = new Sheet();

   //===   CLASS VARIABLES   ==================================================
   //===========================================================================



   //===========================================================================
   //===   SELF-TESTING MAIN   =================================================

   public static void main(String[] argv) {
      f = new JFrame();
      s = new ScribblerSheet();

      //// 1. Make the Sheet the content pane for the frame.
      f.setContentPane(s);
      s.setBackground(java.awt.Color.white);

      //// 2. Set up the gesture interpreters.
      MultiInterpreter mintrp;
      Interpreter      intrp;

      mintrp = new DefaultMultiInterpreterImpl();

      intrp = new TapSelectInterpreter();
      intrp.setAcceptLeftButton(true);
      intrp.setAcceptRightButton(true);
      mintrp.add(intrp);

      intrp = new CircleSelectInterpreter();
      intrp.setAcceptLeftButton(false);
      intrp.setAcceptRightButton(true);
      mintrp.add(intrp);

      intrp = new MoveSelectedInterpreter();
      intrp.setAcceptLeftButton(true);
      intrp.setAcceptRightButton(true);
      mintrp.add(intrp);

      intrp = new StandardGestureInterpreter();
      intrp.setAcceptLeftButton(true);
      intrp.setAcceptRightButton(true);
      mintrp.add(intrp);

      s.setGestureInterpreter(mintrp);

      //// 3. Set up the ink interpreters.
      mintrp = new DefaultMultiInterpreterImpl();
      mintrp.setAcceptRightButton(false);

      intrp = new RandomColorInterpreter();
      mintrp.add(intrp);

      intrp = new LinearizeStrokeInterpreter();
      mintrp.add(intrp);

      s.setInkInterpreter(mintrp);

      //// 4. Make it so that by default, the Sheet will not add
      ////    right mouse button strokes as ink.
      s.setAddRightButtonStrokes(false);

      //// 5. Add some rectangular objects to the Sheet.
      GraphicalObject gob;

      gob = new PatchImpl new Rectangle(0, 0, 150, 150));
      s.add(gob);

      gob = new PatchImpl new Rectangle(200, 100, 150, 150));
      s.add(gob);

      //// 6. Make the frame visible.
      f.setSize(600, 600);
      f.setVisible(true);
   } // of main

   //===   SELF-TESTING MAIN   =================================================
   //===========================================================================

} // of class




Patches

In the code, we added two PatchImpl objects. PatchImpl implements the Patch interface. Think of patches as windows, except that they can be arbitrarily shaped.

You should be able to select, move, copy, cut, and paste these two Patch objects the same as any other objects. It turns out that most of the gestures we have implemented work on GraphicalObjects. Strokes and Patches are just two subclasses of GraphicalObject.

Patch objects can actually contain other objects. In the screenshot above, you should be able to see this. In fact, if you move the Patch, the objects within also move.

However, one interesting thing to note is that the strokes inside of the Patch aren't straight and are all black! But strokes outside are straightened and have the random color assigned. The reason for this is that Patches also have their own gesture and ink interpreters. Again, check out the SATIN powerpoint slides for an animation on exactly how stroke dispatching works.

Other things to know about Patches:

  • Patches have Style objects that you can change.
  • Patches can be shapes other than rectangular. Pass in a Polygon into it and see what happens. In fact, you can pass in any java.awt.Shape object into it. (Sorry, it doesn't handle curves very well yet).
  • Patches can have their own gesture and ink interpreters.

Part Eight - Tying Interpreters and Graphical Objects Together

In this part, we show one way to integrate Interpreters with GraphicalObjects, by doing some clean-up of recognized shapes. Below is an example of clean-up. Given a sketch that sort of looks like a rectangle, replace it with a Patch that is rectangular.

Below is another example of clean-up. SketchySPICE, one of the sample applications that ships with SATIN, lets you do very simple recognition and clean-up of sketched AND and OR gates.


Below, we show one way of doing clean-up. The general idea is:

  • Run the stroke through an interpreter that recognizes strokes
  • See what the top-ranked recognized object is
  • Replace the stroke with a GraphicalObject that represents the recognized object
[This part in progress.]

[Add in how to use Quilt output file] By default, all Quill output files (*.gsa) are read in from the data/interpreters directory (where "data" is off of the directory where you start your Java program). These files can then be read in from a sublcass of RubineInterpreter. For example, suppose you had a gesture file named "alphabet.gsa". Here's a code fragment to read it in: 

...
public class AlphabetGestureInterpreter
   extends RubineInterpreter {

   // constructor
   public AlphabetGestureInterpreter() {
       super("alphabet.gsa");
   } // of constructor

} // of class
There are other constructors for doing more sophisticated operations. See the RubineInterpreter javadoc for more information.

[Add in how to use and override RubineInterpreter]

[Add in how to create a ReplaceShapeInterpreter]

Part Nine - Dynamic Views
This section discusses how dynamic views work in SATIN. You can make applications with SATIN without having to mess with views, but it's here in case you want to do more sophisticated operations.

Views let you change how objects are displayed on screen. A GraphicalObject can have multiple views, which can change depending on context. For example, you could change the view to show more information when zoomed in. This is known as semantic zooming.

[This part in progress.]

[Describe reasons for multiple views]

[Show how semantic zoom views work]

· Copyright © 1998-2003 by the Regents of the University of California · Last updated Wednesday December 18 2002