drawing interface

Katherine Ye July 2018

INTERLACE.AI is a digital interface for drawing that both augments and de-augments the expressive powers of the artist. The novelty of the interface is derived from a simple implementation of a classic AI technique.


I wrote a score for a performance using With several collaborators, we performed the score at the residency SCORES: Propositions for Notating Performance.

Here's a video of the performance. I strongly recommend watching the video first so you can draw your own opinions, then watching the video accompanied by the "liner notes" below.

Warning: performance spoilers follow!


Here are some of the questions and realities we explored in the performance, taken from the score I wrote:

Here are some of the intended effects of the performance, also taken from the score I wrote:

Performance commentary

Below I give some notes on specific minutes of the video. I refer to the person drawing as the "illustrator" (red lines) and the person co-drawing as the "mediator" (blue lines).

Act I

In the process of drawing the still life in Act I of the performance

00:39 This audience member was actually a plant. This drawing went smoothly because we had rehearsed with the interface, the still life, and the mediator ahead of time.

Act II

The still life in Act II of the performance

00:40 This audience member was not a plant, but a genuine volunteer. However, the mediator had seen the still life ahead of time.

The first part of the drawing

1:19 We had to add more objects to the still life because people were drawing faster than we thought they would.

1:29 This moomin-looking character is just one line in the drawing, but it's one of my favorite parts of the performance. We had asked the mediator to demonstrate an interface with predictive and refinement capabilities, so, eerily, this moomin appeared to be drawn on the screen before the illustrator had finished the stroke.

Here are the mediator's instructions from the score:

About halfway through the drawing, the MEDIATOR starts an improvised glitching process (see below [in the score] for possible glitches). Glitches should still be as fast as possible and close to the illustrator’s cursor.

In general, each individual unexpected action the mediator takes should be something that an algorithm or filter in a drawing interface might plausibly do. But taken together, the sequence of actions should become increasingly implausible: the illustrator should begin to suspect that the behavior of the interface, taken as a whole, could not possibly have been programmed in.

In this performance, in addition to predicting lines, we also had asked the mediator to act like a Photoshop brush. Where the illustrator drew a straight line, the mediator might draw a dotted line or a curly line. Some of the strokes were mirrored, and others were drawn normally. Finally, we asked the mediator to push the audience's horizon for "suspension of disbelief" by starting to act increasingly humanlike, e.g. by writing the illustrator's name.

The interface begins to glitch

1:33 The audience member's name is Rachel.

The algorithm reveals herself

1:51 As the drawing becomes increasingly chaotic and unpredictable, I ham it up by saying misleading things like "I didn't program that in!" (it's true, I didn't) and "It seems that the Rachel woke up the algorithm." At the end of the drawing, I ask the mediator to come out of a back room by saying, "Would the algorithm like to reveal themselves?"

Acts I & II: behind the scenes

2:02 In case it wasn't clear, a human—the mediator—had been doing the drawings all along. This is the simple idea behind every line you draw is actually "implemented" by another human. This other human, the mediator, sees every line you draw and can choose to draw whatever they want on top. (They can faithfully carry out their orders. Or they can disobey. Or they can help.) You, as the illustrator, see only the lines the mediator makes for you—none of your own lines.

2:03 In Act I, the mediator is asked to follow the illustrator's line as precisely and quickly as possible. Now you see what that harmony looks like.

2:14 Safari on the iPad is occasionally unable to distinguish a draw action from a highlight action, so the illustrator's cursor (shown to the mediator) is sometimes highlighted.

The first glitch

2:53 The mediator instinctively performs some autocorrection here. The second audience member had drawn a stray line, which the mediator does not reproduce.

3:14 The mediator later told me that it was very hard to decide on improvisational drawing tactics while the illustrator was drawing this fast.

The mediator joins us in the main room

4:04 The artist may be present, but the algorithm is absent.


4:11 For the final act, an audience volunteer was asked to draw the mediator from life. I was interested in seeing how the usual observer/subject power dynamic was complicated. Indeed what happened in the next few minutes was more beautiful than I could have imagined.

4:24 The mediator didn't like the illustrator's first few attempts to draw her. Each of them got impatient and cleared the canvas once (I cut one from the video).

4:52 I love the line quality here. But whose line is it anyway?

The illustrator drawing the mediator

5:10 The illustrator and the mediator drew in silence for the first few minutes. (I hadn't given them instructions to talk or not talk.) Then I asked them to describe what they were seeing out loud as they were drawing it, and to negotiate the drawing verbally with each other. Here's one exchange they had while drawing:

I: Now your bangs are kind of coming up and over your face. Oops, a little low. That looks more like it, yeah.
I: I'm doing the other side of your hair. Coming up, up. This is fun. Oops.
I: Now I want to try some of your eyes, like some of your features. Not totally satisfied with that angle.
M: [laughs]
I: Now I really wanna make sure we get your nose good. So, coming down, it's almost like touching your... like all the way out to here.
M: All the way to there?!
AUDIENCE: [laughs]
M: I'm gonna stop here!
I: No, no, no, keep going, keep going!
M: No! [laughs]
I: Yes, it goes—it goes all the way to your cheek!

Here's the result of their negotiated nose territory:

The illustrator drawing the mediator, shown from both points of view

In the image above, notice how the mediator has corrected some of the mistakes the illustrator made, like forgetting to lift their pen. The tear line was also the mediator's addition.

5:32 The lag got worse here. (We were using a tethered wifi connection in the backwoods of upstate New York, but there were also even more intermediate layers—human, software, hardware—at which lag could be introduced.) So it became essential that the illustrator and the mediator communicate verbally.

6:42 We ended the performance with a group discussion. Several people said they thought the mediator had too much power, and wanted to redesign the interface to equalize the power dynamic. From my experience playtesting the role of mediator, I found that it was the route of least resistance to play the algorithm: I would become hypnotized and enter a flow state when tracing another person's drawing. I would want to reproduce the drawing as accurately as possible, and I didn't often think to exercise my own agency.

Here are some related papers, performances, pieces, and practices.

In human-computer interaction and human/"AI" interaction:

Constrained or deaugmented analog drawing interfaces: Digital drawing or hand-eye interfaces: Scores for drawing: Existing visual artist practices


Let me know if you're interested in doing a co-drawing together on, using the software, or seeing its code. If there's enough interest, I'm happy to open-source the code and the score.

Future work

On the technical side, the software behind could use many performance improvements, server features, and interface features.

On the speculative side, I'd like to explore a variety of performance propositions for the relationship between the illustrator and the mediator. For example:

The illustrator and the mediator proceed line-by-line. Each time the illustrator draws a line, the mediator draws a line inspired by it or improvised off it, but it cannot be the same line. Each time the mediator draws a line, the illustrator draws a line inspired by it or improvised off it, but it cannot be the same line.
Practice together until you achieve groupmind, where both illustrator and mediator are in control and can produce a drawing in perfect harmony.


Thanks to Georgia Wall, Sarah Pitan, and Emma Brown for being willing accomplices in the performance, and thanks to everyone at SCORES and the Rochester Folk Arts Guild for feedback and community. This piece and performance could not have happened without them.

Lucy Siyao Liu provided helpful feedback on an early draft of this writeup, and Lea Albaugh gave pointers to related HCI papers. My understanding of "intimate interfaces" was influenced by early discussions with Robert Ochshorn and Michael Nielsen.

back to home