One of the most famous generative textile projects to date is Janelle Shane’s SkyKnit. The project itself runs on top of TensorFlow: you can see the full library for the neural net textgenrnn on GitHub. Playing with it via Google Collab is an easy way to get some context for how the generative model works, and particularly for its reliance on pattern recognition within a large dataset, which is crucial for understanding and enjoying the results of SkyKnit.

This project is a great way to understand the relationship between knitting and code: the initial dataset was trained from StitchMaps, which is an attempt at standardization of the knitting pattern form. There's lots to explore in the dataset visually and textually at https://stitch-maps.com/ - here’s an example:

The written instructions provide a particularly strongly “code-like” overview of the pattern:

Rows 1, 5, and 10 (RS): [K2tog] 3 times, [yo, k1] 6 times, [k2tog] 6 times, [yo, k1] 6 times, [k2tog] 3 times.
Rows 2, 4, 6, and 8: Purl.
Rows 3, 7, and 9: Knit.

However, this rigidly formatted dataset was augmented by a crowdsourced collection of patterns decidedly more diverse. As this section from the documentation (available as an ebook through Ravelry: ) notes:

“WHERE THE TRAINING DATA CAME FROM
4728 of the patterns were from stitch-maps.com. These patterns are in consistently formatted, very logical knitspeak, as required by stitch-maps. Most of these are simple swatches; none have extra directions. 500 of the patterns were crowdsourced. These patterns are NOT consistently formatted (one even contained an Emily Dickinson poem) and range from sweaters to cowls to unmentionables. A significant portion involve tentacles.
Janelle trained the SkyKnit algorithm with a few different combinations of these - some of the results in this collection are from training just on stitch-maps, some are from the tentacle dataset, and some are from a combination of the two. The original patterns aren't recognizable in the final results - although it should be easy to tell whether a particular pattern was trained on the well-behaved dataset!”

As with GitHub, there’s a repository of sorts for the collection process discussion on Ravelry: this lengthy forum includes both the initial input on patterns to seed the generation, and many resulting interpretations of the generated patterns. Here’s a screenshot of the seed post, which has been updated to include links to the coverage:

There are lots of generated patterns worth exploring in this dataset - we’ve selected one that definitely takes some influence from the tentacle patterns in the crowdsourced data, and importantly one that people have knitted so that you can see both pattern and “object.”

Some of the generated pattern name highlights include:

• 282 flukes
• booties twisted
• fishcock
• jackdom
• make caows and shapcho
• prom net
• sea no
• snowing leaves
• techsex
• tuck socks

And so forth.

Here’s the “code” from one particular pattern for discussion, although given its relationship to both the “original” patterns as code and the textgenrnn code, you might find it useful to not approach it in isolation but as a product of a craft ecosystem:

tiny baby whale soto
co 16 st using a 6– for garrent - purl 1 st at end of round and then insert stitches of the rnd. as you come to be sure to work in the round.
using the tentacles and join for working in the round, being careful not to twist the stitches.
keep the last stitch from co edge, ending with round 5.
instep stitch pattern will be picked up all end of the sock, ready through stitch markers and join to work in round. 33 (58, 62, 68) sts using the tail between you worked left
on the rs row. [12 (18, 19, 21, 13) st and 2 adslip st tbl, p3[3, 3] times, (3-st on dpns), k2tog and turn. [108 (112, 120, 134) stitches]
next round: k2[1, 0, 0, -, -], k1, p2, k3, p2, k2, p2, k2] 3 times, p1, k18, p2, k2, p2, k5, p2, k1, p2, k2, p2, k2, p2.
next row: (p1, sl5) 3 times, p1, (kfb, k1) 3 times, k2, sl m, k5, p
cut yarn, leaving a 6 inch tail.
knit 5 [6, 6, 7, 8, 9, 10, 11] sts of needle 1, k2 tog to the end of the row. [101[120, 132] sts onto the bottom of the heel is facing you, pick up and knit 15 stitches and work the purl stitch (do the ears of the round) in the way purl.
finish with the new front of the hat for seaming from the working needle to join to work in the round.
repeat row 1.
row 1 [rs]: slpwyf, k4, p3, [k4, m1, k1] to last 3 stitches, p2, k3.
row 2 [ws]: k4, [p34, k1] twice, p2.
rows 9 – 31: k to 3 sts before first marker, ssk, k2tog, sm, ssk, k1, turn.
row 18 [mc]: k.
row 22 [mc]: k to 3 sts before next marker, k2tog, sm, ssk, k1, ssk, knit to 3 stitches before next marker, k2tog, sm, ssk, k1, k2tog [3 stitches remain, m1r, slip marker -- 32 sts)
round 21: k
row 26 [rs]: [kfb, k1, kfb] repeat to end of round. 20 sts remain by work by 40 (44, 60, 62, 64) stitches]
row 13 [mc]: k to 2nd m, m1l, k3, m1l, work to end. {10 sts.
rnd 17 (rs): knit to 3 sts before first marker, p1, repeat from * to end of row.
repeat the last round until piece measures 13[15.5, 14.75] inches from the end of the round p more stitch from the top edge. 88 (83, 93, 101) sts on hold on needle 3 on needle, k2tog [16 total st).
next row (wrong side): slip 1 wyib (16 sts)
cut yarn and thread through sts on the needle.
weave in ends with cc and the next step 10 minutes.
finishing
weave in ends.

Here’s Gloria Hanlon’s version of the final output (as shown in this fantastic Atlantic article describing how knitters “debug” the patterns):

There are lots of productive ways to explore SkyKnit - the most interesting, of course, is to try and follow a pattern. In the process of knitting a generated pattern, one becomes the compiler - the threads of knitters who have tried to follow SkyKnit’s generated guidelines particularly capture that process, and the ways in which the “compiler” handles errors and ultimately will vary in their interpretation of the coded directions.

The disclaimer on the Raverly pattern listing notes how this guarantees individually distinct, yet computationally similar, outcomes:

“With one or two exceptions, the SkyKnit patterns aren’t knittable without at least some debugging, and each knitter’s solutions makes the product look completely different. After seeing the initial test knits and the variations that individual knitters produced, it became clear that the variety was part of the charm of this project, and so we decided to create one large collection. The SkyKnit patterns are the result of a collaboration between the knitters of Ravelry (mostly the LSG forum), stitch-maps.com (run by JC Briar) and Janelle Shane of aiweirdness.com. There are a couple four-letter words in the titles and instructions, which is why there is an adult-content warning.”

Discussion Questions

• Are textile instructions a domain-specific language?
• What are the similarities between “debugging” a pattern and a program?
• How do we read “rules” differently when material output is on the line?
• What other connections between code and craft are useful to understanding both?
• Notice the highly computational, procedural aspects of knitting rules - if you aren’t a crafter, does that change how you think about textiles?

Hi, I’m Nick Montfort and am very glad the organizers have invited a discussion of my recent book, Exploratory Programming for the Arts and Humanities, second edition. The Critical Code Studies Working Group helped me write the first edition of this book, which I’ve now reorganized, updated, and expanded. This edition is open access and freely available in three digital formats.

Programming can be used to explore what computing can do, as millions learned when writing BASIC programs in the home computer era, in the 1970s and 80s. In this book, I focus on how programming can be used as a humanistic method of inquiry and for creative computing.

The book is for people without any programming experience. So, if you’re interested in learning to program, you are really the focus of this week. I invite you to make use of the book using the four methods of learning that I outline in 1.10.

Many of you do know how to program. I’ll ask you to help our new programmers begin to understand not only the mechanics of programming but also how to approach computing and code critically.

I encourage you new programmers to put your copy of Exploratory Programming alongside your computer — you can do that even if you’re using a digital format, if you put the digital text on a phone or tablet. This way you will be compelled to type in the code rather than copying and pasting, and you’ll get a better feel for programming from scratch.

Chapter 1 is the only part of the book meant to be read like a standard textbook, so please read it, and then do the installation and setup in Chapter 2. We can then move on — and I hope that many new programmers will, with effort, be able to get all the way through Chapter 7, which includes some critical discussion of typical introductory programming exercises.

Our lives are surrounded by textiles–from the clothes we wear, to the sheets and blankets on our beds, to the covers on our furniture in our homes, offices, and cars–however, they are rarely noticed or appreciated. Historically shunted to the sidelines as a “craft” or “women’s work,” the myriad of textile traditions - from sewing and quilting, to knitting, weaving, and crochet, to embroidery and lace making, and beyond - are only rarely welcome in galleries both present and past. Instead, these traditions are often associated with the domestic sphere and consequently devalued.

Despite this (and in some ways because of this), there is a powerful history in the computational potential of textiles that provides an interesting perspective to our code critique exercise. Throughout history, textiles have been used to give a voice to those who society silenced, and coded messages are a part of this history. While the myth of the Underground Railroad quilts is likely not true, there are plenty of other examples of coded messaging in textiles. For instance, in World War II, women knitters were not noticed, which provided them ample opportunities to encode messages in their knitting by representing Morse Code (you can read a brief synopsis of WWII usage of knitting here.) Textiles continue to be powerful tools for information communication and visualization, particularly in feminist frameworks (here’s a brief discussion of this is the context of pandemic data visualization, for instance.)

Beyond encoded messages, textile instructional patterns that are followed to create a crafted object are quite similar to programmatic code. Patterns are sequential instructions that have specialized syntax which also include variables, loops, conditionals, algorithms, etc. While the vocabulary may be different between textile crafters and programmers, the concepts are the same. There’s also some important fundamental similarities between how code and patterns are distributed, remixed, and reimagined, which we hope will particularly come through in the sample “code” we’ve chosen for this discussion thread.

Our thread this week offers the provocation of patterns as code: indeed, the code offered for critique is a pattern generated through code. We’re focusing particularly on the relationship of textile work to generative projects, as the underlying structures of both code and pattern are made more visible through this juxtaposition.

Generating Context: Our Work

For context, both of us work frequently with different types of “generative” textile projects that are not quite like the one we’ve selected for code critique, but do have some common threads Here’s some links to information on some of our work for context:

• Patchwork Poetics (generated quilt poetry visualization)
• Re:traced Threads: Generating Feminist Textile Art with Tracery
• Masked Making: Uncovering Women’s Craft Labor During COVID-19
• Embroidered Ephemera (generated cross-stitch from tweets)
• Tension Assembly (generated pandemic cross stitch)

To foster discussion in these intersections, this past year, we launched Knit&Perl, a community at the intersection of technology and text/iles. In addition to discussing here, come join us as we develop lines of inquiry, creativity, and reflection. The community is on Discord, ping us for an invite (and, if you happen to be a Discord researcher, ask Anastasia about their graduate student’s awesome new Discord research group D/ARC!)

Along with this contextual post launching Week 2, we have a code critique thread launching on SkyKnit, and watch for a generative thread from Nick Montfort on exploratory programming in the humanities!

Very briefly, face-hallucination is the process of creating a realistic, artificial face from a blurred image. PULSE (Menon et al.) relies on the nVidia StyleGAN engine (Karras et al.), a more general machine-learning framework that, in this case, learns facial features in order to create plausibly realistic artificial faces . StyleGAN in turn was trained using the ffhq dataset, a collection of 70,000 high-quality images of faces collected from Flickr (Karras et al.).

The first impulse in a code critique might be to find a mechanistic explanation within the text of the code used in the performance. We determined quickly, however, that no such simple mechanism existed. The original Flickr images were not encoded by race, and while the distribution of subjects was skewed as compared to the US population (young women being overrepresented, Black people being underrepresented, (Salminen et al.), StyleGAN appeared to be able to generate realistic faces that reflect Western perceptions of different races. The StyleGAN code extracts features from any set of images at multiple scales, with the same code used to generate artificial images of bedrooms, cars, cats and people. The PULSE code simply constrained the facial generation process of StyleGAN to conform to a blurred image given as input. (This video shows StyleGAN in action.)

Therefore, rather than basing our critique in text, we turned to performance theory. Below one can see StyleGAN performing.

Source: Karras et. al.

The above are images of bedrooms that do not exist. StyleGAN took as input 50k images of bedrooms, extracted features at scales ranging from fine-grained textures to larger shapes, and then generated the images above based on those learned styles. The images are the result of the interaction of StyleGAN’s two neural networks. One generated images, and the other sought to distinguish between generated and actual images. As both networks improved, the quality of the resulting artificial images also improved (thus the name of the technique: Generative Adversarial Networks).

Below are faces generated by StyleGAN using the ffhq dataset. Like the bedrooms above, these people never existed.

Source: Karras et. al.

The top row and leftmost column are source images. The remaining faces are constructed based on combining styles from the two sources. It works—we can see the similarities in the source images reflected in the new images.

PULSE is a more narrowly focused project that leverages StyleGAN to solve a specific problem: Given a blurred image as input, create a plausible constructed image that can be downscaled back to the original blurred image. Again, all of these are constructed images. Note that the purpose is not to retrieve the original unblurred image from the blurred image, but rather to create a novel image that is consistent with the blurred image.

Source: Menon et al.

Birhane’s performance uses PULSE slightly differently (again, refer to Figure 2 on page 66). She begins by presenting three non-blurred images of her own face, then blurs each of those images, and finally requests PULSE/StyleGAN to construct an artificial face based on the blurred image. The audience would expect the constructed face to match, perhaps imperfectly, Birhane’s initial images. They do not, although we can see echoes of the styles chosen by StyleGAN.

The focus of Birhane and Guest's article is suggested by their title: "Towards Decolonising Computational Sciences," and PULSE is offered as only one brief example. Birhane and Guest argue that the problem is not only the lack of diversity in software teams and training data, though they do identify those as problems. For Birhane and Guest, lack of diversity "is a symptom of the subtle white and male supremacy under which the computational fields operate, which assume and promote whiteness and maleness as the ideal standards” (Birhane and Guest).

We add the following observations to their analysis. StyleGAN, for all its capability in identifying classes of features, did not (and could not) derive the feature of race. As long as the audience understands the images in play are artificial, the morphing of images across race (and sex, and age) is simply an interesting trick. When a real face enters the mix, the audience expects that race would be preserved across generated images. In Birhane’s performance, PULSE/StyleGAN fails to conserve race, and we are left with a profound feeling of disquiet that was not present for the manipulation of artificial images. Our Theatre Journal paper is entitled “The Nonmaterial Mirror,” and in Birhane’s performance, she figuratively looks into the PULSE/StyleGAN mirror and asks, “How do you see me?”

In our original TDR article, we defined four tenets of Nonmaterial Performance, which we relate here to PULSE:

• Code abstracts: Most code abstractions are based on the decisions of authors. StyleGAN automates the abstraction process by reducing a corpus of similar images to features it identifies.
• Code performs: The codebases of StyleGAN and PULSE are, for the most part, just plumbing for feature extraction and manipulation. An analysis of the text of the code is of limited use. What is compelling about the code is its performance, and that is where we locate our analysis.
• Code acts within a network: The absence of an abstraction for race might be unproblematic in a network in which all faces are expected to be artificial. Changing the network, as Birhane and Guest did, changes the stakes.
• Code is vibrant: Code is fascinating in part because it cannot be constrained to a single interpretation. Birhane’s performance took PULSE in a direction its creators had not anticipated.

We return to our essential question: Why does bias exist in AI? In particular, why does bias exist in this AI when the codebase contains no reference to race and the training set has at least some degree of diversity?

StyleGAN could only create abstractions based on the pixels it was given. When humans look at images, particularly faces, our perception is informed by far more than the pixels. When real faces are introduced and manipulated, particularly when restoring a real face, StyleGAN’s limitations become jarring.

From a Critical Code Studies perspective, the rhizomatic performance of code becomes untethered from its text. The code is inscrutable without its performance.

​​Works Cited

• Birhane, Abeba and Olivia Guest, “Towards Decolonising Computational Sciences,” Women, Gender & Research no. 1 (2021): 60–73. (arXiv)

• Condee, William, and Barry Routree. “Nonmaterial Performance.” TDR: The Drama Review, 64 (2020): 147-57.

• Menon, Sachit, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin, “PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020), 2437–45. (arXiv, homepage)

• Karras, Tero, Samuli Laine, and Timo Aila, “A Style-Based Generator Architecture for Generative Adversarial Networks,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 4401-4410. (github, arXiv, ffhq-github, video)

• Rountree, Barry, and William Condee. “Imagining the Nonmaterial.” Imagined Theatres, 03 (2019): 14.

• Rountree, Barry, and Condee, William. “The Nonmaterial Mirror: Performing Vibrant Abstractions in AI Networks.” Theatre Journal, 73 (2021): 299–318.

• Salminen, Joni, Soon-gyo Jung, Shammur Chowdhury, and Bernard J. Jansen, “Analyzing Demographic Bias in Artificially Generated Facial Pictures,” in Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (CHI EA ’20) 2020, Association for Computing Machinery, New York City (2020), 1–8.

In this thread, I’d like to explicitly invite new programmers who would like to start working through the Exploratory Programming book to post their work along with any questions about it.

I understand that some people may not see this Working Group as the ideal context for this sort of sharing. The timing may not be right, or the way CCSWG is set up may not work well for this sort of exchange.

You should know, though, that while Exploratory Programming for the Arts and Humanities is indeed a “textbook” that can and is used in traditional classroom settings, I have written it, beginning with the first edition, to support learning in other ways. This includes short workshops, use by informal groups of learners, and use by individual learners.

So you are welcome to share some work and start one of these informal learning groups in the next two weeks. But whether or not you do, the book is there (in print and in its open access digital formats) to support your learning, when you’re ready.