Week 3: Code Critique - describeElement()

Comments

• Kir

  January 2022

  I think this is excellent work, y'all. I imagine that there's a way, akin to the Art + Feminism organization's global Wikipedia Edit-a-thons, for individuals to get engaged as a community around adding 'describe()' functions. I would be down to help.
• annatito

  February 2022 edited February 2022

  A couple of things

  could you put the chicken describeElement into the loop? if not this would be a barrier to me as a developer as I tend to do things in a more procedurally generated way and so the descriptions would to be able to handle that to a degree e.g. (forgive the rough sudo code I am thinking as I go)

  for (i=0; i< draggableObjects.length; i++){
      let randomDraggable = CreateRandomDraggable();
      describeElement(randomDraggable.name + "_" + i,"A " + randomDraggable.Description + " is ready to be dragged to the correct container.",LABEL);
  }
  

  In terms of localisation if we can use this system with a keyed string and file for the language ie. each file will be tagged by language and each one would have within it the same keys for the strings if you get the string RandomDraggableDescription from the file you might get something like:

  ENG: "A ${randomDraggable.LocDescript} is ready to be dragged to the correct container."
  
  DEU: "Eine ${randomDraggable.LocDescript} ist bereit, in den richtigen Container geschleppt zu werden."
  

  The problem you tend to hit with this approach is that in gendered languages the structure can change depending on the inserted item's gender so it gets extra complicated. I imagine that most developers even if they use the describeElement would struggle with this localisation layer as it would be an extra layer of obfuscation and to do things like the above you would need a binding system that could get the localised version of names etc e.g (again forgive the rough sudo code)

  for (i=0; i< draggableObjects.length; i++){
      let randomDraggable = CreateRandomDraggable();
      describeElement(randomDraggable.name + "_" + i, getLocalisedDescription("RandomDraggableDescription", randomDraggable),LABEL);
  }
  .
  .
  .
  function getLocalisedDescription(key, item)
  {
      let keyString = localisationManager.getString(key);
  
      const regex = /\$\{\w+\.\w+\}/g; //the binding key using the ${item.bind} format
      if(regex.test(keyString)) //checks if the string needs to bind data from the item
      {
          keyString = localisationManager.BindLocalisedData(keyString, item);
      }
      return keyString;
  }
  

  You could build in a localisation manager to p5.js but it would require a bunch of supporting stuff to make it work (like the binding above). I do a lot of this sort of stuff in games so I am pretty familiar with the process
• SarahCiston

  February 2022

  I love this code example because it really gets my head spinning about the purpose of language and the purpose of access. I'm sure many folks are theorizing this in several overlapping disciplines and I'm a n00b (how great that code can be an entry point to, say, disability studies for someone!). But what does it mean to "describe"? Depends who the audience is and their goals, and as an author of code or other language we won't always/ever know that in advance, we can only imagine and try to account for these possibilities. As a reader of poetry describe means something very different than as as user of accessibility tools (we all may be both at times, accidentally or intentionally). I'm so curious about the opportunities to use these affordances in their offshoots and overlaps.
• markcmarino

  February 2022

  Thank you @QianqianYe for leading us in with this thread.

  What I love about the Red Wheelbarrow use of describeElement is that it helps to identify a few assumptions that seem to circulate with that tag.

  In most uses of describe, the image is considered the primary object and the description is secondary, and often lesser version of the object. It is a passable substitute, but never an equal, certainly its superior.

  In the case of the Williams poem used to describe the drawn image, the description is arguably significantly more engaging and well rendered than the rudimentary chickens and wheelbarrow. Furthermore, while the drawing is rather plain, the poetry is artful and evocative. This example serves to demonstrate the assumptions of priority we bring (I bring) to elements such as describeElement designed to be accommodations with an underlying presumption that the substitute could not surpass the main form of display, the visual. Here I am also thinking of perfunctory use of the "alt" tag in HTML, even though there are readers (human and machine) for whom the "alt" is the primary form of access.

  Like a provocative title for an art piece, such as Duchamp's "fountain" or even the paratexts for a video game, the describeElement can create a productive space in tension with the visual. But considering this case, what would a world look like if the description were prioritized above the content it was replacing?

  The example asks us, how might we use a tool like describeElement to make something richer. And what if the audience for the description were seen as the priority rather than the ones being offered a poor substitute. If nothing else, the example demonstrates the inequities and hierachies that can attend encoded solutions to problems of access.
• hannahackermans

  February 2022

  Thank you for this code critique, @QianqianYe , it is such a great case!

  I agree with @markcmarino that the prominent position of the description is key to the work, both as an accessibility tool and a key element of the work. Often alt texts are invisible to users not using a screen reader*, which makes it easy for people to forget to add them or not know how to create good alt texts (using a browsing extension like Alt or Not on Twitter can help with that but it would be nicer if the platform provided this tool as a default). This is also why I like the idea of adding the describe() and describeElement() to the template as is suggested in the github issue. Perhaps it is an idea to fill it in with a default describe('no description provided yet' LABEL) so that the coder is reminded of needing to add the description at the end without receiving a message about it in console while they're still working on their canvas?

  As @SarahCiston points out, the usefulness of descriptions of images is often dependent on the target audience and the role of the image. As experts in multimedia we know that an image never speaks for itself. The same figure used in two separate contexts, for example, can require two different alt texts. And electronic literature and digital art in particular have visuals that are not only depicting something, but also aim to have a literary/artistic effect on the reader. This might make it initially more difficult to write useful descriptions, but I believe that the deliberate practice of writing alt texts also builds a broader digital literacy on visual content (same thing goes for writing good captions for audio content, especially if it's nonverbal sounds).

  *a bit off-topic, but the invisibility of for example default descriptions of gifs can also have disturbing effects, for example in this case where someone had put QAnon conspiracy theories in alt texts. So if you do not edit the alt text, there is not only a risk of sending something without or with lacking alt text, but also that you might be "passing along propaganda unknowingly".
• Mace.Ojala

  February 2022

  I wonder how this will work for a programmatic objects which might be appearing, vanishing and morphing thousands of times per second, and anyway are (often) nondeterministic, reactive and interactive. Laudable effort nevertheless.
• christina

  February 2022

  I'm admittedly a strictly a hobbyist when it comes to P5JS and I've never really shared my sketches except a quick link to friends or after turning them into gifs/static images, so this isn't something I've thought closely about because I've never really had an audience. I'm now really thinking about how I would describe different sketches I've made over the past couple of years.

  I feel like with the chicken example it's a pretty straightforward visual to describe, and I also agree that it adds to the experience overall. But I don't know how I would begin to distinguish some of my sketches so many of which are just "blobs but a little different than the last time" or " falling". I take a lot of enjoyment in making sketches that are different every single time, really make liberal use of that random() and noise(), but I don't know how I would write the descriptions so that it had a similar experience of seeing subtle-but-different changes each time. Especially because sometimes the differences can be really unexpected and hard to predict. Also what @Mace.Ojala pointed out with animated objects that can't be described at the same rate they animate.
• kjhollen

  February 2022

  Writing generative descriptions feels like an interesting challenge. I wonder how the documentation could encourage users to write poetic (but accurate) descriptions of a sketch given randomized values or parameters in the code itself.

  I've also wondered how to introduce the describe() and describeElement() functions in class when the students have usually moved on to moving images or interactive features by their second assignment. One thing that comes to mind is using p5.js's frameCount variable to ensure that the description isn't updated too often:

  if(frameCount % 60 === 0) { // once a second
    // update describe/describeElement
  }
  

  This still has the challenges that @christina describes: what if it's hard to describe the image because so much of it is driven by randomness? I'm wondering if some additional functions can help: for example, describeColor(), a convenience function for converting r, g, b or color object values into a description. I think something like this is already embedded in the code for textOutput() and gridOutput() but I'm not sure if it's easily accessible/documented elsewhere. I can imagine some creative uses for this like writing a function that describes the mood of the work based on the most frequently appearing hue. What other info/functions could the library provide to support generative descriptions? (I wish I had posted this sooner!)
• yaxu

  February 2022

  Sorry to come to this so late but this thread has been super inspiring for me. I've been working towards making alternative interfaces for creative coding and this has stopped me in my tracks. Making weirdo visual interfaces somehow feels wrong when it limits who can use it. Somehow I feel designing for disability first can end up with better interfaces for everyone.

  The example here is interesting in a different way though. Looking at the wikipedia page, The Red Wheelbarrow has an interesting backstory.

  "This poem was meant to be appreciated only by a chosen literary elite, only by those who were educated, those who had learned the back story (Williams was a doctor, and he wrote the poem one morning after having treated a child who was near death. The red wheelbarrow was her toy.)"

  So there's this idea that you have to be in the elite 'in crowd' to understand it. But then:

  "["The Red Wheelbarrow"] sprang from affection for an old Negro named Marshall. He had been a fisherman, caught porgies off Gloucester. He used to tell me how he had to work in the cold in freezing weather, standing ankle deep in cracked ice packing down the fish. He said he didn’t feel cold. He never felt cold in his life until just recently. I liked that man, and his son Milton almost as much. In his back yard I saw the red wheelbarrow surrounded by the white chickens. I suppose my affection for the old man somehow got into the writing."

  So the wheelbarrow belonged to a street vendor Thaddeus Lloyd Marshall Sr, who kept chickens in his backyard and was the inspiration for the poem. So perhaps now the poem turns from literary elites self-congratulating themselves for understanding a poem about infant mortality, to a white man seeing poetry in a black man's hardship.

  What does this mean for describeElement?

  I've also been reading Stiny's work on shape grammars. This is all about how we calculate with shapes in perception, and are able to see very different things in the same image, each time we look. It's then hard to see how it's even possible to describe a shape using words, when shapes and words are in such different domains. How do we carry that ambiguity of a shape across to its description?