LLM Writes The Chinese Room (Code Critique)

Title: "Chinese Room Conundrum"
Author/s: ChatGPT
Language/s: Inform 7
Year/s of development: 2023
Play the game here: https://markcmarino.com/inform/
Code by: ChatGPT-4
Adapted from this article.

Here was my prompt:

Compose a short 1-turn interactive story in the programming language Inform 7 that uses John Searle’s Chinese Room as a setting. The piece should have five different endings. The theme should be machine translation and black boxes, explainability, and AI. The tone should be satirical. Make sure the code will parse.

In the blink of an AI, ChatGPT (4) spat out the code for a complete little interactive fiction; however, the code was invalid in ways that would elude a novice Inform programmer like me. For example, the original code included instructions like, “Instead of talking to John Searle, say,” which should give a response, or should say whatever follows, if the player types the command “Talk to Searle.” However, since the code the LLM produced had not yet defined “talking to,” it would not compile. Inform has the commands “ask,” “tell,” but not “talk to.” Since I did not know that this code needed to be defined, I too was at a loss. The code was, for all intents and purposes, Chinese.

But with the help of a near native speaker (thanks, Ryan Veeder), I got the code up and running. Here’s the cleaned up version below.

"Chinese Room Conundrum" by ChatGPT

The Chinese Room is a room.

John Searle is a man in the Chinese Room. The description of John Searle is "John looks perplexed. He's surrounded by Chinese symbols and an instruction book titled 'Understanding Chinese'. He doesn't understand a word of it."

A man can be consulted or not consulted.
John Searle is not consulted.

A mysterious black box is in the Chinese Room. The description of the black box is "It's a black box. No one knows what's inside, but it's labeled 'Machine Translation 5000'."

Instead of examining the black box:
 say "The box hums lightly, almost as if it's mocking John's futile attempts to understand Chinese.";

Talking to is an action applying to one visible thing. Understand "talk to [someone]" or “converse with [someone]” as talking to.

Instead of talking to John Searle:
 say "John sighs, 'I'm just following instructions from this book. I don't understand Chinese, but the instructions help me respond correctly.'";
 now John Searle is consulted.

A button is a device in the Chinese Room. The printed name of the button is "a button labeled Translate". It is switched off. 

Instead of pushing the button:
    say "The room is filled with a mechanical humming. Moments later, a paper rolls out of the black box.";
    move the paper to the Chinese Room.

The paper is a thing. The paper can be read. The description of the paper is "It's a paper with some Chinese characters, below which are their English translations."

Reading is an action applying to one visible thing. Understand "read [something]" as examining;

Instead of examining the paper:
 if the paper is read:
  say "The translations are hilariously wrong. One sentence reads, 'Fish flies over the moon tonight.' Another says, 'John's teapot is the father of dragons.'";
 otherwise:
  say "You chuckle again at the absurd translations.";
  now the paper is read;

Every turn:
 if the turn count is 3:
  say "Suddenly, the black box starts to shake.";
  end the story saying "The box exploded. So much for 'advanced' technology!";
 if the turn count is 4:
  say "A voice emanates from the box: 'Error 404: Chinese not found.'";
  end the story saying "Looks like the box wasn't as smart as it thought.";
 if the paper is read:
  if the turn count is 5:
   say "John breaks into laughter, 'Even with this book, I'm sure I'd make more sense than that box!'";
   end the story saying "A human victory, for now.";
 if John Searle is consulted:
  if the turn count is 5:
   say "John ponders, 'If the box is just following its instructions and I'm just following mine, who's really understanding anything?'";
   end the story saying "Deep thoughts in the Chinese Room.";
 if the turn count is 6:
  say "The box suddenly goes silent. John shrugs, 'Guess it's just another black box mystery.'";
  end the story saying "Some things remain unexplained.";

After examining the Chinese Room:
 end the story saying "The satire is palpable. Machine translation, black boxes, explainability... AI sure has a long way to go.";

In John Searle’s famous thought experiment, The Chinese Room, he poses the following situation:

Imagine a native English speaker who knows no Chinese locked in a room full of boxes of Chinese symbols (a data base) together with a book of instructions for manipulating the symbols (the program). Imagine that people outside the room send in other Chinese symbols which, unknown to the person in the room, are questions in Chinese (the input). And imagine that by following the instructions in the program the man in the room is able to pass out Chinese symbols which are correct answers to the questions (the output). The program enables the person in the room to pass the Turing Test for understanding Chinese but he does not understand a word of Chinese.

In the land of LLMS, we’re all standing outside that Chinese Room, slipping in our little prompts and taking whatever comes out the other end. Except instead of producing perfect Chinese, the man in the box (or the LLM), sends out mostly valid Chinese content. The man is not answering questions so much as replying to the Chinese with amalgamations of Chinese characters that typically follow the ones we passed in. No one would say the man in the room understands Chinese. And, what’s more problematic, we, the ones passing notes in, may not understand Chinese either. Or, in the case of this example, computer source code.

When we ask LLMs to produce computer source code, we now have two Chinese Rooms. The one is the LLM that produces computer code when prompted. The other is the computer or compiler (if needed) that is processing the code produced by the LLM. The prompter, as it turns out, may be just as in the dark as the person in the room producing the code without comprehension. How could this go wrong?

It struck me that Searle’s thought experiment would make a fun interactive fiction piece and that it would be a further ironic twist if I asked an LLM to write the piece for me in a programming language that I have only a tenuous grip on: Inform 7.

If you haven’t encountered it, Inform 7 is (the 7th version of ) an English-like programming language created by Graham Nelson. The syntax reads more like English than perhaps any other programming language. It will take:

The Chinese Room is a room.

But it will not take

The Chinese Room is a big room.

That similarity to English makes it fairly easy to read, but deceptively easy to write. For no sooner does a novice programmer learn they can program in the vernacular that they try a familiar word only to get the reply, in the parlance of interactive fiction, “That is not a word the language can recognize.”

As my thoughts of this conundrum circled, I asked ChatGPT to write a work of interactive fiction about the Chinese Room, and to my surprise, it created Inform 7 code. Not valid Inform 7, but Inform 7.

Starter Questions:
What does this code generating experience tell us about naive (as in, from the stand point of not knowing the programming language) uses of ChatGPT or an other LLM to write code?
Are we in the Chinese Room situation or something else?
Whichever Room we are in, what are the implications for readers of code written via prompts.