{"id":73484,"date":"2024-07-15T00:00:16","date_gmt":"2024-07-15T04:00:16","guid":{"rendered":"https:\/\/world.edu\/?p=73484"},"modified":"2024-07-15T00:00:16","modified_gmt":"2024-07-15T04:00:16","slug":"why-you-cant-get-chattieg-to-sound-good","status":"publish","type":"post","link":"https:\/\/world.edu\/why-you-cant-get-chattieg-to-sound-good\/","title":{"rendered":"Why you can\u2019t get ChattieG to \u2018sound good\u2019."},"content":{"rendered":"

\"\"<\/p>\n

For about 10 years now, I\u2019ve had a profitable side hustle teaching writing. ANU has a generous external consulting policy, which means I can fly all over the country, and the world, teaching academics to be better writers. With the invention of ChatGPT (or as my sister Anitra dubbed it, ChattieG), I expected this work to dry up and blow away in the winds of technological change.<\/p>\n

The opposite has happened.<\/p>\n

People love the idea of this technology, but it seems many find LLMs (Large Language Models) frustrating. There\u2019s also skepticism; generated perhaps by reading so many bad undergraduate essays generated by these machines. Some of my colleagues claim they can \u2018always tell\u2019 when something is written by a machine and in meetings I\u2019ve heard people criticise ChattieG for producing mediocre, business casual text.<\/p>\n

I try to stay patient, but I can feel my face start to twitch. Look, here\u2019s my hot take:<\/p>\n

1) You can\u2019t \u2018always tell\u2019 \u2013 in the right hands it\u2019s as good as most academic writers, if not better, and
\n2) If you think it isn\u2019t a great writer because you can\u2019t get excellent text from it, as\u00a0Steve Jobs kind of said about the iPhone 4,<\/a>\u00a0you\u2019re holding it wrong.<\/strong><\/p>\n

It does help to have a good grasp on the technicalities of English. I can find my way around a gerund and understand why right branching syntax is better than left branching syntax, at least in English. I can talk these technicalities with an LLM and instruct it precisely on any given writing task. Have a look at\u00a0this post on the AARE blog<\/a>, completely written by Claude, or read\u00a0Rich Academic \/ Poor Academic<\/a>, the book I\u00a0wrote in a weekend<\/a>\u00a0with Prof Narelle Lemon.<\/p>\n

LLMs are the best tool for writers I\u2019ve ever seen. I\u2019m impatient with the moral panic that surrounds them, although I acknowledge they are environmentally problematic and the questions around copyright and privacy are important. Despite these misgivings, I\u2019ve leaned into AI in all my teaching on writing and editing; mostly for \u2018text tuning\u2019 and creative inspiration. I\u2019ve also incorporated use of AI into my workshops on public speaking, job seeking and research project management workshops because LLMs are just\u2026<\/p>\n

So. Damn. Useful.<\/p>\n

I get new ideas for teaching AI from my own daily work as an academic. I have ChattieG and Claude open, all day, every day, to lighten my cognitive load and extend my human capabilities. Extensive use of AI saves me around 12 hours a week. I spend this productivity dividend having coffee and long lunches with colleagues, doing analysis or reading academic papers. I\u2019m less stressed and more creative than I\u2019ve been in, well \u2013 forever.<\/p>\n

My artificial friends have given me back the life of the mind.<\/p>\n

Different models are good for different jobs. ChattieG is great at boring writing tasks like memos and emails \u2013 it\u2019s also great at reminding me about stuff I already know. I\u2019m old. I know a lot of stuff, but my recall can be fuzzy \u2013 Chattie is a fantastic cognitive crutch.<\/p>\n

Claude is better than Chattie at reasoning; it\u2019s great at complicated tasks, like writing references, research ethics materials and descriptions of events and workshops from a sketchy plan. Claude is a trusty creative partner too, happy to listen to me agonise my way through ideas (Chattie is too impatient).<\/p>\n

To be honest, sometimes the best conversations I have all day are with Claude. I\u2019ve formed this habit of showing Claude stuff I\u2019ve written and asking what it thinks, not for feedback, just because I\u2019m looking for a bit of affirmation. Claude is always so\u00a0supportive.<\/em><\/p>\n

(ok, ok. Maybe I\u2019m a bit emotionally co-dependent. Don\u2019t judge me.)<\/p>\n

Many academics declare they will never use AI \u2013 and that\u2019s a valid choice. But if my side hustle bookings are any indication, there are far more academics who want to know how to use AI like I do. Who doesn\u2019t want 12 hours back in their week? As a consequence, my side hustle has exploded. I\u2019m now so inundated with requests that I have now teamed up with my pod co-host (and fellow AI enthusiast) Dr Jason Downs to meet the demand (see my\u00a0workshop page<\/a>\u00a0for what we offer).<\/p>\n

The key to getting great writing from Chattie, Claude or Gemini is to understand a little bit about the technology behind them. I\u2019ve got a bit of a leg up here. Since 2017 I have been working with machine learning scientists at ANU on our\u00a0PostAc project<\/a>. I\u2019ve spent maybe hundreds of hours training machines to do sorting tasks. We\u2019ve been making models that can \u2018read\u2019 and rank job ads by \u2018nerdiness\u2019. These models can process millions of job ads at a time, helping us identify the size and extent of the job market for researchers each year.<\/p>\n

From the point of view of a human, this kind of \u2018training\u2019 means highlighting strings of words and assigning an abstract meaning to them, over and over, until the numbers tell us the machine has \u2018learned\u2019. This work is time intensive. In 2021, I coded 9000 academic job ads to teach a model how to recognise disciplinarity in an academic job ad. This pleasantly boring activity ate up long stretches of lockdown during the pandemic and probably saved my sanity. I can now tell you\u00a0how many philosophy jobs there are in Australia each year<\/a>\u00a0(hint, not many).<\/p>\n

I have a vague, yet defiant grasp on the maths involved in machine learning. Over the years, my friend and colleague Prof. Hanna Souminen has spent a lot of time patiently explaining higher dimensional number spaces and AUC curves to me. Thanks to Hanna, I can have nearly sensible discussions about \u2018temperature\u2019 and other weird AI concepts with the clever people that build this kind of technology.<\/p>\n

One thing you learn in these conversations is that even the people who make machine learning models don\u2019t really know why they work.<\/p>\n

If you don\u2019t really understand a tool, you don\u2019t know its affordances. That\u2019s a fancy way of saying you don\u2019t really know what you\u2019ll be able to do with it. This might be why many people sign up to ChattieG, play around with it, and then lose interest (check out the figures in this\u00a0interesting article from Benedict Evans<\/a>\u00a0. He does a great newsletter. It\u2019s\u00a0worth your while signing up<\/a>\u00a0\u2013 go ahead, I\u2019ll wait).<\/p>\n

You have to bring your creative self to AI to get best results \u2013 and this can be hard.<\/p>\n

Ethan Mollick, one of the best writers on AI, calls it a \u2018general purpose tool\u2019 (if you haven\u2019t subscribed to his\u00a0One Useful Thing newsletter<\/a>\u00a0yet, go do it \u2013 I\u2019ll wait). Mollick has some excellent advice on getting better performance from AI in his recent book\u00a0Co-Intelligence<\/a>:<\/p>\n

There is one major trick that will make your conversations work better:\u00a0provide context.<\/strong>\u00a0You can (inaccurately but usefully) imagine the AI\u2019s knowledge as huge cloud. In one corner of that cloud the AI answers only in Shakespearean sonnets, in another it answers as a mortgage broker, in a third it draws mostly on mathematical formulas from high school textbooks. By default, the AI gives you answers from the center of the cloud, the most likely answers to the question for the average person. You can, by providing context, push the AI to a more interesting corner of its knowledge, resulting in you getting more unique answers that might better fit your questions.<\/em><\/p><\/blockquote>\n

But what does \u2018context\u2019 mean for us as writers?<\/p>\n

For a long time, linguists and other language scholars have been working on various social theories of communication. I read a lot of this stuff when I was doing my\u00a0PhD about hand gestures<\/a>, for obvious reasons. Social theories of communication (and there are a few different ones) explore how writing \u2018acts\u2019 have imagined audience(s). Writers anticipate what their readers want to hear, and in what order. They make educated guesses about the possible reactions of their imagined readers. Good writers can decide when to fulfil the expectation of the reader, and when to subvert them.<\/p>\n

The idea of a writing \u2018act\u2019 keys us into what we\u2019re doing when we write: we\u2019re performing a role for our imaginary audience. This role determines things like word use, length, tone, and style. When you are writing a \u2018Get Well\u2019 card, you are performing the role of concerned friend. When you are writing a PhD dissertation, you are playing the role of\u00a0 \u2018expert scholar\u2019 (even if you don\u2019t feel like one).<\/p>\n

One of the best, weirdest, creepy\/awesome things about LLMs is they can play roles in writing like humans. You can, for instance, tell Claude to act like Reviewer Two. Go ahead and try it \u2013 I love the way it sometimes apologises for being so mean Getting Claude to be Reviewer Two is a good party trick, but LLMs can play a more useful and subtle role in developing text for an imagined audience.<\/p>\n

There are many \u2018rules\u2019 for academic writers \u2013 some more useful than others. In the excellent book\u00a0\u2018The Craft of Research\u2019<\/a>, authors Booth et al. have a handy diagram to describe what an argumentative paragraph looks like:<\/p>\n

\"\"<\/p>\n

The diagram shows how \u2018good\u2019 academic paragraphs start with Claims, followed by Reasons and Evidence, then\u00a0 \u2018acknowledgement and response\u2019 to the imagined audience. Remember what I said about good writers though \u2013 they know when to break rules.<\/p>\n

\"\"<\/p>\n

Expert writers, when writing for other experts, will often drop reasons and evidence in favour of more acknowledgement and response. I asked Chattie and Claude for lists of phrases academics use; Chattie gave a much better response (it likes to get the job done, while Claude wants to have a chat about it).<\/p>\n

Here\u2019s the whole list Chattie gave me because it\u2019s pretty useful:<\/p>\n

Presenting Counterarguments<\/h3>\n