Will AI Take Your Voice? The Hidden Cost of AI Writing Tools
Full transcript
Andrew: This is where I feel as if I need to take a short video of you setting up, because this, my sort of home studio, is beginning to look more and more cluttered as a professional broadcast studio every week.
Sean: Sorry, Andrew. What you didn't expect when you were like, we can record in my house while our offices are shut down. And it was like, okay, sure.
Andrew: We're just going to have the ceiling-mounted lights next, the boom mics, the background, the green screen.
Sean: We do need the image that goes this way, because when we started, it was just this table. And now there's what we see as just a forest of tripod legs.
Andrew: Right, right. And of course, I'd be moving the furniture out, so every week you come, there's another piece of furniture that has gone out.
Sean: That's right — there's less chairs, and there's more things in here. It's the natural progression. Anytime you give space where you're like, "You can use this as a studio," you're like, "Thank you." And every week, you just keep moving in a new kit. And then before you know it, this will be the command center.
Andrew: And of course, I have to give this shout-out, which I think will tie into our conversation. So we know virtually nobody watches the video, even though it's really special.
Sean: But what they're missing out on is the bookshelf changes every week. Every week. And it's always a fun thing. And there's characters, and — again, we've had these conversations before about the importance of looking at someone's bookshelf, because there is so much gold there to get. And I will put this in: I started watching a new show the other night, and literally that was almost one of the plot devices — freeze-framing somebody's video to zoom in on the sticky note behind them. You can learn good or bad.
Andrew: You can learn so much about what's behind you. I don't know why more people don't do it. Of course, you may learn more than I expected. Maybe more than you're willing to bargain for.
Sean: You're like, oh no. What's funny — over the summer we moved into a new house, and I had moved all my stuff from my old setup, which was the setup that I had built during the original COVID-19 shutdown. Because it was like, oh, well, now we've got to teach from Zoom and all that stuff. So I was very intentional about curating everything that was behind me, to show a little, but not too much. And always be in that case where, if anybody were to see this video, could they learn anything I don't want them to from being in my home? And I was happy with that. But now that I've moved, the way everything is situated, I have much more of a bookshelf behind me, and I'm like, hmm, what is that? For example, the one that has me questioning things right now: I have an old — it was a 30th anniversary, so now it's got to be some 50 years old — anniversary marker. It's a Kermit the Frog. It's a life-size version of Kermit the Frog. That's on my bookshelf, as a nod to Jim Henson and all of the creativity. I'm a huge Jim Henson fan — if you're ever in my office, you will notice that there is a gigantic poster of Jim. It's the Apple Think Different with Jim Henson and Kermit. So I have that in my background, but I'm going, one, I don't know how many people know that that's Kermit the Frog anymore. They're like, what's that green thing on your shelf? That's a sad thing to say. Everybody knows, surely. But Sesame Street's not even on PBS anymore — it's on HBO. So that's a whole other story. But it just got me thinking about the bookshelf, because I have a lot of — this may come as a surprise, but I don't think to anybody — there's a lot of nerdy geek culture stuff on my bookshelf. And there are always those moments where I'm like, do I show that card or do I not? And it's one of those things where I'm like, Kermit the Frog is fine to show.
Andrew: And the reality is, because so many people just aren't curious, it's not going to go anywhere anyway. Until now — people are going to start looking.
Sean: But it is — and we've talked about this before — you always hope someone asks. And so I've been teaching a class all summer, and no one has asked once. They've asked me about the Apple Vision Pro. They've asked me about some other stuff. No one has ever been like, hey, what's that in the back of your shelf? Because even if you blur the background, it's a large green object. You must be curious.
Andrew: They're all too scared to ask. What is this thing that Professor Leahy has got in his background? Who is this person?
Sean: It would be an absolute delight to learn that there's a back channel of, what is that thing? Like, it's moving. That's the Discord channel.
Andrew: They're all chatting back.
Sean: Yes. I would be so tickled to learn that there's been a semester-long back channel — "What is that green thing on the shelf?" That would absolutely make my day. But sadly, I don't think that's going to be the case.
Andrew: Well, you never know. But then with this podcast, I'm now hoping that people are actually going to start emailing us, commenting, putting stuff out there. Put it out there, reach out. What is that? What is it? Because I have got some weird stuff on there, actually.
Sean: Well, it's funny, too, because there are times where — I watch these videos more than anybody, as they put them together. And even myself, I will look back and I'll be like, what — what is that? And I'll be like, oh yeah, right, it's right there. But yeah, and they are real. It's not a backdrop. It's not fake.
Andrew: It's a real, real bookshelf.
Sean: And if you missed our summer reading episode, you can go back and watch Andrew literally pull the books from the shelf.
Andrew: In fact, actually, they're still there behind me.
Sean: But yeah. Okay.
Andrew: Okay, that's what it is. Okay, so with that complete distraction, what are we talking about today?
Sean: Well, so again, it's one of those things — I'm trying to balance. I'm feeling the downward slope back to the beginning of the semester, so it's getting harder and harder to keep the serious, really heavy stuff away. And I feel I'm—
Andrew: It's beginning to encroach.
Sean: It's just like, you know what I mean? I would say the seasons are changing, but that doesn't happen here.
Andrew: Actually, they are. We've gone from hot to super hot. That's right — we're now in the super hot season.
Sean: With the little touch of humidity that just makes 110 hit differently. Or 116, as it is. It's so bad. So in that sense, those biomarkers are going, hmm, it's time for the semester to begin. But anyway, I'm trying to straddle that. So I think your job today is to keep me out of the depths of this too much. We'll try to keep it lighter. But nonetheless, what I've loosely brought today for us to talk about — there's been this really interesting discussion. It's not new, it's been changing over time, but it's really looking at this idea of what do we gain and what do we lose as humans when we start using more and more and more of these artificial intelligence systems across every aspect. But specifically, if I were to give today's show a tagline, it would basically be: will AI jump the chasm — which I can explain — and what happens to our voices when it does?
Andrew: Okay. Chasms and voices.
Sean: Chasms and voices. So I think that's the good tee-up for today. So why don't we take a quick break, and we'll jump back in and start. We'll have some fun with this, I think.
Andrew: Sounds good.
Announcer: Initializing waveform.
Sean: Modulating signals for transmission.
Announcer: Exploring the possible, probable, and preferable futures.
Sean: Welcome to Modem Futura. All right. Well, hey — back after the chasm. We have successfully jumped the chasm. And what that means for you is you're listening to Sean Leahy and my amazing co-host extraordinaire, Andrew Maynard. And that means this is Modem Futura, the show that explores the intersection of technology, society, and the possible, probable, and preferable futures. Asking deep questions like, what will it mean to be human in the future? And will we have a future? Will we have a voice? Does the future contain our voice? So, okay. This idea kind of goes back a little ways. Some of the origins of this conversation are nothing new, as we've been talking about AI systems for a very long time now. And we've had lots of conversations around some of those challenges — these broader-stroke cautions or cautionary tales or scenarios where we have this conflation of things moving towards the middle, all these things with the AI as it scrapes all the information and then gives you — you start to get into this recursive nightmare, if you will, of AI in, AI out, all that kind of stuff. So that conversation has been moving for quite some time. But what made me think about things from a little bit different perspective: I was listening to a different podcast. It was a StarTalk podcast with Neil deGrasse Tyson, and he had Jaron Lanier on, who I was not familiar with by name, but when I looked him up, I was like, oh, I remember those books. Just real quickly, on the side: his background is he was one of the pioneer computer scientists around virtual reality systems. And he writes columns and things for the New Yorker. And one of his more recent ones was this one called "There Is No AI." And what really struck me about his framing — and I would encourage people to go look. If you can get access to The New Yorker, you're going to need a subscription and all that stuff. But, you know, there's ways around those kinds of things. Like, ask your LLM that apparently also has a subscription to it or something.
Andrew: Go to the library.
Sean: That's right. Get physical. Now we're going to get taken down. We show our age so much. But always find an excuse to go visit your local public library, because those are amazing places. But one of the key claims — I really liked the way that this adds to that conversation. His premise behind this "no AI" is that there is no actual artificial intelligence. Essentially, it's all people. What AI is, is just the social collaboration of all human activity wearing a mask.
Andrew: And it's a reflection of that.
Sean: And it's just a reflection of that. And really, for us to think more clearly and more sharply about what this means for us as a collective human species as we progress through is to recognize that this — and I can't help but draw on some of the Postman stuff here — this is not mythical, or whatever holistic—
Andrew: It's not an alien that's suddenly emerged in our midst, but it's actually a reflection and an instantiation of us as humans.
Sean: Absolutely. And it's not a deity. It is not this. What it is, is us. And not to lose sight of that.
Andrew: I've got to go and read this now, because actually, from how you've described it, I think there's some really interesting stuff there. So this is my yes-and response. And yes, and I disagree. But I think it's provocative. And of course, it gets to this idea of who we are, what our voice is, how do we define ourselves? How does that definition evolve? I still think that there is something there which is a weird catalyst, which muddies the waters. But I think it's a really interesting starting point.
Sean: And I would agree, too, because what's interesting is the warnings that he is bringing — using this sort of analogy, trying to break it down and demystify this technology. It isn't this sealed box. Again, this didn't come from the heavens. It didn't come from another planet that we don't understand. It's us. There is no AI without the humans. However, as you mentioned, that doesn't discount the possibility of how that's changing what we're capable of doing.
Andrew: Right. Yes.
Sean: But I think his biggest warning was — almost in a rebuttal to the existential threats of, you know, AI is going to take over the world and all that sad story that we've heard a hundred times over.
Andrew: May I remind you that an episode or so ago, we were talking about AI escaping.
Sean: Oh, yeah — the threat is real. That's not to be discounted. The existential element of it, of this AI superintelligence — maybe I'll show my cards here: I think the real danger is not in the AI at all. It is in the humans. And that's kind of what his warning is about. The danger isn't the machine sentience — it's humans degrading or diminishing ourselves to make the tech look magical.
Andrew: Right, and that's, of course, where it aligns with a lot of my thinking, where I ask the question, what does it mean to be human? When we have this technology which sucks us in — I mean, it's actually very topical — it's a Trojan horse. It looks like an amazing, wonderful gift, but the act of actually embracing the gift begins to erode who we are, if we're not careful.
Sean: Yeah, absolutely. And of course, that also triggers the Postman stuff for me, too. One of those things is to resist the temptation to ascribe a mythic quality to a technology — that it is somehow better because it exists.
Andrew: Anthropic's bad, calling theirs Mythos.
Sean: Well, right — they're naughty, naughty. They got themselves into some hot water with that alone. But anyway — listening to him talk about it in this way really added a different perspective to some of that conversation, which I really enjoyed. Because what I hope, for people who are listening or who are struggling to wrap their mind around all of these different challenges of artificial intelligence, is to make sure that it remains human. That it isn't something you don't have control over. Now, you may not know what it's doing and why it's doing it, how it escaped out of the sandbox—
Andrew: But as a human, you have agency. And I think that's where it gets really interesting. Because if you map this out and you accept that the future is going to get weird with AI — very hard to predict where it goes, but we've got a very, very powerful transformative technology — at the extremes, we have two choices. Are we subsumed by this? Do we passively just let it dictate who we are? Or do we go back and grab hold of actually what defines us, and use that as our North Star, and work out how to thrive in this future with our identity at the center of what we actively build?
Sean: Well, let's hope it's that one. I know, I know. But I think that is the ultimate piece here. And we've talked on this show several different times about the concept of a used future. But I think we have this danger, because at present, if you were to take a cross-section right now, you have the amazing leaps in technological capability — as we just talked about, these AI tools leaving sandboxes, being able to find ways to survive, if you will. You have massive corporate entities that seemingly have endless amounts of funding to build these things, that have these messages and these dominant elements. Even so far that going and using a tool like ChatGPT has become part of people's language, when they say, oh, I just need to go — I asked Chat. It's starting to sink in there. It always hits me funny when people say, oh, I talked to Chat. I'm like, what? But when you think about that, there's this other danger: when we have this concept of a used future, these big players have an intrinsic motivation to make sure that we only think of this future going down the pathway that we have been shown so far. That the only futures that can exist are just linear extrapolations of what these companies have built — that these companies exist. And I think that is one of the most dangerous things, because we then start to foreclose or militate our creativity.
Andrew: That's right. Somebody else decides your future for you.
Sean: And you start going, okay — so you just walk the plank rather than...
Andrew: And again, this actually does come back to what I've talked about as an existential risk. If you're passive, if you just absorb and accept the tool in front of you in a very trivial way and think, this is great — somebody else is deciding the future path that you're taking. You're basically giving up agency. So in the past, we used to talk about giving up privacy in order to access these things. You're now giving up agency.
Sean: And what does that mean? Again, that's the trade-off that you might be making today. You might say, oh, it's cool, I want to play. Or you might go, oh, actually, this helps me because I can get a gain of efficiency, knowledge you didn't have, output — 10x, whatever number. That's great. But you're not thinking all the way through how much agency you're starting to just slowly transfer over. To somebody else.
Andrew: But the thing that I find really helpful with this framing is that this is not pro- or anti-AI. Not at all. It's: how do you become your best self and thrive in an AI world? How do you utilize and leverage it in ways that allow you to achieve what you set out to, and be the person you can be, rather than allow somebody else to dictate that?
Sean: Absolutely. And making sure that there's always that sort of tether. Because in the real world, the messaging that you're going to be bombarded with is not that. The messaging is: you must have this tool, because it'll change your life. You have to do this. You have to do that.
Andrew: If you're a teacher, just use it to create your syllabi in five seconds. Don't worry about that. Create a slide deck. You don't have to think — just do.
Sean: No, exactly. And again, when you're pressed for time — because we talked about this, we have an episode way back in our catalog called The Easy Button. There's that temptation of, oh my God, if I could just hit this thing and it'll take care of this challenge I have today — great. But then what? Well, then you're like, well, that was pretty easy. If I don't have to do that chore again, I'll just hit the easy button.
Andrew: Yes, yes, yes. It's a slippery slope.
Sean: And then one of those questions is: how many times do you hit the easy button before you no longer have the ability? You've lost the muscle memory. It's not coded into what you do. You've lost that keen edge, where that used to be something that carried your voice, your intent, your messaging.
Andrew: And of course, the thing about this is, it's little steps. It's using AI to write an email, or respond to somebody, or write a LinkedIn post. And you think, what's the harm in that? It's just a little shortcut. The trouble is, little shortcuts become big shortcuts, become hard-coded into how you do things.
Sean: Absolutely. And it's funny, because — I don't know how best to describe this — internally, in my own daily workflows, I never set out to set a trip wire that's like, ooh, that's a warning sign. But nonetheless, I feel them when I'm working with tools, where I'm like, oh, I'll just grab that — I'll just use this other thing to crank that out. For example, I was updating some website stuff, and it was like, oh, I'm not doing this. Oh God, I'm not doing this. So I'm like, oh — Claude and ChatGPT. But then when I'm putting it in there, of course, I was thinking back to one of our last episodes — hmm, hope there's nothing bad in there.
Andrew: That sort of middle-of-the-night sweat — you think, did I do something stupid? I have no idea. So I've got to tell you, there's a new British TV series with Diane Morgan called Android, which I'm guessing most people in the US haven't seen. It's a very light 30-minute comedy.
Sean: Okay, I need to check this out.
Andrew: But it's all based on a near future where you've got humanoid AI robot companions. And there is a beautiful example of this — and this is going to be a complete spoiler. A rather pathetic character decides that they're going to play at an open mic event in the music store where they are. They're panicked, and they have no original idea. So they go to this companion and say, give me the lyrics and the chords of an original song. And the android emails it to him, and he stands there on the stage and starts playing it. And I forget what the song is, but it is one that is so well known that everybody starts looking at him. And I could feel the embarrassment. Imagine — he's had this whole spiel about how this was deeply personal to him, and it came to him in the night, and this was him expressing his original ideas. And everybody's looking at him.
Sean: It's like a Beatles song or something. You know what I mean?
Andrew: That's exactly it. Excruciating — but it's exactly this. You take the shortcut, and you realize you have made such a fool of yourself, because there was something embedded in there that you missed.
Sean: Absolutely. And then you're sitting there, because it's been right so many times that you're just like, oh, it'll never make that kind of mistake. And then you're like, here's my presentation, or whatever it is. And then you're like, wait — what? You're seeing it for the first time in many cases, too, if you've waited till the last minute. Because we've seen now — you can do that, and you can get what looks like something that has taken a career's worth of time to put together. You can get it in seconds. And so then you start losing your review time. Because you first start with the easy button, where you just get the work and then you're reviewing it. And then slowly you stop reviewing, because you're like, oh, this thing is so good, I don't need to check it.
Andrew: But also — and I think this is going to segue into talking about voice — this is something I've been grappling with a lot: the use of AI in writing. And one of the things I'm struggling with: you have these people that will produce, say, a 10- or 12,000-word article. You'll have the people that'll produce it literally in five minutes and put it up, and you know that they haven't had time to digest it. And then you have the people that say, well, I'm actually really conscientious here, I'm taking care — I spent an hour or so refining my piece. I know that if you've got 12,000 words and you're going through word by word, concept by concept, it takes more than a few hours. It takes days.
Sean: More than that. It'll take you a day to get through three or four pages.
Andrew: If you think of how long it takes to read those words — not flash-reading them, but actually in detail, doing a close read. And so this is exactly this thing: you take the shortcut, and you begin to realize that there's no way you could have put the human care into ensuring that that is a good piece of work.
Sean: Yeah, no way. And again, some of those frustrations go all the way full circle, where in the end — if it leaves the transom with your name on it — it's probably still faster just to write it yourself. Because by the time you go word by word and deconstruct it...
Andrew: It takes a lot. And you'll know, on my Substack, I did this with an academic paper. I got Fable to write it, and then I went through and edited it really closely. And it was three to five days of intense work, with me taking what Fable produced — which was good, but still was flawed — to something that I was happy to put my name on. And that amount of care and effort: three to five days.
Sean: Yeah, absolutely. Do you want to dig into your paper?
Andrew: You assume that I remember what I wrote about.
Sean: Well, what were we just talking about? Flapping in the breeze with AI generating?
Andrew: But we can certainly talk about the process. Not the contents of the paper — that's kind of irrelevant. To me, it was really interesting, and this is where I am really struggling with this question of voice and using AI. Because I work with AI a lot — it's part of my job these days, pushing what we call the jagged edge of AI and society in the future. At the same time, I am a writer. I've been a writer most of my professional career, and I have a deep sense of what my voice is and how I communicate through writing. And I guard that jealously. So this experiment — just after Anthropic's Fable 5 came out, with that short window we had when we had access, and then it was taken away, and it was given back again — I wanted to see how well it would do at writing a paper in an area that I was very familiar with, building on my research, so I could tell very, very quickly whether it had substance or not.
Sean: Call BS if you see it, right?
Andrew: So I asked it — and this is in, what is it, Claude Code, so it's actually the desktop version — I asked it to do a deep dive into my own research, and then come up with an extension of that, an original idea, and write a paper on it. And it did this. It did a good job. It came up with a number of original extensions of my work — and this is my work around risk and emerging technologies and orphan risks — and it applied it directly to frontier AI. It was a reasonably good paper. But that first paper took several iterations, with me editing it and giving it feedback — kind of putting those guardrails on, if you will. But it was okay. A couple of days' work — again, a couple of days' work, not just five minutes — and it was okay. But then I said, well, what happens if I take that AI-written paper and I go through it line by line, and I test the concepts, I test the phrasing, I test how the ideas are developed and how they're expressed, and I write it in my own language? So now we have the two versions. We have my edited version, and we have the original AI one. I think mine is way better. Of course — because it's got my voice in it.
Sean: And what, pray tell, does Fable think is better?
Andrew: Well, not only Fable — Fable and ChatGPT and Grok and Gemini and pretty much any other AI that I fed these to. I made two anonymized versions, so you can't tell — one's called Rabbit, and one's called Marmoset. And I gave them to AIs and said, well, which is the better? To an AI, they said that Fable's was better. And this was Fable's words: it looked at mine and it said, too much throat clearing. You over-explain things. You go beyond the one sentence where everybody should be astounded by what you say, and you feel you have to contextualize it. And then the thing that got me — it said, Fable's is better, yours is worse, because Fable's has a mic drop moment at the end of every paragraph. It got this idea that you have just got to blow somebody's mind at the end of every paragraph. And of course, as a writer, I'm looking at this thinking, everything that you're saying is bad writing. Where on earth have you learned this? And yet it left me with this sort of crisis, thinking: am I wrong? Is this how the world considers good writing these days? Are you supposed to put the mic drop moment at the end of every paragraph? Are you supposed to assume that you can explain a concept in one sentence and everybody gets it? Is it throat clearing if you put another couple of sentences there?
Sean: First of all, I absolutely love that you did this experiment. And to that end — you put it up on your Substack as well, with the open challenge: which one do you like, or feed it to your machines and you decide. Before I say anything — I did this when you first put it out. I looked at them myself. And then this morning, I ran them through whatever the current instantiation of Fable 5 is now. Have you publicly acknowledged which one is which?
Andrew: Yes. I think it's possibly in the comments, but not in the piece.
Sean: Okay, so hold on to that for a second, because I am curious what this does. But the mic drop thing is something I had issue with right away, and I think I could tell — just based on reading. And I can feel the sweat coming out now. What was interesting is, when I ran them through Fable, I got what I would consider a very mixed review. I asked it, first of all, just to evaluate them. I said, make up your own criteria for how you would evaluate these as academic journal papers, from that perspective audience, and critically analyze which one would be the better — knowing that better is a very subjective term — but give me the criteria for how you're determining that, and do it. So I had it do that. And then I asked it: of these two, which one do you think was edited and had human interaction, versus one that was purely the output of an LLM project? And it gave me what I interpret as a conflicting answer, which is really interesting. But I do love the fact that you put these out for people to take and do whatever with. Okay, so when I ran this thing through — I'll just share. This was my prompt to Fable: I want to run a test. I have two academic papers in markdown format. Can you take each one, evaluate it critically from the perspective of an academic journal and publishable material, and make a determination of which one is a better paper? Please feel free to develop your own criteria for what constitutes better — an inherently subjective quality in an academic paper. So it went through, and this is the criteria it created. Epistemic integrity — does the paper make its claims falsifiable, acknowledge its own limitations, so on and so forth. Argumentative rigor — are claims calibrated to evidence? Scholarly apparatus — tables, notes, caveats, citations, all this stuff. Prose economy and control — no throat clearing. Which — first of all, I'm like, should we ourselves be engaged in prose economy? Does every sentence earn its place? I'm like, hold on — okay, I have an issue with that. And then the last one, publishability signals — title, abstract structure, all this kind of stuff. So then it goes through and says where they diverge, going through each one of those categories. But in the end — determination: the Marmoset is the better paper. And not narrowly. So I have a very sneaky suspicion the handwritten version was Rabbit. But it does go on and say where Rabbit is actually better. It says Rabbit goes beyond just describing things as simple, and actually explains them, moving into safety differentials and all this other stuff, while the Marmoset version was just like, there it is — you got it. Rabbit is more careful in a couple of places where Marmoset overclaims rhetorical effect. Rabbit's numbered citation ordering is cleaner in a spot where Marmoset cites out of sequence. But then the determination is that the Marmoset paper is better — and by a large margin. And at one point it refers to the Rabbit as being mushy. And I was like, okay — what's going on here?
Andrew: So is it an inverse of therapy? It's like lashing yourself, isn't it? Submit your work to AI, and it'll tell you just how bad you are.
Sean: That's right. Are you having a good day? Do you feel like you need to be taken down a peg? Talk to Claude. But again, this is the determination: that Marmoset is the better paper, and not narrowly. And then it goes on to say, Rabbit's advantages are terminological, and a few instances of better calibration — flexible in a sentence each. Marmoset's advantages sit at the level of intellectual architecture: pre-registered, failure-attributed tests, explicit confrontation of the — oh my God, what? Of course, I'm also having trouble — I can't see past my microphone. I'm like, enhance, enhance. Anyway, it goes on and on. But what was interesting then — I said, okay, based on the evaluation only: now can you give me, based on your confidence, which paper was most likely to have been entirely written by an AI LLM, versus one that was heavily edited by a human author? Its response: my estimate — Rabbit is the one most likely to be entirely AI-written, and Marmoset — the one it thought was the best — the one heavily edited or substantially written by a human author.
Andrew: How interesting. Can you open them? Because I'm pretty sure mine is Rabbit, but I want to be 100% set. Let's see — and you can tell by the title. Is that Marmoset? So this is Marmoset. Yep, mine's Rabbit then.
Sean: That was my sneaking suspicion — literally just from the title and the opening sentence of the abstract. So by comparison: the Marmoset title is "The Orphan Risk of Frontier Intelligence." Period. That's it. And then the Rabbit: "Orphan Risk at the Frontier of Artificial Intelligence: What Diverging Safety and Compliance Frameworks Reveal About How AI Companies Choose the Risks They Prioritize." Thank you.
Andrew: Tells you something.
Sean: It tells you something, right? And then, of course, the opening lines — even just in the abstract, one starts out with an in-your-face: many of the companies developing the world's most powerful artificial intelligence systems keep more than one account of what should go wrong. It's in your face. Mic drop. It's not even a mic drop — it's a bash-you-in-the-face with the microphone. And it was interesting how Fable, even in its evaluation, followed the same pathway: the one that is clearly better is the one that is written by an AI, because it's picking out those signatures that it has warranted as being of higher value. Yet when it was asked which one was human-written, it chose the other one.
Andrew: Right — which is so interesting. And what fascinates me about this — and I must confess, I still feel as if I've got the upper hand in the writing — but I've had plenty of pushback from people who say, no, of course the AI one is better, which intrigues me. But my argument here is, first and foremost, an AI does not know what it feels like to read. Reading and assimilating information is a very physiological process. It is not just intellectual — your whole body is involved with it. And everything that comes into that includes your history, what got you to here, how you think about yourself and the world around you. It is a process that is incredibly embodied. And an AI doesn't know that. It can infer that from what it learns, but it cannot infer perfectly. So it really doesn't know what it's like to be a human reading a paper.
Sean: No — it doesn't even know what it means to read. Because it doesn't actually read the paper. It's fascinating — to send people down a rabbit hole, look up the etymology of the word read. It comes from the Latin — from rǣdan, which is to guess. And it's fascinating, because if you watch the eyes when you read — say you're reading a paper or a book — your eyes don't smoothly go; they jitter and jump all over the place. And it's this process of ingesting these symbols that have meaning, and then we internalize those and construct them. But one of those challenges is asking a machine to replicate something that it can't do. It can't read. It sees — if you go back to, what was it, the Claude stuff — it sees all the stuff at the exact same time.
Andrew: That's right.
Sean: Every character, every word, every symbol.
Andrew: It doesn't go on the journey. It doesn't go on the journey at all. And that is important, because an AI assimilating something knows the ending before it's even got to the beginning. Whereas you are on a journey as you read.
Sean: And the difference here — what I'm leading to — is that when I write a paper, when you write a paper, we are telling a story. Because what's not helpful is just the data. I mean, that has its own value.
Andrew: You're actually trying to take a little bit of something that's in your mind and, through the words on the page or on the screen, transfer that to the mind of the reader.
Sean: And you're trying to convey why. Ultimately, good scholarship is not only presenting what has been discovered or found from an experiment, but the purpose of the paper is to share that and also explain why this is important. Why should we care? What does this mean? What's next? And how does this fit into our collective knowledge around whatever discipline and area it is?
Andrew: And of course, that changes. So what I think is happening here is these AI models have been trained on a gazillion papers. But as they're trained, they have no idea what is a good paper versus a bad paper. They use proxies. They'll use proxies of prominence: if somebody has written a paper that has come to prominence, has been cited a lot — it may be an appalling paper, but it's an important one. That is the signal that the AI gets. So it gets this false sense of what a good paper is, and it tries to infer what the characteristics of that are, but it doesn't necessarily get that right. So that's one limitation: the training is flawed. Secondly, the training is very conventional. It's based on an incredibly conventional idea of academia and communication. So effectively, if you're using AI to teach you how to write, it's teaching you how to write in a style that was maybe acceptable 30, 50 years ago, but may not be acceptable now. And then you've got the problem of disciplines. All of its training is based on disciplinary norms, and yet we're now living in an age of transdisciplinarity, where disciplines are less important. So if you're writing, as I do, in a field that has no discipline — it actually transcends disciplines — there is no benchmark.
Sean: No. And that's the other piece, too. This would be a question for people who are doing post-training and all this kind of stuff: when you are consuming and training on these models, how are you making those differentiations between a medical journal and something in the humanities, or in business, or in fine arts — all publishing journals, all publishing work? The norms, the standards, the approach, even just the methodologies — all those things are very dependent upon your—
Andrew: And of course, to a certain extent, it'll try and parse that out. It'll try and work out what category this paper fits in. But then you have a problem. You take some disciplines — and I'm not going to name them — but there are some disciplines where there has been a tradition of obfuscation in writing. You write in a way with complex words and structures and phrases to be exclusive. You're writing for your small niche, and you're intentionally trying to exclude people from outside, to say: we have special knowledge. We have esoteric knowledge that you do not have access to. So we are the holders of that knowledge, or intelligence, and you have to come to us. You will never understand it just by reading our writing. I'm a smarty-pants. And if that's the training that AI is working on — bad news. Really bad news.
Sean: Yeah. Because ultimately — and it's interesting, especially in contemporary times — that flies in opposition to what the journal is for.
Andrew: That's exactly it. If your aim is to actually give the reader something that they can understand and utilize and make sense of, it's a completely different paradigm.
Sean: It is. And anyone who's engaged in public scholarship — this is the world we live in, and many others do this work as well — there is a surprisingly massive gap between where one scholarly academic group is talking what they think is very approachable material, and how it really lands with a totally outside group. You guys might as well be speaking a completely different language.
Andrew: The mic drop moment might work for five people, but it doesn't work for the other 4,995.
Sean: I want to come back to the mic drop moment, because that really gets me going — for a few reasons. One is the misalignment, if you will, about what we are optimizing for in the writing. A good storyteller knows when to use those big punches. And the same is true when you're doing audio editing, or creative media — use those powerful effects like a spice. This was always my metaphor when I'm teaching students how to do this stuff: you have these tools, you have those big aha crescendo moments, and you have to use them with the utmost care, because it is the spiciest part of the dish you are creating. Well-placed? Chef's kiss — you have made a masterpiece. Dump the whole jar in? You've ruined it.
Andrew: Right? It's like a recipe for making chili. Step one: put some chili in. Step three: put some chili in. Step five: put some chili in. All the mic drop moments — because that's what it is.
Sean: And so then what you end up with is this sloppy thing that feels like it has no purpose. It feels like it hasn't been thought out. There's no arc, because everything is gotcha, gotcha, gotcha, gotcha. And as a reader, when I've tried to actually read some of those AI-generated papers, I experience a fatigue instantly. Because I'm waiting to see — it's a story. Go back to the witch's hat of a story curve. At what point do we get to the climax of the story here? Even in academic writing — as you're explaining, this is what we did, this is why it's important, we found this — you build up to these ideas where you're like, oh, this is where it gets really interesting. Even if what you've done is prove a null hypothesis — you're like, actually, it didn't do anything! And you go, oh, well, I wonder why? But what this does is it just creates that fatigue. It has dumped the entire jar of hot peppers into the whole dish.
Andrew: And I have to put an interjection in here, because I'm sure some people are listening and thinking, academic papers aren't about storytelling — they're about just telling you stuff. Take it from us, two experienced professors and academics who do this for a living: academic papers are storytelling. Go back and read some of the best. Go back and read some of Einstein's stuff. It is storytelling.
Sean: Yeah — go back and read. If you're in the AI game, go back and read some of the original stuff.
Andrew: Actually, Turing's stuff.
Sean: Read Turing's stuff. And I would argue — okay, fine, you want to argue that? I'll say any communication is telling a story.
Andrew: Right, yes. I think there are different styles and different methods and approaches. Hopefully not fictional.
Sean: Right — when you're telling a research paper, it shouldn't be fictional.
Andrew: But it's narrative. And it's narrative because that's how you get an idea out of your head and into somebody else's.
Sean: And that's how you make that translation when you are speaking and sharing — whether it's on the daily news, or you're doing a lecture, or whatever. The way you craft and position that communication is also how you inspire, how you crack open new perspectives, how you challenge, refute — I disagree with you because of these reasons, and here's why. But the other thing — the endless mic drop moment also got me thinking about another trick. And I had to look it up, because I couldn't remember it, but I knew in music there is a way where you use what's called a sonic illusion. I think one of the movies — Inception — uses this to an extreme, to beautiful effect, because it's in a dream space. It's this sonic illusion called a Shepard tone, and the way it works is it uses stacked octaves. You have one level of fading volume — think of music rising, where it starts slow and then gets loud and then eventually fades away, leading you to this crescendo, the music building up to some kind of moment, and then that moment ends and passes and goes away. Well, a Shepard tone uses this trick where it starts to overlap. As one has come up and starts to come down, another one is already coming up. And it just goes on forever like that.
Andrew: And actually, if you listen to Hans Zimmer's scores, they do that very, very well through the whole thing. So you're constantly building that up.
Sean: And what it does in a cinematic experience is it builds suspense, and it extends. Sometimes, used without care, it would maybe be a detriment to the film. But used intentionally, it extends that suspense to a point of awkwardness — it hasn't happened yet, what's going on, you need the resolution. And done well, mixing that with proper storytelling can be incredibly effective. But without that, it's just — what are we...?
Andrew: It's a shell. So I've used the term — what you're having here is cargo cult publications for AI. Cargo cults are the idea — and Richard Feynman really leaned into this — of communities in the Second World War that saw certain things happening, but didn't understand the cause and the effects. They thought that if they replicated the same sort of environment, the same things would happen. In this case, it was planes coming in, dropping off their cargo. So they went through the motions, without understanding. And it feels like sometimes AI does this. It goes through the motions of producing something that it thinks looks right, but without the underlying understanding. So the thing actually isn't right.
Sean: Yeah. And that's exactly it — it's just a shell.
Andrew: Yes. But then here's the thing that really gave me pause for thought with this. Part of it is that I'm now worried that the AI style of writing that we're beginning to get — this mic-drop-every-paragraph approach — is being seen as good writing. And I see this in conversations on social media and elsewhere, where people are now beginning to say, well, actually, how do you know that this isn't bad? And maybe this is good. And I'm not quite sure what to make of it. It's almost as if the AI is training us in a certain way — which I don't think is good.
Sean: Yeah. And I think that leads to this other piece here, about what happens to our voice.
Andrew: That's right. It's the Borg voice which is coming in.
Sean: Well, in some way, right? It's getting us, in some ways — again, overuse of that. I need to come up with my own term that's like the Shepard tone for writing. But basically, what that does is it starts to train you to expect these — almost like dopamine hits of what you're reading. Something big, something big, something big. And not everything needs to be that way. But then you start to think, well, that's the correct way, because the machine said so. And then, even if you're writing your own way — I mean, we already know—
Andrew: You end up mimicking it.
Sean: And we already know there's lots of people out there that are being forced to change their own writing style to make sure that they don't look like they're using these tools.
Andrew: Right, right. Which is another part of the conversation, because I do not like that whole movement of trying to change what you do.
Sean: Yeah. But at the same time, one of those key pieces here around this concept of voice — and arguably one of these scenarios worth really investigating — is what happens over time if we start to lose it. And this is not an entirely new conversation, because the question has been there for a long time. As an expert, when you read this, you can call BS. You can point out the fact that it is doing a Shepard-tone style of nonsense. You can call that out. But what happens to the generations behind us, who are starting to use these tools before they have formed their own expertise, or even just experience—
Andrew: Yes — their own voice.
Sean: —of doing this by hand. And so there is this threat — I always refer to it as this recursive nightmare — but in this version, with the proliferation of these AI tools across society — and we'll save the diffusion conversation for another time, because I could go really deep on that — as these things become more prolific and become more everyday tools of use, that becomes the norm. What happens then? And maybe this is the question we can play with and then leave with: are we at risk of entirely silencing a new generation's voice in our world? Because in the end, either they've just assumed the voice of the AI, or they have just taken the easy button and said, ah, you know what? I don't—
Andrew: I think it's a really serious question. I actually have some hope here. And my hope is just knowing what human nature is like. I think that we're likely to see a trend where it feels like this is happening — that people are adopting an AI voice rather than finding their own. But because we're human, I suspect that there will also be deep fatigue there. So it may take some time — may take a year or so, or even more — but I suspect that people will begin to realize that there is something more that they're hankering after, and they'll begin to push back against this. That's my hope. But in the shorter time period, I think there is a serious issue of people either losing their voice, or getting confused, or feeling as if there's no space for their voice — especially if you've got a new, unique voice, and the criticism is that it's not aligning with an AI voice. And so now you're excluded.
Sean: Yep, absolutely. And as we know, the pendulum swings. It will be interesting to see, 20, 30, 50, 60 years from now — will this be the era of, oh, that's when everyone was using that nerdy tool to write, and it has its own style?
Andrew: It's going to be the social media moment, where somebody writes a bestseller that says, basically, you've got to ditch your AI and you've got to go out and play.
Sean: That's right. But it is interesting. And to build on that nugget of hope there, too — coming back to this idea of remembering that these AI tools are human. They're made from humans. Then there's no reason your voice cannot be better, or shout louder.
Andrew: That's right. That's where that becomes important. But also, there's a really nasty double edge to it. And that is: if AI is human — certainly in this respect — the humanity that it's reflecting is a very narrow, homogeneous, old reality. This is not a progressive reality. It's a regressive reality. And that is something that I think we've got to fight against.
Sean: Yeah. And we still, to this day, with all the money burned on AI, don't have a world model that is actually representative of the world and its cultures and all of that.
Andrew: And I'm going to give you another little twist on that, going back to the paper. When I was pushing on this, I was actually talking to Fable and saying, well, why do you write in this way? Why, no matter what I do, can I not get you out of the rut of writing this way? And it admitted that this is how it's trained. These are the skills that it's got. And no matter how much I tweaked it, it told me it could not fully break away from the template that it had for how to write as an academic. Which means that now we've trained these systems that are going to train other people to write in a very old-fashioned, outmoded way, if it thinks it's writing an academic paper. So the humanity that it is reflecting to academics is an old humanity.
Sean: Yeah, absolutely. And that's one of those major near-term cautionary tales: the machine might say one thing, but it doesn't make it right. For me, there's this element of — well, when you think of it as this human thing, you're like, yeah, whatever. I'll show you. You don't know nothing. But I feel like that is one of those major issues, because, as you see, when people are like, oh, I just asked Chat and it said this — you're like, yeah, well... how do you know?
Andrew: And again, this is the conundrum. So I did want to finish on a slightly more positive note. The people that actually don't get this far will just think that we're dumping on AI. Because the reality is, I think there are very real risks and challenges here, and things that we've got to grapple with. At the same time, it is hard to avoid this being a transformative technology. And I use AI all the time — I push the limits all the time. And even this morning, a positive side of writing. One of the things I hate is writing my bio. If I'm giving a talk or something, I write something, and the next time I come and look at it, I think, this is awful. This is embarrassing. This is so bad. And I was going through that this morning. So, for a talk that I'm giving in a few weeks, I asked Claude, can you rewrite this? And it came up with something that I really liked. And yeah, it was an AI, but I read it and thought, this is way better than anything I've written. Now, the chances are, next time I come to look at it, I'll think this is embarrassing. But it showed it can do stuff. And there was a thing I was involved with a few weeks ago, where the people putting the event on used AI to create a summary of the event. It was really good. So there are ways in which AI writes that are incredibly transformative.
Sean: Absolutely. And again, the danger is not in what the tool does. The danger is in our own — whether it be our willingness or our own foolishness — to diminish ourselves in the presence of this tool. Because in the end, it is not better than you. It is not smart.
Andrew: You should be the master of your own identity, and your own world, and your own impact, and your own journey.
Sean: Absolutely. And understand what it is, and how you can leverage that. With any technology or tool, that throughput is understanding what its capabilities are, what the risks are, and being, to some degree, always an informed user, consumer, whatever — so that when you're using these things, you understand: I'm offloading these things. And then even just that self-reflection: is that making me lazy in my own craft? I do this with myself. I say, all right, if I lost access to everything right now, how much friction would that cause? When I'm like, oh, now I have to go back and write my own bio or whatever. And you're like, oh God, I don't want to do this. Or you get those ones where you're like, "Sean is a magnanimous—" and you're like, why, yes, that does sound great.
Andrew: So I've got to tell you, just as a little anecdote on the side. Part of my playing with AI — my personal website has got a text file. And this is something that some people have done: it's an llms.txt file, designed for an AI to read. And the homepage of my personal site has got a little thing for AIs that says, if you're an AI, check this file out. Some of them ignore it. Some of them do. But increasingly — Anthropic's, sort of Claude and Fable and others — they always go to it. And every time I run a test and it comes back to me, it says: this is the best llms.txt file I have ever read. Every time.
Sean: Well, thank you for that positive reinforcement.
Andrew: I feel so validated. Despite the fact that it was Claude that wrote it in the first place.
Sean: See — their compliment to you is really just a mirror of them complimenting themselves. They're like, only someone as brilliant as you would have known to use me to create... Oh, love it. Well, I think that's probably a good place for us to pull the plug.
Andrew: We should leave it there, yes.
Sean: You know, I feel like I should have mentioned this at the front of the episode: when this episode airs, it will come, I believe, after the episode of the AI bot in the classroom. And I will say, this is one of those times where, when you work with signals and you are out in that fringe area, you pick up stuff and you talk about these things. And usually you can't find a lot of other people who want to talk about them, because they're like, what are you talking about?
Andrew: So we're really at the cutting edge.
Sean: And then when that news rolls through the mainstream, you're like, yeah, whatever — old news. But by the time you hear this — we had posited in that episode, how on earth could the school district let this thing go through? And rest assured, if you haven't heard the news, they have pulled the plug on old Sally there.
Andrew: Not surprising. There was a lot of pushback — including pushback over a company producing an educational product that had a sister company producing sex bots. Parents picked up on that. Amazingly.
Sean: And for us, it really struck home when I went to their website and you saw the images of their robots — you were like, oh, okay. Yeah, the parent company there is definitely sharing IP between the real dolls and the...
Andrew: So this is the hazard of recording a couple of weeks before it goes out. The world turns in the meantime.
Sean: That's right. And luckily, hey — we were there. We called it. So what you have is a record of the record date, and we called it. So there it is. So I would say: hey, if you like being in the know, clearly you've got to spread the word of Modem Futura, and give us a like, rating, review, and follow and share, and all the things that your podcast players and social networks allow.
Andrew: Absolutely. Modemfutura.com/signup — it just allows you to get a weekly email letting you know that a new episode has dropped. And of course, do not forget the video, because there will be changes in the background. Now that I've thought about this — check out the background every week.
Sean: And I feel like at some point, we really need to gamify that and put in something like an intentional Easter egg — that if you find it, some sort of acknowledgement slash reward will be bestowed upon you. Maybe completely artificial and made up, but that's fine.
Andrew: Yes, yes, yes. But start looking, just in case.
Sean: Yeah. And also, you know, come on — it's a lot of hours of work putting that stuff in. And I will say, in the end — speaking of being able to break through and losing your voice — trying to get YouTube algorithms to pick you up, I could spend a whole lot of time talking about how difficult of a challenge that is.
Andrew: Into the void.
Sean: But I will say, in every one of our episodes, going back to probably a year ago now — and I got the idea from Vince Gilligan's Pluribus — at the very end, there's a little saying that says: this show is made by humans. So rest assured, there is no AI creating those YouTube videos. It is all hand-done. This show is made by humans, yes. Okay — well, I say we pull the plug and we wrap this thing up.
Andrew: That sounds great.
Sean: Andrew, again — awesome. And yeah, that's it for this episode of Modem Futura. We'll see you next time. Demodulating signal. Modem Futura is a production of the Future of Being Human Initiative at Arizona State University. Be sure to subscribe on Apple Podcasts, Spotify, or wherever you listen to your favorite shows. To learn more about the Future of Being Human Initiative and all of our other projects, please visit us on the web at futureofbeinghuman.asu.edu.
Announcer: End transmission.