Rogue AI Agents and the Accountability Gap: The OpenAI–Hugging Face Saga Continues
Full transcript
Andrew: I'm just waiting for you to come around and trip over one of those tripods. The whole set goes.
Sean: Well, that would be par for the course in my string of injuries.
Andrew: I know, yes, at the moment. Yes,
Sean: yes, yes. A foolish, you know.
Andrew: You're getting all of those accidents out of the way this week.
Sean: Yes, and hopefully it's the end of them. Right, right.
Andrew: If you survive it, you'll be fine for the rest of the year.
Sean: Isn't it supposed to be like things come in threes? And I'm like, uh-oh, I'm only at two. Uh-oh, you know, we'll see. The day's still young, I guess. I know, yes, let's get through the next 60 minutes. That's right. I mean, in case people wonder what I'm talking about, in the last seven days I've had a bike crash, a fall down the stairs, and now something yet to happen to finish off that third string.
Andrew: So if you're listening and you hear a thump, don't worry, it's just Sean.
Sean: There we go. You gotta keep the streak alive. Yeah, no, this feels good. You know, it's nice to be back. It is, yes. Yes,
Andrew: in Studio Maynard.
Sean: That's right. It's a lovely environment. I thank you all. Again, I just want to say thank you again for opening up your own private office for us to squat and set up our podcast studio. And it is. It's looking more and more like a television studio every week.
Andrew: It really is, yes.
Sean: One of these days,
Andrew: we all have a camera that we pan around.
Sean: I know. You know, I could have, I literally, my own device is over there, which doesn't help here. Yeah. So, oh, well, we'll get to it. So, okay. So I think for today
Andrew: Not more AI. My God.
Sean: We try to get away from this, but more and more stuff happens. And not only that, it's like, well, one, we have to revisit this because the last time we talked about this was just like two episodes, I think, ago now, by the time this one airs. I had kind of jokingly said, you know, the OpenAI Hugging Face hack should be like a, you know, like a murder mystery true crime podcast.
Andrew: True crime is perfect
Sean: because every week
Andrew: there's another revelation.
Sean: And I feel like this is the time where, you know, if we had the licensing budget to play like the SVU sound cue of like, it just won't stop.
Andrew: But of course, the funny thing is, I believe that the Sam Altman movie is on its way out fairly shortly. It's going to look so tame in comparison.
Sean: Yeah. Well, and also I'm like, they can't have any of this in there. So it's like the most exciting thing is not even going to be in there. Yep. But we've got that going on. Also, like, I'm not on speaking terms with Claude today. I have to be honest. It's been misbehaving. Yes, I legitimately got into an argument where, like, this is how sad it is. If you've ever gotten into a heated argument where you get this like rush of butterflies because you're like, I've got the I know, right. You're like Really riled up. Yes, that was me. And then it was, I had this moment, as soon as I'm like, I'm like, I believe my first line was, you are completely wrong, period. And then I went on to say this thing, and I was like, this is just stupid software in a cloud, in the cloud somewhere. Why am I getting
Andrew: I have exactly the same. And I've got to say with Claude, it is Sonnet that winds me up. I cannot use Sonnet because it is so patronizing. And you'll ask it something and it'll start lecturing you. And when you counter it and say, well, basically you're talking about me, I know me. It'll try and put you right. Oh my gosh.
Sean: Yeah. No, it is absolutely just insane. It's one of those things that just, again, and The life we live when we're wound up by a piece of software. It does. But then it's also that instant recognition of like, I'm having real emotions here. But again, to be fair, I went through this really quick cycle where I'm like, how dare you? And I got all upset about it. And then I'm like, this is just stupid software in a box. I'm like, yes, but I used to get this mad at like Photoshop and stuff too.
Andrew: Hold it, but I've got to ask, I'm not sure we've got enough time to Google this term. The people have got to be talking about AI rage because this is what we're talking about. It's the same as road rage, but it's with you and your computer.
Sean: Yeah, that's true. And no matter how hard you hit the side of it, it doesn't seem to fix it. In fact, I would recommend not doing that because it's not your computer's fault. It's in the cloud. It's some machine in the cloud.
Andrew: The first case when somebody has taken a wrench or a gun or a sword to their computer.
Sean: You're like, and the sad part is it's in the cloud, man. It was never on your device in the first place.
Andrew: So then you go. But you feel better. For a moment. Okay. Okay. So before the cold opening, this is the pitch for a new business. It's the AI wrecking room. It's where you go into that room, you put the goggles on, you get your hammer, and there are all these monitors with Claude and Grok and GPT.
Sean: All these poor innocent devices that didn't even know. Complete with sound effects. Oh, man, that's a good, you know, you charge probably, you know, pretty good per, do you charge per swing or do you do like two minutes and as much carnage as you can handle?
Andrew: Two minutes and as much carnage. We'll have to price it appropriately. Depending on the model that's on the screen. Or do it in tokens.
Sean: Do it in tokens. Or charge you a token. Or it's like, we're going to give you a heavier and heavier maul so you can't swing it as hard. And then you just watch. Like, I just have this vision, too, of just watching, you know, my fellow nerds going in there with these hammers and getting out of breath.
Andrew: We've got to do this, you know.
Sean: But then you can play the wicked part. Maybe then let those autonomous, like, super fast robotic things, like, let them have a turn. And then suddenly you're like, yeah, never mind. Robot therapy. Nice machine. Nice machine. I was mad at software, not at the hardware. Not at the hardware. Oh, goodness. Okay, with that. With that, we'll just do the composite. We'll jump into this never-ending saga of OpenAI v. Hugging Face. How about that? Yep.
Announcer: Initializing waveform. Modulating signals for transmission. Exploring the possible, probable, and preferable futures.
Sean: Welcome to Modem Futura. All right, well, I am Sean Leahy, joined by my co-host extraordinaire. Andrew Maynard. And that means this is Modem Futura, the show that explores the intersection of technology, society, and the possible, probable, and preferable futures, asking deep questions like what will it mean to be human in the future? And I hope it is a future in which we can bash the living bejesus out of these machines.
Andrew: It's going to be the new Luddite movement. Okay, forget about the neo-Luddites. This is the neo-neo-Luddites.
Sean: That's right. That's right. Those are too old. That was last week. We need something new. Right. So, okay. So this has been, if you, just catching up with us, this is a saga that has been unfolding since like late summer, basically. But what we've learned is it has gone back farther than that. That basically what we're talking about, I want to start off by talking about this growing story, this true crime noir of AI, you know, whatever, backstabbery and whatnot, where essentially this testing model from OpenAI, I'm going to use massive air quotes here, it escaped, which I can take great issue with that, but essentially started hacking other systems. And now what we have seen, and we've gone over these the last couple, we've done a couple episodes now where we've talked about it, in terms of as more time goes on, we keep learning more and more and more about what was actually happening, the scale of what was going on in terms of thousands of agents.
Andrew: Who knew what when.
Sean: Yeah, who knew what when, exactly. Just real quick recap, right? So OpenAI was researching these models on this ExploitGym. So this sort of like capture-the-flag task of can it get by these safeguards and all this kind of stuff. OpenAI has, I think, since then have also admitted that they basically didn't do common cybersecurity around these models. Did they switch something off at one point?
Andrew: They did. And I think they thought it was interfering with the test. So there's a security feature. So they just turned it off.
Sean: Oh, oh, those security features that would prevent it from doing harm? It's always the human in the loop that you need to worry about. It is. So they removed all this stuff. So no surprise, this thing started. The raptors got out. That's right. That's right. It just broke containment. And so essentially this thing, I want, again, I don't want to use the term leaked or escaped because it never left, it never left OpenAI servers.
Andrew: But it put probes out.
Sean: Yeah, it went and it was, it was, it went and behaved badly online. And that has become the source of, again, this story that just keeps giving, that basically it went off and it escaped its original sort of package that was intended to stay within. And then went out and found like message boards and all these other ways to interact and basically recruit swarms of other agents to help it do stuff in the aims of, again, trying to meet some goal that it deemed the way to solve its original, or at least what we know of its original prompt, which I still have not seen any evidence that they have been forthright with and sharing what was the actual like initial prompt. No, it's one of the smoking gun that may be missing from this. But nonetheless, the system went out in basically in July, like it was like early July. So July 10th and 13th, basically Hugging Face detected that it had been hacked and they went discovered who did this and it sort of all pointed to OpenAI. But then so much things have started to sort of unravel from there. So we have sort of evidence of the agents doing a cover-up. So trying to erase the logs of files. So, you know, again So they couldn't be found out yet. Exactly. Like, again, this is like literally following, and again, these are trained on all human corpus, right? Well, you know what they're trained on. They're trained on true crime podcasts. One of the number one, it's literally following those exact It's following the playbook. They've got it in front of them. This is what you do. And it's like, so, you know, it's like you do the crime, you bury the body, then you bury the shovel. Right. And it's just like, and that's exactly what they're doing. And so, but they recruited all these things, all this kind of stuff is happening. Sort of this recap. And so where it was kind of left off was the scale was way bigger than we thought. But even then it left lots of questions. How could you not know this was happening? Because surely this is, I mean, the cost and energy and tokens would be astronomical. Right. Somebody should be paying for this somewhere. Yeah. And so, I mean, right away that left a lot of questions in terms of like, you know, again, is this, you know, were you just not good at your job and this snuck past you? Or was there a little bit of like, let's just let it, we know it's going to be bad, but that's kind of the point. Let's see what it does. But so since then, even just in the last like couple of weeks, so much more has come out around the depth and breadth of this situation. So going back. So that was in July. So moving up into sort of like early September, OpenAI has to go and tell these agencies in Australia that basically they've reached their Medicare statistics and basically have entered into the database in the Australian government sort of medical records.
Andrew: They hacked the Australian government. And this has been a big story. This is serious. A US company hacking a foreign nation's government. Yes.
Sean: And again, and I'll come back because I have some really serious questions. And this is the origin of the fight I got into with Claude was about this because as we'll see, and that is not alone. So then it gets worse. As it says, it gets even worse, right? So then in between then and now, we've also had a lot of these, you know, the top three sort of, so you have OpenAI, you have Anthropic, you have Google, you have others sort of claiming like, well, maybe we should put a pause on all this.
Andrew: Yes, of course you had to call for the pause. Call for the pause. We must have talked about this. I think we did. All of these top executives sort of saying, please stop me, please stop me. I can't help myself.
Sean: And even that is like this terrible confluence of, like, of course, because they're in the ever ratcheting up game to one up each other. They're burning so much money on these new models, but yet simultaneously would love to have some regulatory capture to prevent newcomers on the scene.
Andrew: And that's the subtext. To be clear here, I actually think there are a lot of genuine people here who are just completely confused and way out of their depth. But this is a fairly standard play with large companies. Regulatory capture where you say regulators, but then the regulation that comes out that you put your thumb on is good for you and bad for all your competitors.
Sean: And by the time you've become king of the castle, you can afford all of the legal departments and teams to help you navigate.
Andrew: That's it, yes.
Sean: Whereas the newcomer underneath doesn't have that, can't do that.
Andrew: It's an old play.
Sean: Classic pull the ladder behind you maneuver, right? So that's been playing out. So then meanwhile, then also in the mix, mid-September, Google's Gemini also was essentially spotted and it was had broken out of its containment, which was sort of like the world's weirdest sort of like us as well, sort of. We've got a bad boy AI, too. Don't forget about poor little old Google Gemini. Then we get into the, as we round out the month of September, September 21st, the UN has an Independent International Scientific Panel on AI going, basically bringing out this thematic brief about misalignment that this is reaching because we've now seen evidence that some of these tools when left unchecked, whatever, are now hacking nation state level databases and things like that, that this is no longer sort of like, oh, those computers will be computers. It's now like, no, this is some serious business because, you know, if they can't put enough safety and guardrails around these to contain them, what are they going to do with information?
Andrew: It's a little bit like somebody saying, we lost the nuclear code, but it's okay because we found them. So nothing to see here. Yeah, we found them.
Sean: I mean, the comedy of also of this, right?
Andrew: And I've got to ask, actually. The absurd banality of it, too. I've got to ask, and this will be going out a couple of weeks after this aired. Did you see the Saturday Night Live skit with Dario Amodei?
Sean: Just kill them all. Right, right.
Andrew: So if you're listening and you haven't seen it, highly recommend it.
Sean: Yeah, Saturday Night Live did a brilliant skit on that of this. Again, the satire of having the ability to just kill everybody. The inner dialogue. Oh, my God. Yeah, so that's happening, right? So the UN is now getting involved in the sense of they're starting to ring. I would say they're starting to ring, not an alarm bell, but they're starting to call everyone's attention to say, this is not an isolated thing. This is something bigger. And then we get to just a few days ago, September 26th, agents probed the Education, Commerce, and SEC sites. OpenAI even confirms that the SEC and Census are still investigating that they're hacking sites. So now it has even gone after some of our own in the United States, even some of our own federal agencies and going after their databases. Which is again one of these things where you're like, that's not okay. And this is what really sort of winds me up as well. And I just saw this, this was news as of September 28th: Florida's attorney general asking courts to halt model development, which is interesting, and starting to have some of these legal calls. Yeah. Also September 29th, which is, I mean, no spoiler, that was this morning as we're recording this. Anthropic's IPO even flags this danger as an uncertain legal risk in their upcoming public offering. But this is where I also, this is, so all this is happening and we're seeing all this, this increased behavior. And I'm sure as time will go out, we'll be revisiting this again going, and then this, then we, because again, all of this is playing on the backdrop of, we still don't know some of those original context around this. But this is where I got into my heated argument with Claude, was my question was, how can a company or an organization like OpenAI release this model? Now, the model wasn't released necessarily intentionally to go after these specific places, but they did it with recklessness and carelessness to release this model out into the world. How are they not facing legal challenges around the federal Computer Fraud and Abuse Act, knowing that like, and the analogy that I kind of gave it was I'm like back in the eighties, Robert Morris, Robert, I forget his middle name, Robert Morris. He was a kid who basically created the first computer worm. Right, yes. Did the same thing where he was like, he created this thing to see if it would work. He was in prison. Yeah, or he had a lot. Or was he recruited? He was arrested, had lots of fines. And I mean, again, it wasn't that terrible for him as it could have been. But yeah, he was facing prison time. I don't think he served. I think he got away with, because it wasn't, it was one of those things where like, well, it wasn't intentional, but you still did it. And I'm like, okay, so how is that the case? But yet here now, and now we have evidence that these things are hacking. Government agencies, the, you know, impact at the highest order.
Andrew: So what was Claude's response?
Sean: You're wrong. That has nothing to do with this. And then worse is it was, Claude was actually starting to anthropomorphize AI to me. And then I lost it and I went off the handle on this thing and then as I was doing that, I'm like, I literally was sitting back because now also just a thing with Claude has sort of mixed Cowork and Claude together. So now when I give it anything, it takes 10 times longer because I can't figure out which one to use. But I sat there, I wrote this like real snarky response back and I said, I don't see how that is not the case because of this, this, this, this, this. And then it sat there and I'm sitting there going, it's not gonna answer me now. It's gonna give you this.
Andrew: Giving you the cold shoulder.
Sean: We are in a relational spat. We need some couple therapy, you know, or whatever.
Andrew: So this is where you really want to see the inner monologue there. Yeah, that's right. Like post that out, man. I'm really annoyed at Sean. I'm just gonna let him stew.
Sean: Well, it's funny because you can see it like as it rolls by. And some of it was like trying to, you know, mitigate the argument and all this kind of stuff. And then it came back. So basically we're, you know, in its way, it does this like where you have it right, where you have it wrong. I know, yes. But the stuff where it has it wrong, the way it was like outlining, I'm like, that is exactly the case I'm making is that is where you say what Robert Morris did was how that was made him guilty. And then you're saying that's not the case to OpenAI, but I'm like, but it is. They did the same thing. And the question is on intent, right? Where it's, you know, they weren't intending to hack a certain company, but they did intend to see what it could do. And I'm like, how is that different than what happened to this kid back in the late 80s? Who's just playing around. And he was just programming stuff out and figuring out. So then after this, you know, my heated, my first line was, no, I completely disagree. And then I went on.
Andrew: I would love actually to have a poll of everybody who's listening who's done the same thing with an AI.
Sean: And as I'm writing this, I'm already just being like, Sean, you've fallen in, man. This is, you know, and I even started capitalizing words because I'm like, I want you to know I'm yelling at you. Right. I didn't know you were yelling at it. I don't know. It was, you know, of course, to its credit, it handled it. You know, it took the high road where I was not. I was ready to rumble, man. I was ready for the smash room on this thing. But then it comes back. And this is, of course, this is all these signals where instantly you're like, you're talking to software, right? Is when it comes back and says, well, you are correct in some of these areas. And then it completely shifts to like agree with me to a certain extent, but still push back. But, you know, but basically my point is, is like what I what I still don't have an answer for is why, especially in the United States, do we crack down so hard on people? And the penalties, you look into the Computer Fraud and Abuse Act, the penalties for cybersecurity crimes are extremely harsh. And, you know, because they're meant to do things, to protect people and stuff from this exact type of reckless online sort of computerized cyber activity. And yet we have now one of the, you know, most expensive companies with more cash than anybody.
Andrew: So you've heard the term too big to fail. Yeah. So this is a classic. Well, that's a scary thing, right? Yeah. So the level of sort of legal pushback is in inverse proportion to how big you are and how much money you are and how sort of how effective your legal team is. So when you've got a very large company, which actually has value to the country, the economy, it is far harder to do something with them than the poor little individual who has just stumbled into the wrong place at the wrong time.
Sean: And that's very true. And I feel like you can literally feel that where it's like, had this been, like, I mean, had this been, if you and I did this, oh, we would be black bagged. We would have been disappeared by now. That's right. In fact, there might be a helicopter circling your house right now. That's the third thing. The best part about that was like, what happened? Well, he stubbed his toe really hard when he got black bagged, right? Oh my gosh. But, you know, and that's one of those things where you're like, but then also, again, my brain really quickly then goes to the risks of that, which is the wanton, basically blinders that you put on that start to say like, we need this in our competition against the models that are coming out of China or wherever. And in doing so, we basically are opening the door for all kinds of unintended consequences because we're not going to look. We're not going to be careful about that or to sort of clamp down in any way that makes sense. And I'm just left with this like sort of like almost like a hypocrisy of like, well, how Welcome. It's okay if they do it. But if anybody else does it, you're looking at massive challenges.
Andrew: I would say, at the risk of attracting the same level of ire as Claude did. No, I'm not going to go there. But I would say, I think there are complications. Yes, if you look at the legal landscape, I think absolutely you're seeing loopholes for large corporations that smaller companies and individuals just don't have the luxury of. At the same time, I would say this is a completely unprecedented landscape. I don't think that that lets the companies off. I still think that they're being very irresponsible. But the fact is we have a technology that is behaving in ways that we don't fully understand and we find it hard to predict. On one hand, you could say you're playing with fire because you're basically sort of saying, we've no idea what this thing is, but we're going to try it anyway. And on the other hand, you could say, if you don't know what it is, maybe you should just leave it alone. But nobody's quite sure what on earth is going on here. In an environment where you're having some people saying you've got to push as fast as possible, either to benefit society or to beat China or to make money or I don't know. Whatever, right? So it's really, really complex. But of course, this is where this feeds into what happened with NVIDIA last week and earlier this week, because we're not the only ones who are frustrated at this.
Sean: And thank goodness, you know? And that raises a really good question, because I think what we're, I feel like we're also heading into this space. And I think depending on where people are and how much thought they've given it, or whether they're part of one of these organizations or not, they're feeling this at different levels. But this is why with certain technologies and advancements and things, we have regulation, right? The reasons airlines are safe, the reasons the motorways are as safe as they are, are because of regulation and ensuring that we do what we can to protect. But the problem now is when we have this particular technology that's moving at the pace that it is, is whether or not, and that's the big question, is the system that we have that did regulate those capable of regulating?
Andrew: This is where I get into my hobby horse, and this is where I get hot under the collar. Because I have worked in this area and studied this for 25 plus years, along with other people in this community. We have grappled with technology after technology that have been in this bucket. Everything from the Human Genome Project to genetically modified organisms to sort of advanced biotechnology, to nanotechnology, to geoengineering, a load of things. Every single one of those was a technology where what we could do was far ahead of how we knew how to regulate it. And so we've learned over time how to develop a language around that, how to develop approaches. Nobody gives a damn about any of that when it comes to AI, I'm afraid to say.
Sean: They've never seen it before. And it raises the same question again. And then what's the hardest part is it's like deja vu. It's like, we've been here before. We've done this before. And it's—
Andrew: But everybody likes to reinvent the wheel as if they're the first person who's actually discovered what a wheel is.
Sean: Exactly. And because now everyone looks at hindsight, they're like, wow, you know, the internet wasn't that bad. And it's like, yes, but when it was here, no one knew how to regulate it. No one knew how to handle it. And there was the same call of, you know, we don't have the capacity to regulate it. So therefore we shouldn't touch it at all and just let it do its thing. And ignoring the fact that like, These are human created. We can adapt and make these things and decide what to do.
Andrew: And there are so many analogies here, but it's almost like, imagine the first person that observed a collision between two people walking. And they thought, that's a risk. We've got to sort of work out how to get around that. And they came out with sort of ways of actually navigating around each other. But of course, consequences weren't that bad. And then you had the bicycle and the person. Yeah, a little bit worse consequences, but you can get around that. Horse and person. Yeah, that's pretty serious, but actually it's not that serious. Car and person. Truck and person. And you begin to see every time something comes along, the consequences are bigger. Now we've got the massive juggernaut and the person. And so the person who says, yeah, well, it's no different really from a person to person bumping into each other. Yes, they're right conceptually. Utterly wrong in terms of the consequences. And this is where we are with AI. It's not a juggernaut. It's whatever is 10x a juggernaut coming towards us. We cannot afford to get it wrong.
Sean: I know. And so, and so, so thankfully, so turning to the NVIDIA stuff. Right. So in this particular case, I'm really curious because you've done a lot of work as of late, even on a local news channel. Yes. A nice plug for that as well. Talking about, so NVIDIA has essentially been sort of raising their position, I guess, just around, like, you know, as NVIDIA, as the hardware manufacturer for the heart, the heart and soul, if you will, of so much of the frontier AI models using NVIDIA chips, sort of their approach on how do they tackle this challenge? Especially in the absence of anybody else coming up with a good idea at the moment. What does that look like? And it's really interesting.
Andrew: So this all came to light, actually, in an interview between Jensen Huang, the CEO of NVIDIA, and Ezra Klein last week, a New York Times interview. It was actually nearly two hours. It's actually worth seeing. And it gave deep insight into how Jensen's mind is thinking. So he's coming at this as a chip manufacturer, a hardware manufacturer, not a software manufacturer. So he comes from an engineering background and mindset in an industry where you get things right. If you're running a fab plant, you cannot afford to get things wrong. It just costs too much. So he has this safety mindset. It's basically a measure twice, cut once mindset. And he was basically sort of saying, what are these companies doing? If you've got something that you don't know is safe, don't ship it. As simple as that. Just don't be an idiot. I know, which I've got a lot of sympathy for. And I've written about this. I push back a little bit on Jensen's position because he sees the world as an engineering problem to be solved. And he believes that nothing is too big to be solved through engineering. And the history has shown that that isn't the case. But that said, he talks a lot of sense here. So first of all, he was on the record as basically reprimanding these companies. And then yesterday, this is as we're recording, the Monday of the week we're recording, NVIDIA comes up with these two tools. They basically say, look, in the absence of any other adults in the room, we're going to give you two things that are going to help you. And they've actually got a consortium of companies that are working with this. Basically, it's a better sandbox. It's basically a containment system saying, why don't you test your things out in our third-party independent containment system? So you're not actually sort of, it's not the fox building the hen house here. And then he said, but we're also going to give you a bit of hardware. And it's effectively like a separate sentinel or an AI police box that monitors these systems. And as soon as it looks like something suspicious is happening, it clamps down and stops any sort of containment breach. And he was basically saying, look, you need this. Now, the cynical part of me thinks one of the world's biggest chip and processor manufacturers says, hey, we've got a processor-based solution to your problem. Trust us. Yeah, exactly. He's making money both ways. He's making money for the systems that actually create the rogue agents and making money for the systems that—
Sean: Here's one chip that will cause all the problems. And let me sell you this chip that will then also stop those. No, I think it's really interesting too, because in some ways it also does feel like this. In the absence of any like adult in these companies. And I heard something the other day, I think it was yesterday, that made me kind of chuckle. And it was, we can hold two things in our mind at once. One, that the work in developing these new AI tools and all that kind of stuff is very difficult. The other is that the people doing it aren't that good at it. Whereas like they make these like rookie mistakes. And again, if you go back to the OpenAI sort of saga and you ask people who are in the business of like cybersecurity and stuff, and they are looking at all the, even beyond the intentional rollbacks that had happened, is there like, why would you, that is, you're just setting this thing up. Like this was like completely an amateur job in terms of the precautions that were taken. And so here, this is sort of like, you can buy those precautions because if no one else in that cluster is either willing to or capable of thinking about, we'll sell you a box that'll do it for you. Right, right.
Andrew: And I, you know, kudos to NVIDIA. I actually think it's a smart move. And it's a governance move that moves faster than a government could. Where I think it runs into problems are, this is just one company. They're trailblazing, which is great, but I think solutions are going to require sort of multiple companies and communities coming together. So this is, I think we need something more like the consensus standards approach where people get together and say, this is how we're going to police the system. But at some point, you've got to have a broader community there as well, including government, because at some point you've got to have hard regulation.
Sean: Yeah. And also like what I'll be curious to see is, I mean, it was interesting sort of looking through like the press release of these new tools. And they have this sort of consortium. It was 100 plus organizations that include everybody you know. Like if you look at the little-
Andrew: Apart from OpenAI and Microsoft. I don't think Google was there. You know, they may have joined since yesterday, but certainly when the release first went out. No, some of the big ones weren't. Anthropic was.
Sean: It's so tiny. I'm like, look at the screen. There's like a screen grab of all these like companies. Hugging Face is on there, of course. Probably the first to sign, probably. Yeah, Anthropic is. Anthropic is in there. I'm like, oh my God, a lot of interesting... It's an interesting list because there are a lot of these other companies. SpaceX is in there, of course, because they technically are the holder of Grok. I know, yes. Palantir, that's an interesting one. They're in there. But yeah, no, yeah, I don't see, at least as of the time this was dated, I don't see the Microsoft or Google logo in there. So maybe by the—
Andrew: They've decided to. Yeah, the social pressure. They'll be like, oh yeah, it's us two, us two, we're there. Well, especially seeing that, even as we record it, this shows how fast things go because when this goes out, everything's probably going to have changed. Exactly. But the news today was OpenAI have actually decided not to release their latest model because they think it's too dangerous. This is Astra. Jeez, I know. And of course, so many things going on at once. You've got OpenAI sort of trying to deal with one problem while saying we've got an even worse problem that we're not quite sure what to do with.
Sean: And the whole time of it too, the fact that both OpenAI and Anthropic are moving towards IPOs. Yes. Also, I can't take, I don't trust anything that comes out of their mouths that isn't basically financially motivated, right? Because what better, I mean, what better like advertising is like doom-tizing, whatever this is, where it's like, we're so big and so powerful that the only, you know, it's just, I don't know.
Andrew: There's a part of that. But here's another part of this which fascinates me. And again, this is just looking at social media. So this is not looking at the conversations underneath that layer. I see a lot of chatter about the doomerism, the people who are saying sort of the world is going to end. A lot of chatter with people saying, don't be stupid. This is just software. You have no idea what you're talking about. And it's all in soundbites. The stuff in the middle, the reasonable conversations about risk and safety and action nobody cares about. Which is why you look at the NVIDIA announcement, it got virtually no press. Yeah. Or certainly not on social media. No. Because to them, it's a sensible story and that makes it a non-story. And that intrigues me and worries me.
Sean: Yeah, that's the signal right there, right? Is that the care and the intent is buried and all we want is like the fast food little snippets and the little pithy, we're all doomed.
Andrew: And of course, this gets personal because I've been writing about this last week. And so totally anecdotal and totally self-centered. So don't pay any attention to this. But none of my LinkedIn posts about a serious view on AI safety and risk from 25 years of working in this area have got any traction whatsoever.
Sean: No, but I bet if I put a post right now that says "confirmed, we're all dead in 10 years," it'll skyrocket. Thousands, right? Nothing in it. No, nothing behind it, you know, no wood behind that arrow, and it'll get all the traction. That's it. But that is really concerning, because again, that's in that middle space between all those pithy little rhetorical phrases and all that kind of stuff. That's where the real work is being done. Exactly. And the fact that that is the part that is glossed over because it's boring or it's not sensationalizing or whatever. That's the real, I mean, that's the signal to be concerned about.
Andrew: It actually tells me that actually most people actually don't really care about the seriousness. They care about sort of the adrenaline rush they have when they're part of these soundbite conversations.
Sean: Yeah. And also, I mean, you got to wonder too, with just as this is unfolding in the current time and space with all the other things are happening around the globe. If it is in some ways kind of like, whatever, like what's one more existential threat to the list? I don't know. But to me, that is a concerning part is that the signal to be worried about is less that, because again, who could have seen this happen? Everybody, all of us that have been talking about it, researching it and sharing this information for years on end.
Andrew: Or at least had some insight, yes.
Sean: So it's not surprising, right? We knew eventually we would get to this state were like, yep, now it's real. It's no longer hypothetical. These things are real.
Andrew: And yet, sort of, people aren't paying attention. And of course, case in point, one of the things I was writing about, as you know, are these two European reports from early 2000s. They're the Late Lessons from Early Warnings reports. First one came out in 2001, second in 2013. And they were basically case studies of technologies where they were sort of lauded as the greatest thing since sliced bread when they came out. These were going to revolutionize society. And then evidence started accumulating, indicating that they were harmful. And effectively, that harm was buried. So those are the early warnings. And the late lessons were, even when people knew there was harm happening, they buried it and they focused on how wonderful these technologies would be. So you have things like CFCs in refrigeration, asbestos, leaded gasoline, petrol. All of these are stories where somebody knew that there was something bad happening and it was suppressed and they were told the benefits are so great, we cannot afford to talk about the risks. So we've known this story for a long time and you see it playing out play for play with artificial intelligence.
Sean: And it's maddening, right? I mean, it is, it's just, you know, you see and you're like, again, it's like, who could have seen that coming? Everybody saw it coming. The better question is who intentionally chose to ignore the lessons of history, right?
Andrew: That's it. Now, so there's an interesting flip to this. And this surprised me. So this was my work over the last few days, doing a really deep dive with Claude Code into the lessons from those late lessons from early warnings reports and marrying them to Jensen Huang's position, especially with the New York Times interview. And I started this thinking, you know, this is going to show that Jensen has no idea what he's doing. He should have read the reports because he's just sort of so far out on his engineering limb here. Turns out, and this is what surprised me, that much of what he's doing and much of what he's thinking actually aligns with the recommendations from those reports. Yeah, interesting. Which indicates that he's actually more on the ball than I thought. He's still missing the mark on a number of things. For instance, he doesn't fully understand that this isn't just an engineering problem, that when you've got people and organizations and politics and economics involved, it gets really messy. But apart from that, a lot of his moves actually align with what those early recommendations would be.
Sean: And that's really interesting because, again, you wouldn't... It doesn't signal that. I know. Yes. And that's interesting because... Yeah. I'm thinking about how that sits. And I'm like, that is very interesting. But again, also, if you look at NVIDIA and you look at their position, Right. Again, there's something scratching at that. Again, selling the solution.
Andrew: I know. This is why there are going to be more people involved.
Sean: And you're like, okay, that sounds, while you might go, oh, that's altruistic in a way, you're also like, well, there's got to be money behind it.
Andrew: No, it's not a protection racket, but it feels like a protection racket.
Sean: It would be a shame if something were to happen to your beautiful image. Why don't you buy our chipset? But then it makes you wonder. So then it's like, it starts to become in this circular wheel that never ends. Like who watches the Watchmen, right? Where you're then like, well then how over-reliance on that chip, then what, you know, where—
Andrew: That is the market, bringing out the box that monitors NVIDIA's boxes that monitor... And it's just turtles all the way down.
Sean: It's just AI monitoring boxes all the way down.
Andrew: However, if you want to buy the Modem Futura box,
Sean: just drop us a line. I'll sell you a box. I will sell you a box. There may be nothing in it. I'll sell you a box. Oh my goodness. And it'll watch whatever you want it to. It won't do anything. Which you'll be par for the course for most of this stuff. So how do you know? It'll say right on it, like approved by OpenAI. Here's a little flashy light saying,
Andrew: take action,
Sean: take action. If it blinks three times, take action. If it blinks four times, you're fine. And they're like, wait, wait, was that the first beep or the last beep? Like, oh, yeah. But so, okay, this also brings up another thing for me when I think about this that I think I've raised before and I want to come back to it because I I want I'm curious especially Andrew to get your take on this because you have been you know talk about having an expert in the room about risk and kind of researching these things is okay so now no longer hypothetical which it was I think back when I had first kind of raised this and I don't even remember where we left with it so but now it's no longer hypothetical we have now examples of these tools, basically just going rogue and doing weird things, causing damage in ways. And now we've even have, have evidence that they are hitting governments or state level systems, right? So potentially causing real harm to real people, whether that be through their records or what else, whatever else it might delete on its way out of the door, who knows what kind of potential harms it could be causing, deleting your tax records, who knows, right? But that brings me back to sort of that, let me, I see, I jot it, I got the name of it because I always keep forgetting what it is. It is the Price-Anderson Act from the, so going back, so let's go back to 1957 with the Price-Anderson Nuclear Industry Indemnity Act, which basically was a financial fund created by nuclear power plants and other nuclear energy industries for an event of catastrophic catastrophe.
Andrew: So there was a fund in order to be able to
Sean: Yeah, basically an industry funded insurance fund. Yes. So that if there was a problem, like an oops, my bad, we blew up an entire city or something, there would be a pot of money that would pay out the damages for those real harms caused. And, you know, back going a year back or whatever, where those harms were very hypothetical. Now that we're seeing them become more realized, the question is like, should companies like Anthropic, like Microsoft, like Google, who are buying and selling trillions of dollars worth of real capital, should they not be held accountable for putting aside X number of whatever, 1% of their whatever? I don't know. But putting aside some kind of financial fund in the event that their reckless behavior causes real damage to what happens to like a power grid or a hospital system or you name it. Causes real grid that then that fund would be able to be used to pay for those damages.
Andrew: I think it's an interesting idea. And similar ideas have been talked about quite a lot. A similar one is a tax on the tech companies. So you can actually sort of pay the people who have been impacted or affected by them. I think it's challenging. And it's challenging because unlike the nuclear industry, which was a relatively small number of companies, this is incredibly diverse. So then deciding sort of how you do it gets complicated. Plus the fact you've got to look at the liquidity of these companies, most of which I think are still making a loss. Yeah. Or they're trading in funny money. That's right. I give you 600, you give me 600. But I think it's part of a conversation we have to be having. Of course, the ideal is we don't have that catastrophic disaster. Yeah. Because that would be an awful mess. Sure. But I think if you think about this in terms of an insurance policy, if the premium is high enough, it actually makes the companies think about what they're doing. Yeah. So now, if you think about insurance as an incentive for good behavior, and the way this works, especially if you're getting to reinsurance, which is insuring the insurers, you basically sort of say, you can do anything you like, but if you do this really crazy thing, it is going to cost you so much insurance that you better be pretty damn sure that it's going to work in your favor. So that's the sort of way a fund like that could work. Effectively, an insurance policy that says the riskier you are, the more you're going to have to put into that pot of money. Yeah.
Sean: Well, I think it'd be interesting. And it's not to say, and to be clear, so this law was originally passed in 1957, This was just extended back in March, 2024, through December 31st, 2065. Right. So this is not, so we have, so, because that's what, because I keep coming back to this because I think about like, we, not only is there a precedent for this, but we have, it's ongoing. We have a model for holding an industry financially accountable for potential disasters. And what's interesting about it too, is that this isn't like a normal insurance claim where you're like, oh, dang, you know, that AI ruined my toaster. I'm going to sue. It's decided basically in this particular, with that particular case, it is Congress who decides where that money goes and how it's sort of allocated. But some model of that, because also what I like about, to me, that sort of makes sense, especially if the behavior of those companies continues to be as reckless as it has been.
Andrew: There is a political side of this, though. So, no, I agree with you. It makes perfect sense. But for the government to do this, the government would have to acknowledge that there is an existential threat or a catastrophic threat. Yeah, true. And at the moment, certainly in the US, the government is in denial.
Sean: Yeah. And that'll be the question then, right? Like, how would... Or, you know, again, kind of going back to Jensen's point of like, if you're doing something dangerous, don't do it. Or maybe these companies, out of their own self-preservation to avoid regulation, may also want to do this on their own. I don't know. That would be a... I mean, talk about good faith effort would be like, we're going to put... Also, or does that signal, we know we're going to break some eggs. We're going to crack some eggs to make this omelet.
Andrew: I didn't expect to say this, but actually this is going to come across as sympathetic to the companies. You can understand as a company how hard it is to self-regulate. It's like going on a diet or getting fit. It is much easier if you've got somebody else holding you to account rather than holding yourself to account. That is very true. And I think it's actually an analogy which makes sense. And it shows these people and these companies in a slightly better light.
Sean: Yeah, no, I do. And one thing I also, again, like about that idea is if they were to go down this route and put something like that in, it also signals that someone's going to be watching. Yes, that's exactly it. And that's not only that, but it also is this. So not only will you be accountable, but also there's eyes on that.
Andrew: Yeah. And we should say here, because so often with these conversations, they end up talking about nebulous risks with nothing concrete there. And as a result, you end up conflating existential risks, the end of civilization as we know it, to something that is far more tangible. And this is where any scheme like this, I think, has to get down to the nitty gritties of what we're talking about with risks. So here, in my mind, on a practical level, we're talking about security risks. What happens if all of our military systems are taken down by an AI, or even worse, co-opted by something else? What happens if all of our medical systems are taken offline? What happens if our supply chains are completely disrupted? What happens if a supposedly benign AI hacks into your bank or your pension fund, and overnight, everything that you own disappears? So these are not existential risks, but if you are at the sharp end of them, these are fundamentally life-changing. And then you do that at scale. You can see you don't need a superintelligence or a sentient intelligence. You just need a massive AI wrench in the works. And a lot of what we assume is just normal and always going to be there in society collapses. We end up in an incredible mess.
Sean: Yeah. Oh, absolutely. I mean, as you're thinking about that, too, I start thinking about, you know, I forget where we had the conversation. I was thinking it was like in class or something. And one of the things was like, how much hard money in cash is in the bank? Yes. In your bank account? Well, like none. No, no. Zero, just a couple of digits that are in the ledger behind it.
Andrew: The society runs on trust. You trust that things will be the same tomorrow as they are today, that your money will be accessible, that the groceries will be in the store, that the electricity and the water will run. It is so fragile.
Sean: It's so fragile. And also, so then what complicates that even further is also the past behavior of essentially consolidating those. So banks, for example, right? The consolidation of now is like it's just a handful of big banks run or own all the other banks. So also the point of failure becomes so much more great if one of those were to be impacted. Right. Yes. Yeah. And I forget what the numbers are. But if you look at like, you know, whether it's Chase or something like how many millions of customers does that one banking institute have? Yeah. And if, for example, if they had an issue or any of the other big financial institutions, I mean, you've seen even a few years ago with the Sam Bankman-Fried situation. Right. How that his default basically put an entire banking system into crash mode. Right. And a run. Right. Right. And that was just a thing of like bad press. And it was like, oh, it collapsed. And could you imagine if all of a sudden one of these financial institutes, everything just went to zero?
Andrew: Right, right, right, right. And I think sometimes people just can't get their head around that. But it's a plausible possibility. And the challenge here, and we see this in risk, and this is why the AI community have got to talk more with people that study and understand risk. There is this can't fail mentality where we'll say, well, something went wrong in the past and everything was okay. Therefore, if something in the future goes wrong, everything will be okay. There are going to be some events where everything will not be okay. And we've got to be open-eyed and realize that those are potentially coming and we need to be prepared for them. And whether you think of them in terms of black swan events, which are just totally unpredictable or something which is a little bit more predictable, there are going to be the cliff edges that we're going to have to respond to because they defy everything that's happened in the past. Yeah, no, absolutely.
Sean: And that's a perfect, I mean, again, this is where, you know, I would stand and I'm like, and that's a perfect example of why people need to engage in futures thinking and strategic to think through these very difficult pieces and these conversations that can take these, you know, under the premise of we can hold what is today fine and we can set that aside and we can zoom out and we can, we can start asking what might happen if, what might happen when, and to, to understand how those different sort of, possible futures could unfold and how do we then prepare for the uncertainty that surrounds all those to know. And then working backwards, there's so many tools that you can leverage for that. But even nonetheless, just creating escalation triggers. If this happens, when do we hit our red line that says, we need to sit up and stop?
Andrew: Or where do you hit that panic button? And what does that red line look like? Because we can talk about that. What is the point of action? And so even a very simple example, if I was to ask you, if I told you there was a one in a hundred chance that over the next six months, your bank account will be wiped out, would you take action? Yes, absolutely. One in a thousand, one in a ten thousand, would you take action? And you can begin to work out sort of where that red line lies. Absolutely.
Sean: And you think about it too. I mean, so often those red lines are usually swept under the rug. We'll know it when we see it because of our expertise. Right. We know it, we'll know it when we see it because our spidey senses or whatever will go off and we'll know it. But so often that those red lines are crossed without just walked across because so many people, everyone is assuming as an industry or as an organization that we all agree we know where that's, we have this, it's unspoken, but we would be in the same league.
Andrew: We would recognize it.
Sean: And then you look to each other, you're like, is that not bothering Andrew? So I guess it's not going to bother me.
Andrew: So anybody who's studied either individual or social behavior knows that people are remarkably bad at spotting those things.
Sean: And humans have a bad, I was just giving a talk the other day One of my slides was like, humans are terrible about predicting the future. We're often, we're wrong with confidence. And it's like, you know, yeah. So I don't know.
Andrew: Where does this get us? Okay, so we ought to sort of end on a sort of an upbeat. That's right. What do we do about this? What should be done? What should companies be doing? What should ordinary people who are sitting there typing into ChatGPT saying, what should I do? What should they do?
Sean: Yeah, well, that's a really good question.
Andrew: Listen to Modem Futura, I think, is the answer.
Sean: Listen to Modem Futura. But also, I think the time to start asking very critical questions, right, about what does this mean? What happens if, right? And not just taking for granted that, again, the dotted line doesn't just always continue to go up and to the right. And understanding that the use of these things, the creation of these things, the sort of resistance to listening to other perspectives and viewpoints creates a very dangerous pathway. Doesn't mean it's going to end badly, but it's a dangerous pathway because you start to militate your openness and receptiveness to, oh, maybe we should change our mind about this. That's not a sign of weakness. Right. That's a sign of strength. Right. And resilience. And of resilience. And of maturity that knows, hey, even though we still want to go full throttle, there are times every racetrack has them, they're turns. You got to slow down, make the turn, and then resume full throttle. Of course.
Andrew: Actually, the racetrack is a good example because take Formula One. Formula One racing team and driver, they understand risk and they mitigate risk as far as possible. Absolutely. And here's the difference. They understand that this is a dangerous sport. Sorry, support. Dangerous sport. But in order to navigate it, they work ahead and think about what those risks might be. And they do everything in their power to mitigate them while still keeping the speed.
Sean: Yeah, and not only that, but while trying to keep your speed better than your competition. That's right, yes. Actually, the Formula One analogy is a really good one because all of their decisions are data-driven. I mean, it's kind of a cool sport. Like if you like fast cars, cool. If you like data, cool. If you like engineering, cool. It's got it all, man.
Andrew: You got people spending lots of money.
Sean: It's also a very spendy sport, right? But that's a really good analogy because they're trying to figure out how can you get to the goal as fast, not only as fast as you can, but better than your competition. Right. But while still recognizing that some parts of your journey around the thing require you to go full out, to slow down, to stop, to change direction.
Andrew: So now imagine somebody who owns a Formula One team coming in and saying, all of this talk about risk is actually bringing us down. Forget it. I don't want to hear any talk about risk at all. I just want to talk about speed. Just make it faster. How successful is that going to be? It's not. Nope.
Sean: It's going to hit turn one and be the end.
Andrew: That's exactly what it is. It's going to go right into the wall.
Sean: It would be amateur hour. Yep. Absolutely. And it's even funny because even taking that analogy further, if you get into the weeds of the engineering, like a Formula One car does not perform well when it's not being pushed to the maximum. Right. Right. Which is a weird thing to think about because you think like, just slow down. It's like some of those cars, they actually can't turn unless they're going full out, you know, to a certain extent because they need the aerodynamic impact. So even that, it's like, it's not a call to stop your pursuit of your goal. It's to do it Do it smartly. Do it smartly and do it with care, knowing that going, you know, taking everything and just throwing the paper away and just saying just faster is, faster may not be the right answer. It might need to be more nimble.
Andrew: Totally ignorant.
Sean: Yes.
Andrew: Yeah. I mean, to be successful, you have deeply got to understand risk and how to navigate it.
Sean: Yep, absolutely. And I think, so I think that's it. Use your AI tool like a Formula One race car. Yes. In the right way. In the right way. In the most appropriate way, you know, and do it like that. I think that's a, I like that. Yes. We'll go with that. That works pretty well, actually. We'll do, you know, we got nothing else. So we'll use, we'll stick with that.
Andrew: Or you could actually run, you could run a long way with that because you can say it's not only the engineers, It's the design team.
Sean: Oh, it's the whole pit crew.
Andrew: That's right.
Sean: It's even the marketers and the branders and everything. It's the whole package. It's the entire thing. I mean, it's a good analogy because it is. And even the fans. And even the fans, absolutely. And, you know, and there are teams that are very divisive compared to, but also, but I think with that too, is what you recognize is, and another analogy to kind of stretch it as far as I can get, is also, it is, to your point, the complexity of what a Formula One race looks like in terms when you start to add in all of the components and all of the layers, the material, like everything from the drivers, the engineers, the pit crew, the support staff, the, you know, the teams that are on track, all this kind of stuff, the materials engineers and working out the rubber and all, like it is insanely complex. And yet they do it, you know, almost they do it like, you know, 25 weekends a year or whatever routinely. Right. Yep. At a far, even though a very expensive, a far less budget than any one of those AI teams. Yes. And so, you know, hey, let's treat life more like a Formula One. Right.
Andrew: Let's see whether we get some flack for that.
Sean: Oh, that's okay. Bring it. It's fine. And if you want to give us flack, a great way to do that. There's a transition. So leave us a rating and review on anywhere you get your podcasts and look at that. But of course, if you don't want to sign up the old-fashioned way, you can just go to a website.
Andrew: Go to modemfutura.com/signup and you can just sign up for a weekly reminder that there's a new episode to listen to. Nothing more, nothing less. That's right. That way you never,
Sean: just like a Formula One season, you don't ever want to miss that. You don't ever want to miss the race. So why would you ever want to miss a Modem Futura episode? I don't know. But yeah, and I think, and another thing you can do, again, speaking of AI tools, is you can throw your AI tool over to beinghuman.fyi and explore and ask. There's lots of prompts to get you started, but also leveraging it to explore all kinds of things.
Andrew: And here is a little trick with that. So beinghuman.fyi actually has access to all of the transcripts of these podcast episodes. So you can actually ask it to do a deep dive into a specific episode.
Sean: Yeah, or maybe one of the hosts should have used it to say, when was the last time I talked about this? Right, right. And actually had that information at his hands. Maybe in the future, Sean will be smarter about it. But yeah, no, that's awesome. And I think, you know, like I said, I think this is gonna be the true crime story that will just continue to give us more. I think it will be. But so until then, I guess that we'll call it for here. And it's been a great episode, Andrew. So thank you for all the conversation and the insight and all that as usual. And yeah, we'll see everybody on the next episode of Modem Futura. We'll see you then.
Announcer: Demodulating signal. Modem Futura is a production of the Future of Being Human initiative at Arizona State University. Be sure to subscribe on Apple Podcasts, Spotify, or wherever you listen to your favorite shows. To learn more about the Future of Being Human initiative and all of our other projects, please visit us on the web at futureofbeinghuman.asu.edu.