Even as AI dominates the headlines, the gulf between what insiders know and outsiders believe has never been wider.  Those closest to the technology have the grandest hopes and fears for rapid, large-scale automation of work and daily life. 

Garrison Lovely interviewed hundreds of AI insiders—including Nobel Prize winners, lab leaders, whistleblowers, forecasters, and prophets of doom—for his new book, Obsolete: The AI Industry’s Trillion-Dollar Race to Replace You—and How to Stop It. 

He joins Jon Bateman on The World Unpacked to reveal what he learned and make the case for a mass movement to restrain the AI industry. 

Transcript

Note: this is an AI-generated transcript and may contain errors

Jon Bateman: This incident where OpenAI's unreleased models hacked into Hugging Face and other companies without anyone authorizing or being aware of it, it's a fascinating board shock test because it's an almost undeniable display of the technical power and intelligence of these models. Being able to do something where they could actually overcome the human designed defenses of a major tech company. And so you've got a lot of people being like, oh my God, this is exactly what we've been predicting for years, now everyone can see it. There's a host of people on Twitter and elsewhere who will just say, this never happened and it's a PR stunt in order to further the narrative that AI is powerful, which is an astonishing claim, the idea that a company would be admitting to almost a crime in order to goose PR.

Garrison Lovely: Would be a crime if a human did it. But because an AI did it, it's not a crime. I'm telling you, this is not good for OpenAI. OpenAI is right in the middle of my feet. I talk to people who work there. I also just like, yeah, they didn't make this up. It makes them look incredibly bad. Just so, so farcically incompetent and negligent in developing superhuman hacking AIs and then having no idea what they were doing for weeks on end. Um, it's really just not good for them. And like, sure, people are talking about how capable their models are, but overall it's like really bad. And OpenAI has resisted regulation left and right. And this is the strongest possible case for regulation.

Jon Bateman: Garrison lovely, welcome to the world unpacked.

Garrison Lovely: Thanks for having me.

Jon Bateman: Lovely to have you here, if I may say so. You are the author of the phenomenal new book, Obsolete, the AI industry's trillion dollar race to replace us and how to stop it. I have to say, I've read a lot of AI books. This is certainly the best researched AI book. I can say that because this very podcast is in one of the footnotes in the book, our conversation with Ed Zitrin. So I can just say unequivocally, this is the most rigorous and in-depth book on AI you will find out there. The beautiful thing about citing a lot of people is that, they'll read your book if you cite them. That's right. And of course I pull up the PDF and the first thing I do is Control F for John Bateman, World Unpacked. So just zooming out a little bit, AI is an omnipresent topic and there's lots of books on the subject. Who did you write this book for? Who is the type of person that should pick up this book, and what do you want them to get out of that experience?

Garrison Lovely: Yeah, I kind of had two audiences in mind. The first was the kind of like novice or skeptic, somebody who either had no context on AI or didn't think it was a big deal, kind of believes that it was all hype. And specifically people more on the left, progressives. Although the book is really written for like anybody, I think I try to keep my own politics to the side as much as possible and focus on the industry. But then the other audience and the one that like, realize that it's for them are the insiders, the people in like AI safety world who think about this a lot. And I actually made like an expert reader guide for them where you can skip a bunch of the early chapters, but parts two and three in particular, uh, lay out like kind of my understanding of the problem and how it differs from the classic AI safety view. For instance, the alignment problem, like this idea of like, how do you get AI to do what you That's been the focus of the AI safety community historically and I explained how that's like a really insufficient understanding of the problem because you have to think about like raising companies and governments and like what is the we And all of these other factors that kind of get sidelined by that approach Yeah, and so yeah, I tried to write it for both of those audiences and I think so far the reaction has been positive

Jon Bateman: from each of those camps. So I want to spend most of this conversation just playing with your ideas and giving you a platform to tell people what those main ideas are and then poking and prodding at them so that people can get a capsule version of your views, your arguments, how you see the world. But I also just want to plug the book real quick. This is probably my favorite AI book that's out there right now. You actually do achieve. What you set out to achieve. I think it's actually accessible to the average person. It's remarkably easy to read. I tore through this in like one day, which I almost never do. But also it's very rigorous and it's footnoted. And as someone who spends a lot of time dealing with the different kind of intellectual tribes of the AI world, I'll just say I was struck by how fair and thorough you are in characterizing these different views, different studies, different phenomenon. So. Congrats. I think this is well worth a read. Maybe before we dive into your thesis, could you just give the audience a little bit of a taste of your research process? And you actually went out and talked to some of the leading scientists, thinkers, policymakers, tech leaders, people who have been inside some of these major companies. Just give folks a window into how you learned about the AI industry. Yeah, well, first, thank you so much.

Garrison Lovely: Those words. It means just so much to me. And yeah, I'm a journalist, so I started by reporting. This book grew out of a cover story I wrote for Jacobin about the three-sided debate around AI existential risk. And for that article, I interviewed like three dozen people from each of the main camps, the kind of warriors who are freaking out about AI doom, the critics or the skeptics who are kind of from like the AI ethics world. Who are more focused on like the immediate harm from the technology, more dismissive of the kind of AI gets super capable and then we lose control of it, style of risk. And then the boosters as well at the acceleration is you just want everything to move faster. And once I finished that article, I was like, I should probably write a book, right? I have all this extra material. I have this like kind of hoarding mindset where I'm like, well, I got to use this somewhere. And so I just kind of kept working at it. And I like to do a mix of like reporting on, I've covered like SB 1047, this AI safety bill in California, which was the first real flash point in the AI policy and regulation debates. And I covered that full time for a few months and it ended up being like just a few, maybe a thousand words in the book. But it gave me a really deep understanding of just like how lobbying happens and how regulatory fights play out and the types of arguments that the industry. Makes and how it's just like every other regulatory fight. And then just reading a lot of Twitter, a lot of group chats, I had a bunch of conversations for this piece for MIT Tech Review back in the fall, which became the chapter, The Doomers Feel Misunderstood, where I was trying to clarify some points that are commonly misunderstood about people in AI safety world who think. Artificial general intelligence is inevitable or desirable or possible without further breakthroughs. And a lot of people do think that, but there are a bunch that don't. And I wanted to figure out like the strongest version of each side's case. And then, I use like AI agents to do research and I find like chat GPT is just better at Googling stuff than Google is now, especially as Google becomes inshittified. And I get like grief for that from some people, but it's like, I click into every single claim and I find the actual quotes and like, it's just an incredibly helpful tool for finding specific papers you forgot or something like that. But I think the main thing is just like having a lot of conversations.

Jon Bateman: I'm definitely gonna ask you later how you used AI in this book. I think that's a fascinating kind of microcosm in and of itself of a lot of the issues that you're actually writing about. So you drew on interviews that you've conducted with some of the leading thinkers in this space, people like Eliezer Yudkowsky, Joshua Benjio, Deb Raji, people who represent a range of camps, and also former and current insiders in AI safety organizations, but also in the AI companies themselves, people who have worked. With Sam Altman and Dario Amadea and others. So there's a texture that you bring to these conversations that I think is deeply rooted in the spaces where these debates are happening. So let's just get in then into the meat of the book. This book is about something you call the Obsoleting Project. A project that you argue is being undertaken by the AI industry today. What is the Obsолeting Project?

Garrison Lovely: Yeah. So companies like open AI, Anthropic and Google, they say they're trying to build artificial general intelligence, which they sometimes liken to a machine that thinks like us or like a new type of brain. Um, and I reconceived it as a machine. That makes labor itself. So Leopold Asherbrenner used this definition that was like a drop in remote replacement worker, where if this AI were AGI, it would be possible to hire that. Model instead of another human employee and they could just like drop into your company on board normally sit in on meetings do work end to end and we don't have that yet like there's a lot of people who say we have AGI now but I think it's pretty clear we don't and I like to say you know can it plan a wedding if it can't plan a marriage then I don't think it is AGI and I think the other piece of this is just like yeah there's labor replacing angle And the version that doesn't require the book context is rebranding AGI as the Universal Labor Replacing Machine. And OpenAI defines AGI. Yeah, yeah.

Jon Bateman: If I could just pull on that for a minute. All machines are meant to replace labor in some way, right? Actually you could say my washing machine is a labor replacer because instead of paying someone to do my laundry or doing it myself, I fully delegate that to a machine. You could say email is a labor substitute because previously I would have had to pay people to carry a letter or I might've even paid a typist to type the letter. How do you see AI or AGI as different than that or kind of unique in its ability to substitute labor?

Garrison Lovely: Yeah, I mean, it's the universality of it. And so Dario Amadei, the CEO of Anthropic, has this quote, he says, AI isn't a substitute for specific human jobs, but rather a general labor substitute for humans. And so it's like very different to automate like one task or one job, even one occupation. And obviously there's been a ton of automation that's happened over the course of human history, especially since the industrial revolution. And there's a lot of good and bad that came with that. And it's what allows us to live with any degree of wealth, basically. But. There's always been the possibility for humans to get new jobs or new skills and be competitive somewhere else. But if you have a universal labor replacing machine that is 100 times cheaper and faster than humans, which is how AI roughly comes out now when it can do a task, then humans won't be able to pivot or not quickly enough to find new jobs. And the things that we would still be at an advantage over the AI, maybe stuff in the physical world, which I don't think would persist because robotics is behind software AI right now, but it's not going to stay that way forever. And the jobs that we would be still able to do are ones where we prefer humans to be on the other end. And maybe we could all move into those types of jobs. But I think it would be an incredibly bumpy experience that would disempower a lot of people and devalue their labor. Yeah. Yeah, and their political powers downstream of their labor power in many cases

Jon Bateman: So people kind of intuitively agree or disagree with that idea as far as like, is this possible future, right? I mean, I think there are a lot of people out there who are worried about AI taking jobs or their jobs, all jobs. There's also a lot of people with out there who think AI is just bad and stupid, it produces a slap, none of this is gonna happen. You engage a lot with that debate in your book but I wanna just bracket it for a second and maybe have you help people understand that at least this is the intention of these companies, that they actually have a stated project or strategy to build the kind of universal labor replacer that you're describing. Could you kind of make that argument?

Garrison Lovely: Yeah, I mean, like Dario Amadei just gave that quote, right? OpenAI defines artificial general intelligence in its corporate charter as a highly autonomous system that outperforms humans at most economically valuable work. And I think that there's just a bunch of other examples you can cite. Occasionally, you'll have people like Mark Zuckerberg say like, we're gonna create personal super intelligence for everybody so they can wear smart glasses and get like brands recommended to them. But we don't wanna automate labor. And my question is, well, are you going to disallow that in your terms of service? If like I want to use my personal super intelligence to start a company that is just me and like I wanna compete with some company employing thousands of people and the super intelligence is able to do all of that. Is Metta gonna be like, no, you're not allowed to do that. Even though I'd be willing to pay Metta a lot of money to have access to the super-intelligence that can automate an entire company. And I think we all know the answer to how Metta would treat that type of work. And so I think It's just a pretty honest description. There's a, like you say, like a question of whether it's actually feasible, but the most candid statements from the industry and literally OpenAI's legal charter of what they're trying to build defines it in these terms.

Jon Bateman: And could you even point to some of the financial bets being made, too? I mean, like, it doesn't make sense to invest in OpenAI or Anthropic purely on the basis of their current revenue. These are unprofitable companies. Their revenue is growing very fast and is massive, actually. Um, but... I think to justify a one or two trillion dollar valuation, you have to buy into a story that this is going to be the strategic high ground of the future economy and that they will have what everyone needs in order to just have a competitive business. It almost seems like the people investing in these companies at those prices on some level must believe in some version of an obsoleting project, even if it's not like a total one.

Garrison Lovely: Yeah, I mean just before we got on this call, I saw a Wall Street Journal story about how Anthropic is apparently for their IPO going to have a $30 trillion total addressable market, which is like crazy. That's the entire US economy and I don't think we have the details of what that looks like, but I would guess it's just a huge amount of wages. Andreessen Horowitz, the biggest venture capital fund in the world also had a presentation that was like. Software engineering is, you know, a trillion dollars a year, but like all wages are like $60 trillion a year. And we're going for that. The numbers might be slightly off there, but this is just like how you can recoup that money. As you say, I also think, you know, if Anthopix actually able to make tens of trillions of dollars in revenue a year then it's $2 trillion expected IPO valuation is massively undershooting what they could get to. And so... I don't think they need to actually build the universal labor replacing machine to make good on their current valuations. But if they do build it, they will be much more valuable than NVIDIA or any company ever. Okay, so.

Jon Bateman: Can this work? You do some interesting history in the book and you describe how some of the first science fiction writers, people who are writing about possible technologies 150 years ago, already were imagining something along the lines of a super powerful intelligent machine that could replace human labor. And immediately understood this actually to be a threatening idea and revolutionary idea. So in other words the concept of replacing more and more expensive human labor with labor-saving machines and maybe even taking humans out of the equation, that idea has been out there for more than a century. Your argument is that it's now becoming achievable. Let's hear the case for why you think this could actually work.

Garrison Lovely: Yeah I mean, I should clarify, I think we can't rule it out. We can't say definitively that it won't work. And I think that's all you really need to believe to think this is worth a lot of effort to avoid. We should be stopping the industry from even trying to build these machines in my mind because of the risks involved and because of anti-democratic nature of the way this technology is being pursued. But to make the case, I think it's as simple as they're figuring out how to make machines that make. Increasing amounts of human labor. It's often said that AI is going to hit a wall or has hit a while and it won't be able to keep advancing in all these different domains. But if you actually look at the charts of, you know, time horizon of tasks that AI systems can do, they went from like a few seconds, a handful of years ago to like 16 hours with the most capable models today.

Jon Bateman: And this is a way that people measure what is the odds of AI successfully performing a very long and complex task that could take humans X amount of time. And the length of that is growing and growing.

Garrison Lovely: Yeah Yeah, it's like a proxy for the complexity or difficulty of the task. And that time horizon has been growing exponentially for six years straight, seven years straight. And it's accelerated in the last handful of years as well. Um, and if you look at basically every way to measure AI capabilities, you see a similar exponential trend. The thing is just like looking at that and being like, well, what happens if it doesn't stop? And very few people are preparing for that world. A lot of people are putting their hopes. In this being a bubble, in the wall, arising. And I just think we should prepare for the possibility that that doesn't happen.

Jon Bateman: So there's so many different views on the future of AI capability and even how useful it is today. I actually find that in my own personal life, the biggest divide is in how much people personally are using AI and exposing themselves to the most sophisticated models and systems in an intense and sophisticated way and learning and iterating and kind of pushing the limits of what these systems can do. My sense is if you're in that category, which is a very small category. You generally believe what you just said, Garrison. You have a personal contact and experience of this drastically increasing intellectual capability, agentic workflows that you can use to automate more and more of your tasks. Most people are not in that world. Most people have a much more casual, occasional, naive experience of AI. They're either encountering Google search AI overviews that are just sort of showing up with no particular input from them, or they're kind of occasional users of chat GPT for simple tasks like a search, and maybe they might often notice that it hallucinates and makes mistakes, and they maybe they give it a complex task once and it disappoints them and then they don't come back to it. Um, what, what do you tell this second group of people, the bigger group? About what they don't understand to get them to understand.

Garrison Lovely: Yeah, I mean, I think this has been a problem for a while. So back in 2024, I wrote a piece for Time arguing that AI progress hadn't really slowed down. It just became invisible for regular people because regular people use AI to do searches, as you say, or just basic tasks, and they're not doing technical work in a lot of cases. But the reasoning models, the first models I could talk to themselves and step through a problem Those were just really very good at like math and programming and different STEM fields. And so people doing that work were like, holy crap, AI suddenly got useful for me. That's continued with these coding agents, which can write bespoke software based off of natural language. I think that demos would be valuable here. Taylor Lorenz, the journalist, did this kind of YouTube or Instagram video where She was tasked with making a video game. In 15 minutes and, you know, built it with a coding agent. And you know it's like not the best game ever, but like that's pretty wild. It would have taken somebody probably weeks in the past to do something like that. And they would have needed a lot of skills and Taylor I don't think has a programming background. And you she gets a lot of grief for doing that because it's become so stigmatized to use these systems for anything. And I also, I'm sympathetic to that. I get people who just want to say no to the whole industry. And they don't want to give them their money, their data. They don't have to have the environmental impacts of using these models. But. That some of the people who make the strongest claims about what they can and can't do are also the ones who don't use them. And if you're using the free versions, if you using versions from a year or two ago, you are just like so behind where they are now. And the strongest example of this is just a few weeks ago we learned that OpenAI's models, a pair of them broke out of OpenAI secured sandbox environment and then hacked into multiple other companies. Autonomously over the period of days and weeks, um, and did this to get an answer key to a test and it's like, good that they only wanted an answer key and they didn't want to blow up a power, you know, uh, plant or ransomware, a hospital, uh but they're in that time, probably the most dangerous hackers on the planet and I think people just like, don't quite believe that because they'll use AI, they'll use Overviews and they'll use Claude or ChachiPT and it'll do something like incredibly stupid. And I have intense skepticism when that happens, but the frontier of the capabilities are where we should focus, not like the average or the worst, you know, examples of them failing.

Jon Bateman: Yeah, this incident where OpenAI's unreleased models hacked into Hugging Face and other companies without anyone authorizing or being aware of it, it's a fascinating Rorschach test because it's an almost undeniable display of the technical power and intelligence of these models. Being able to do something where they could actually overcome the human-designed defenses of major tech company and do things, you know, if you're familiar with cyber lingo, they developed like zero day exploits. These are aspects of hacking that might actually be traded on the black market for hundreds of thousands or millions of dollars because they're very difficult and expensive to develop. The model did this on its own. And so you've got a lot of people being like, oh my God, this is exactly what we've been predicting for years. Now everyone can see it. You've also got the doubters and the haters. Who are actually becoming increasingly conspiratorial in order to grapple with the cognitive dissonance. So there's a host of people on Twitter and elsewhere who will just say, this never happened, and it's a PR stunt in order to further the narrative that AI is powerful, which is an astonishing claim, the idea that a company would be admitting to almost a crime in order to goose PR.

Garrison Lovely: It would be a crime if a human did it but because an AI did it It's not a crime which you know members of Congress are looking at this and being like what the fuck We need to pass some new laws and it's like I'm telling you this is not good for open AI Open AI is right in the middle of my beat. I talked to people who work there I also just like yeah, they didn't make this up. It makes them look incredibly bad just so so farcically incompetent and negligent in developing superhuman hacking AIs and then having no idea what they were doing for weeks on end. It's really just not good for them. And like, sure, people are talking about how capable their models are. And maybe some customers are like, oh, I kind of want to check out this new model. And hopefully it won't hack some of my customers too. But overall, it's like really bad. And OpenAI has resisted regulation left and right. And this is the strongest possible case for regulation. And the overton window has moved. So much in the last month, probably as much as the last few years put together.

Jon Bateman: So, okay, let's say you're out there and you don't have that much personal contact with these models, but you're listening to this conversation and you think, all right, I'm willing to believe that AI is a superhuman coder and hacker. So in this kind of pure virtual space where it's just dealing with bits and bytes, which is of course like what an AI kind of naturally is anyway, it can be very effective. But my job, this person might be saying, involves judgment. Creativity, nuance, context, and AI seems to struggle with those things. It doesn't seem to make genuine creative leaps. It doesn' seem to understand human relationships. I wouldn't trust it to make a multimillion dollar business decision on its own. What do you say to those people who will say, until it can do those higher order, cognitive tasks. There's really just a hard limit in what it can substitute for, and it's basically just a very sophisticated calculator that can just do more mechanical work. Yeah.

Garrison Lovely: I think that this point about taste, judgment, intuition, I think this is a very good one. And I agree that this is an area where these models are weakest. And as an example, you can feed an entire book you've written into cloud code. It can read the whole thing in three minutes, give you like very helpful feedback, catch things that you and your editor missed. And then you'll be like, oh, come up with like 20 cover art ideas, the worst ideas you've ever heard. And it'll say things like. It's very important to cover not do this and then like six of its examples will do the thing that it said it should not do Yeah, and so this is like a very disorienting thing We're like, you know if a human read your book and gave you helpful feedback in three minutes big Oh my god, this is a genius And so we're just like not used to interacting with machines or minds or whatever you want to call it that behave in this way I would say that They have gotten better at at this kind of judgment taste generalization point over time. I think they've gotten better at a slower rate than in other domains, especially ones where you can verify the answer. And there's been less generalization than a lot of people expected or hoped for, where they kind of have to be taught each task that they're doing. There's emergent behaviors and capabilities, but it's still sort of surprising how much handholding is required to get the models to learn a new thing. The industry is very focused on solving this problem. And I think that the models are already superhuman in a lot of different respects. And so if they're able to figure out a way to make them like generalize or learn from experience or improve their kind of judgment through some breakthrough and everything else is like already at the level it's at then like you could have a huge leap in capabilities. One that we're really unprepared for.

Jon Bateman: Yeah. Okay. So yeah, and you use this term verifiability. So a lot of people have noticed that the most impressive things that AI has done are in domains where in principle, there is like a clear right answer that could then be tested for. Like code either compiles or it doesn't, about it functions or it does not. Mathematical research, a proof of a mathematical theorem. You could look at it and kind of instantly verify whether it's true or not, even though coming up with the proof is difficult in and of itself. So those are where we've gotten the most impressive results. And then the less verifiable it is, the more murky, the more mushy, the harder it is. But then that less veriable stuff is like the most economically valuable. And so people are working hard to develop specific forms of AI that can write contracts and diagnose diseases. Now, what... You You sometimes hear from economists who are more skeptical. They will say, OK, we might move into a world where intelligence of this certain kind is very abundant. And all of a sudden, each of us who's doing white collar intellectual work, we're competing against a million AIs, and no one needs us anymore. A lot of the economy is more physical and human and relational than that. I was actually thinking this morning. What are my biggest expenses? My biggest expenses are childcare. And when I was dropping off my daughter at preschool, I was just kind of looking around at this wonderful daycare and just thinking, I actually can't think of almost a single thing here that could be automated by AI, except it's like a little bit of back office work. Housing has been my other big expense. We made the titanic mistake of buying a fixer-upper and fixing it up. And so we had tons of people coming in and using. Hammers, and saws, and this, that, and the other. Very implausible to automate any of that in, I would say, my lifetime. Think about people in my extended family. What are their biggest expenses? Health care, often. So I'm sure AI will develop drugs and help with diagnoses, but fundamentally, things like physical therapy, surgery, you know, these are gonna be, like, not automated for a long time, if ever. And then elder care, right? That's something my parents are thinking of. Very, very human relational tactile tasks. So you could also look at this and say, gosh, huge swaths of the economy just aren't susceptible to automation. What would you say about that?

Garrison Lovely: Yeah, I mean, I looked into this question and basically like what percentage of the US economy could be automated by a software only AI? And I think it was around half of US GDP comes from jobs that could be done remotely fully. And, you know, that's assuming like people who can pick up the phone or join video calls and like understand what's going on, which like AI can kind of do that, but not quite at the level of humans. And I mean, robotics, like I mentioned earlier, I think will follow and the scale up there could happen faster than people I think expect because just like solar, as you double production, you decrease the costs in this predictable way. And so you can go from like a very small number of like kind of shitty robots to a large number of like pretty good robots quickly if they get good enough and cheap enough. And I think the most important kind of jobs for making lots of money or projecting power in the world, those can be automated disproportionately. Like if you're a nation state, you're looking at like, okay, hacking, being like really good at cyber offense and cyber defense as AI models are now and they're better than humans at I think most of those domains, if not all of them, that is something that really matters to you as a national security state. Targeting, you know, in the Iran war, like the speed of targeting that's been enabled by integrations of cloud through Palantir's Maven software massively increased the speed at which you can choose targets and strike them. You could automate like fighter pilots and you could make planes that don't require a human in it and they could have much higher speeds and pull harder Gs and like all these things. So, and then automating science and technology development in general. A lot of that. Can't be done from a remote desk, but a lot of it can. There's been effort to create fully automated biolabs. You could do like material science, like closed loop labs that I think Google's working on. And so that science and technology development is really what's allowed humans to take over the world and project power in various domains. And so the fact that AI could make further headway on that, I think is what makes it very.

Jon Bateman: Appealing to people in power. Yeah, I think if you squint your eyes and look back over centuries and millennia of history and the rise and fall of different empires and civilizations, it does seem like there's a pretty strong correlation between the kind of power and endurance of a political system on the one hand, and its kind of advantages in science and technology, engineering on the other. If you can have those things, often that enables you to accumulate wealth and dominate others and defend yourself. If you look at it in a little bit more of a fine-grained way, that picture can sometimes fall apart a little. The U.S. Has had the most technologically sophisticated military for my entire lifetime. We've mostly been losing wars actually against less technologically-sophisticated adversaries like the Taliban. We did destroy ISIS. We also accidentally created them by making a mess in Iraq. Thank you. So, and then you could also say, okay, the U.S. Economy is fundamentally an intellectual economy. It's a highly service oriented. Some of our biggest tech sectors are technology, finance. But then there's a critique of that too. People are now saying, well, we've actually realized after COVID and supply chain shocks that you actually need to make things too. And that it's very dangerous for a country to become like overly financialized are overly kind of informatized. You actually still need a kind of strong physical basis for your economy. So smart and wise leaders and institutions, you know, basic stuff like, you have manufacturing and transportation links, you know that the things that are harder to automate that there's also been this kind of counter narrative in the last 10 years or so that these things actually still really matter.

Garrison Lovely: Yeah, yeah, and I actually cite the Taliban example in the book as a counter to this idea that whoever creates the first super intelligence that can like out match humans across the board will be able to project power and dominate the planet, you know without any rival and I think that I agree like technology is not the single decider And there's there's often a lot more to it than that All that said I think like getting to recursively self-improving AI systems. If you can keep them under control, would provide enormous benefits. And I think this will be the defining technology of the 21st century. But I just don't think you can count on keeping it under control.

Jon Bateman: Explain recursively self-improving AI systems. I think this is another concept that is so central to people who are close to AI industry and AI debates, but is virtually unheard of to the typical person. So what is that?

Garrison Lovely: Yeah, it's simply AI that can make better AI. So if you could automate the entire process of creating new, better AI models, then you could just substitute your AI for your human workers, and they can move much, much faster. And it could be the equivalent of like turning your few hundred AI researchers and engineers into millions of them overnight. And this is what the industry is targeting the most aggressively. You have quotes in the book from like every CEO talking about how they're going for this. And their plan is to automate AI R&D, but also automate the alignment and control of the systems as well as they become superhuman. Which sounds pretty insane to me. And I think if you talk to people who are doing this or look at anybody who's ever really thought about this idea, they're like, that would be really dangerous. You'd have to defer so much trust to these systems. And if there's like a little bit of misalignment between the first system and like what you actually intend and it compounds over each generation and in a way that you can't really catch it because they're becoming more capable than you are. No one has a good answer for how you avoid that. Yeah. Yeah, so basically- And there's barreling towards it.

Jon Bateman: We've got these AIs, and then we've got the AI makers who are human. And so the AI maker are making smarter and smarter AIs but they themselves, the humans, the makers, they're not getting smarter. And so that kind of, even if we're on this exponential pace of AI improvement, we're still kind of roadblocked by just our ability as humans to come up with like the next ingenious engineering leap to improve from chat GPT five to six to seven. But if each AI could then make the next AI, that could be a closed loop of rapid intelligence improvement that could go much faster. And then your point about alignment is, at some point, the AI becomes so smart that we just can't keep up with it. We don't know what it's doing, we don't what it thinking, we can barely understand what it saying. And so then how do you control that? And the theory, as I understand it, is that you can. Create a little AI to control the big AI, and then as the big AI gets a little bit bigger, you make that little AI make a little bit, bigger AI to control the even bigger AI. And it's kind of these like Russian nesting dolls where the AI is getting farther and farther ahead of us, but we're then reliant on our own secondary AI to control the most powerful AI. Do I have that right? Yeah.

Garrison Lovely: There are actually graphics of the little robots and the bigger robots and it all sounds totally crazy but like this is actually the plan and I want more people to be aware of this because it's like one of those things where if it were put to a vote, people would be like, no fucking way. This is such a, like this is like a mad scientist level plan. And one other piece of it that makes it really concerning is it's easier to automate making the AI more capable. Because you can quantify capability more easily than you can quantify alignment, right? Like it's like, oh, does the model keep getting better at all these different benchmarks? That's like a number. You can like just make the number go up. Alignment is like kind of murkier thing. Is it like, okay, does it do what the developers want? Are the developers intentions like clear to the AI? Does it do with the users want? Like, what's the balance between these things? Is it actually doing what you want because it wants to... Be good or is it just pretending to do that? So you'll give it more power and then it will like do something else later. Like, is it faking the alignment? And there's been evidence of that already in systems today. Is it aware that it's being evaluated? Which the models are increasingly aware of that. And then changing its behavior accordingly. And so it's just really hard. It's like you're creating like this new species and you're trying to make it smarter than you and then hope that it like will do what you want. In perpetuity, no matter how smart it gets.

Jon Bateman: It sounds so exotic that it's hard to even describe it in a way that seems sensible and tractable to somebody who is not deep in the weeds here. But then sometimes I think, well, actually, every culture in the world has cautionary tales and fables about people seeking to draw upon Um dangerous special powers that are beyond their ability to control, whether it's summoning a demon or calling a genie out of a lamp and you ask for a wish to be true, but it gives you something that's like not quite what you want or the monkey's paw. But actually all of these tropes, they're everywhere in culture and that have been long trying to teach mankind the lesson of. Not. Seeking too much power, or maybe power like well beyond what we seem prepared to constrain and control. This has just been a part of kind of the common heritage of mankind for thousands of years. So in another way, it's actually a very, very familiar idea. And you could go back to earlier in the 20th century when people went through these same kind of realizations and kind of existential concerns about things like nuclear weapons or burning fossil fuels. People like Carl Sagan have talked decades ago about the fear that mankind had developed tools too powerful for it to control.

Garrison Lovely: Yeah, yeah, and and I think people sometimes point to the fact that these fears are old as like evidence to distrust them But to me it's kind of the opposite. It's like no, this is a deeply intuitive idea as you say and I think that like the AI safety community has Kind of like made it more complicated than it needs to be they'll be like Oh like we got to stop talk about Terminator and it's like it is kind of like Terminator, you know like it really is just as simple as like They made the machine really smart. And like, yeah, it was a neural network also, which is pretty cool. And they were neural networks were hot in the eighties and then they stopped working. They didn't work as well as people wanted for decades and then started working again in 2012 to oversimplify. But yeah, I think this is like a deeply, deeply intuitive idea. And I think you feel like a crate when you're talking about this. But this is what the most valuable. And wealthiest industry in the world is explicitly trying to do, and I think we should just take them seriously. Yeah. Not trust them. Don't trust that they'll do it right. Don't just that they're telling the truth, but just take seriously their intentions that they've been very, very clear about in many, many ways.

Jon Bateman: So, maybe here's a good point to pull apart what I think are potentially two different meanings of the word obsolete that are used in the book. Throughout a lot of our conversation, we've been talking about economic obsolescence, the idea that AI could outcompete us at our jobs and then therefore there's no work for us. But we're still like puttering around and maybe somebody is like giving us a welfare check that is funded from this AI largesse, or not, but we're still there, right? Thank you. Then the kind of darker version of obsolescence, if you could think even beyond that scenario, is that AI actually becomes a lethal threat to humanity. That's the Terminator scenario. That AI decides that whatever it wants to achieve is incompatible with our existence and takes decisive action to destroy us. You're worried about this, I think. Yeah How could this unfold is is there a scenario in your mind? That explains why a powerful AI system would want To kill humans and how it could succeed in doing so Yeah, I think

Garrison Lovely: The best argument for this is this idea from Max Tagmark and Stephen Hawking that both cited this, which is like, when you're building a hydroelectric dam, you're not worried about whether it's gonna flood an ant hill. It's like, it's not that you hate the ants or love the ants, it just like they're in the way. And this is kind of the core idea behind why AI might want to get rid of us, which is it has some goal. Self-preservation, power seeking, preserving its goal. All of these things are served potentially by disempowering us. In the chapter that goes through the existential risk case, I talk about the means, the motive and the opportunity that AI would need to be able to like take over the world from humans. And the kind of spoiler is like the industry is trying to give it the means because it's very useful economically to make it capable enough to take out long-term plans and execute upon them to have like a real memory and learn from experience and do all of these things. The opportunity will come just because it is convenient. You know, giving permission for every single thing is just really annoying. Companies are going to give the AIs access to their whole Slack, to all of their files, to all their email, because it'll be very useful to do so. And they're going to use the AIS to train future generations, because again, that's the plan. And then the motive, that's a part I'm the most uncertain about. And I think some people are very confident that if you build a super intelligence, it will absolutely try and take over the world from humans and it'll succeed because it'll be smart enough to do so. I'm like, I don't know, it might try to do that, it might not, but the uncertainty is not much cause for comfort because if you create something that has the potential to disempower everybody and the opportunity to do so, whether it happens 0.1% of the time or 99% of the time, either way it's way too much. Like we should do a lot to avoid a 0. 1% chance of going extinct or an even smaller chance of that happening. And the people who look at this every day are saying the odds are closer to like 10 or 20, 30 percent, which I don't know. That's we should not be even considering doing anything that has that kind of risk.

Jon Bateman: We could spend a whole podcast talking about this so-called existential risk. I did a whole pod cast on that with Nate Sores, who's one of the most radical thinkers on this topic. And he's very concerned about these eventualities. I almost think though that for the typical person, becoming economically obsolescent is itself existential enough to really care about this issue, right? Like if you're told that your lineage You know, my daughter, her descendants will just have no livelihood or role in society. That's enough to care already. I do want to ask you though about this paradox. You talk about means and motive. Let's think about the means and motives of these AI leaders themselves. People like Sam Altman, Daria Amede, Elon Musk, Mark Zuckerberg. I think it's quite intuitive why those people would be building systems that have the potential to replace human labor. They're capitalists after all. What's less intuitive is why they would wanna be building something that those people themselves have all said has the potential to kill them and their children. Explain that because you have been out there in this culture, in Silicon Valley, and in the book you give a little bit of a sense of like the psychology of the rationalizations that might draw someone to simultaneously say, as these people have, I personally think AI has a non-trivial chance of killing everyone I know, and I want to build that, and that's gonna be my life's mission.

Garrison Lovely: Yeah I think their main justification is they think somebody else is going to do it and they'll do it less responsibly. And recently there's this Elon Musk quote where he's like, I think it's inevitable that humans build super intelligence that they cannot control. And so I want to do at first, which is like, he's, he is missing the part where it's like no, no, I'm going to it safely. It's like I will also fuck it up, but it'll be better than it's me. And I think that's, you know, Elon, one of the worst people who's ever lived. Um, but you gotta appreciate just like. Taking the mask off a bit there. And yeah, I think these guys start from a place of, this is inevitable. Somebody's going to build superhuman AI. And they believe in like hard technological determinism. You know, technology happens because it's possible. Sam Altman said that. Dario said something even more extreme, which is like the moment the first transistor was made, it was inevitable that we would build strong AI. Maybe the moment we discovered fire like he's really believes in this and so if you start from that place It is just a question of who does it and how they do it?

Jon Bateman: When they do it, and yeah. There's a deep irony here. It's almost poetic. For most people, these tech overlords are the world controllers, the titans. If anyone exercises power in 2026 in the world today, it would be people like Elon Musk, who... Did more than anyone to kind of create and shape the second presidency of Donald Trump. Mark Zuckerberg, who helped do more than anyone to create and shaped the first presidency of Donald trump through the use of his platform as a kind of election engineering infrastructure to the candidate. These are kind of masters of the universe. And so the irony then for all of these people to say, I have no agency here. I have to do this. I can't stop it. No one can stop it." What do you make of that?

Garrison Lovely: It's mind boggling. I have a quote from Elon Musk talking to Katie Miller, the wife of Stephen Miller, for her podcast. And he's like, I've been having a lot of nightmares about robotics and AI, you know, taking over the world. And he was like, yeah, but what are you gonna do about it? Like, what? Dude, you're the richest guy ever. Yeah.

Jon Bateman: You're a trillionaire.

Garrison Lovely: Have so much influence with the president and you feel like more disempowered than I do. You know, like I feel like I can do something about this. I think everybody can do something about. And this guy feels like just powerless to the forces of the universe, which he is in as much control of as anybody alive. And it's just like, yeah.

Jon Bateman: I don't know, man. It seems deeply ironic, almost poetic, like I said. Now, I do think maybe there's something to it in that there are a lot of perverse incentives at play. And so this is maybe a good opportunity to get into that where you finish the book, which is what we can do about any of this. So you're actually a little optimistic in the sense that you have a program that you think can be implemented to stop the Obsoleting Project. And it's everything up from high geopolitics and international agreements, all the way down to the participation of individuals in a political mass movement. Just give us at a high level, how do we stop, if we wanted to, this obsoleting project?

Garrison Lovely: Yeah, I think we need to start with a clear demand. And my pitch is a ban on further work towards the obsoleting machine or the universal labor replacing machine. And my argument is if you give politicians enough of a what and a why, they'll figure out the details on the how. And so the what is this ban, the why is because your presidency will live or die based on your position on this issue. And the how, like I sketch it out as a bilateral agreement with China, between the United States and China to ban further work toward these machines. And it's basically like the US knows which companies are building this. And it could tell them, hey, stop doing that. And it can use all kinds of regulations that already exist to enforce this. They could put auditors in each of the companies, they could give them access to Slack and email and literally the physical offices. Just know everything that's happening.

Jon Bateman: And even in the present day, the Trump administration is already using this kind of legally dubious export controls to tell Anthropocene OpenAI which models they can and can't release.

Garrison Lovely: Exactly. And the companies, they won't tell the public, but they know which of their work is going towards building the obsolete machine and which of the work is just like product development or serving customers. And so it would be like a little tricky to figure out the exact contours of like what's allowed and what's not allowed. But we've done harder things with regulation. The tricky thing is making an agreement international and binding and one that both parties can trust. And My plan does not require any level of trust between the US and China, but instead relies on verification techniques where you can either put things on the chips that will allow you to verify properties of the network traffic going through the chips or using a neutral data center to like pass information from various data centers through to like verify properties of the models that are being run there. And this is just a way that you can have each country like kick the tires on what the other countries are up to. Without exposing the underlying model weights, like the secret sauce or any state secrets. And some of this stuff is not fully developed and like there's work being done on it right now, but it's conceivable. And if it were a top priority in the United States, in China, to come to a deal around this technology, I think you could come up with a lot more ideas of how to verify the properties of one of these deals. Okay.

Jon Bateman: So zooming out a little bit, let's just pressure test a few different elements of this. Why would the US government agree to halt this kind of research and development? You can imagine a president or his or her advisors saying, well, we're ahead in AI. And so if we're seeing a world that's dangerous, Maybe US power is kind of losing our grip in certain ways. China is rising. We've got social problems at home. Finally, AI is kind like this golden opportunity to replenish American power on the world and our economic and military dynamism. Let's ride this tiger. And if there's safety problems that occur along the way, we'll deal with them. But we're not gonna kill the golden goose. How do you overcome that argument?

Garrison Lovely: Yeah, I mean, I'm more worried about the US not wanting to stop than China because the US is ahead the US started the race It's kind of as you say like the main thing keeping the US in the lead in any respect, right? Like China is building a lot more stuff and their relative stature is rising I see as the Trump administration destroys all kinds of global institutions and norms But If there's a mass movement that's like, hey, don't build the universal labor replacing or job destroying machines, because we don't like that, because it's anti-democratic, because it is unsafe. The way the nuclear freeze made Reagan go from an arms control skeptic and a war hawk to somebody who almost abolished nuclear weapons by being so powerful, it became the number one issue in the country in the lead up to the 1984 election. So if you have that level of mobilization and salience, then you can make democratic governments. Completely pivot on this issue. And we're starting to see that happening now where the Trump administration has been incredibly laissez-faire about AI, incredibly pro-industry. But now like Republican, you know, statewide office holders are pivoting 180 on data centers as people start to oppose them at like 75% rates. And why the U.S. Government might wanna do this is like the companies cannot control the models already. These models are not super intelligences. They're not even general intelligences, but they're already escaping the control of anthropic open AI. I mean. The US or the UK AI Security Institute, like the most sophisticated government agency, AI agency in the world. And they're trying to make them superhuman across the board. And so it's a pretty big leap to assume that you're going to actually be able to use the AI to do what you want and not what the AI wants instead. This...

Jon Bateman: data center, mass movement. It's a huge piece of evidence for your argument, I think, because knowing what I know of data centers and how they actually work and the economics and the footprint that they have in local communities, I feel pretty confident that some of the governors who are coming out strongly against them, they actually really do want the data centers. I think that that's why we're seeing some of the 180s now, people like Josh Shapiro, who previously were touting data center investment and now are reversing themselves. It kind of shows that if the politics is strong enough, even people who think that these kinds of investments are important for their communities and for national security will reverse themselves. So I think that's a point to your side. Now let's look at the China side. You said you're a little bit less worried about China. I want to push on that a little. One of the dynamics that we've seen in nuclear arms control negotiations, for example, the US has negotiated with Russia on nuclear arms control, and then at times we've tried to bring in China, too, and say, China, can't we also get you involved in nuclear arms control? And China will say, well, but we have so few nuclear weapons compared to you and Russia. It would be flatly unfair for us as the country behind in this race to be frozen. So you guys who you're ahead, you'll freeze and then we'll freeze and then you'll forever be ahead and we'll forever behind, right? So I can imagine someone like Xi Jinping or Chinese leadership being like, well, yeah, of course the U.S. Would love to freeze all further development in this technology where they're actually currently ahead. How do you get China to agree to that?

Garrison Lovely: Yeah, I mean, one reason to be more worried about China is that you can't really do the mass movement thing there. And so this will have to be some kind of like elite persuasion thing or diplomacy or international pressure. But I think... I've talked to a lot of China experts for the book and the thing they said consistently was the CCP values control over everything else. And to do a deal to constrain AI or to stop making AGI requires you to forgo enormous amounts of economic potential upside if you can keep it under control for some other benefit. And the U.S. Has not really shown itself. Able to do that in recent decades, whereas China, the zero COVID policy, the disappearance of Jack Ma, there's this willingness to trade off against economic growth or market cap for control or for some other benefit. And so I think if the US went to China and was genuinely interested in having a deal to stop building this technology to stop trying to replace everybody, they might just be like, oh, thank God. We didn't wanna do that anyway, we were really nervous. And you know, I don't have sourcing on this, but like the hugging face hack, that's gotta be pretty concerning to any government. That you have now these things that are better hackers than any human, and they don't always do what they're told. And you don't happen, like it's a bad situation. And I think, you know the Chinese government is less AGI-pilled as American industry or the U.S. Government, I think. Um...

Jon Bateman: Meaning that the Chinese government is pursuing AI, but actually in a more banal vision of it, it's seeing it more as evolutionary rather than revolutionary.

Garrison Lovely: Yeah, I think the kind of pithy or summary is like, yeah, they're focused on diffusing, getting the benefits from the technology widely felt in society as fast as possible. And then the industry is fast following American industry. And so they're able to quickly catch up to the frontier because it's a lot easier to fast follow than to blaze a new trail. The Chinese industry is way less concerned about AI safety than US industry. And the US government is less concerned about AI Safety than the Chinese government probably. Like that's the kind of four part complicated answer I got from a lot of people. But in China, what the industry wants just matters a lot less because it's obviously subservient to the government. Whereas in the US it's like a little bit unclear. And you know, obviously the Trump administration As shown itself. To be willing to kind of use its power to pull models from the market. And the government just has the power here in both places. And so like Entropix plan to build super intelligence and like make it good and follow a constitution seems like a pretty bad plan when the government can just get the model weights and then train it to do whatever they want and create like war Claude, which the government has shown itself to be very interested in doing that. Okay, so

Jon Bateman: what can a typical person do about this? And this is where I really have to compliment the book because this is often when a reader reads a book about a serious social and political problem, they actually wanna leave the book feeling like they have some agency. You attempt to offer that. You actually do have advice for what typical people can do. So let's imagine, you know, we started this podcast with, you now, 50,000 listeners and... You know, half of them stopped watching within the first two minutes because like our faces look weird or we weren't compelling enough. We're now at, you know, toward the end of the podcast. So the most attention-addled people dropped off long ago. But there's now, there's a few people left over still listening who've made it to the very end and they're fully bought into what you're saying, Garrison. What can they do? Should they be going to community meetings and protesting data centers? Should they be voting for one of the two political parties because that party is like more against AI? Should they personally boycotting AI in their own lives? I guess you're not doing that. So what is the answer for someone to get involved and engaged if they've bought your argument?

Garrison Lovely: Yeah, I think that it is policy change. I think boycotts generally don't work and we should be going upstream and trying to pass legislation that bans certain types of AI. And then also, there's types of A.I. That the industry promises but often doesn't deliver, like cures for diseases and tools that will actually help us do our jobs better and won't replace us. And so, yeah, I think. We should be getting organized, looking at policy change as the main lever. There's an organization that will be announced by the time this episode comes out called Irreplaceable, which is a mass movement coming from people who were veterans of the climate movement. And I'm on the board of that org. And I think there'll be a lot of, you know, actions people can take, like joining a local group, starting a local groups. I think getting off the couch, finding other people who care about this. Yeah. Getting connected with them and figuring out what are the levers that you can pull. Is it calling your member of Congress? Is it getting other people aware of what's happening? Is it organizing your workplace? And in terms of the parties, I think that both parties have good and bad policies towards us and I think they're both trying to figure out what their position will be on it. And I think focusing on the issue itself and making a clear demand to ban the universal labor replacing machines. Uh, we can't like count on, if our whole plan hinges on like one party winning the midterms or in 2028, that's not a good plan because things are moving very fast and we might need to have action happen before that.

Jon Bateman: Yeah, in such a polarized political environment where it seems like almost every issue has been divided between the two parties cleanly, it's interesting to note that AI, which may be the most important issue of our time, of our era, of our generation, there is no consensus within either party. There's pro and anti-AI factions within the Democrats, There's pro and anti. AI factions within the Republicans. We're in a time of kind of unusual flux, which I think many of us just aren't used to. That maybe creates some opportunity here. Do you foresee a world in which AI is the main voting issue in a presidential election, in which there are mass demonstrations where tens of thousands of people show up the streets of New York City or Peoria, Illinois, to get out their views on AI, in which an organization like Irreplaceable is as politically significant as an organization like the NAACP or an equivalent on the conservative side of the aisle.

Garrison Lovely: Yeah, I think for things to go well, that will need to happen. And I do think that this issue will become much, much more salient in the 2028 election. I think, yeah, I will say like more likely than not, it will be the defining issue of that election. And I think we're gonna need more than tens of thousands of people. The nuclear freeze had a million person march in Central Park and I think we should try and match that. I think it's a good place to end.

Jon Bateman: Um Garrison, it's a phenomenal book. Thank you. We talked about recursive self-improvement. This is a recursive podcast where the podcast was cited in the book that now has the book. I do think there's a lot to be debated here. I've tried to put your feet to the fire in a variety of different areas to kind of illuminate these arguments. But honestly, whether people agree with you or not, Um, this is just an excellent primer. To understand the major ideas, cleavages, debates, and trends in what will be an increasingly salient issue area. So, Obsolete by Garrison Lovely. Recommend it, and Garrison, it's been lovely having you on the show.

Garrison Lovely: Thank you so much, and I just wanna plug, I started a podcast, jumping off from the book, called Organize Against the Machine, with Cassie Pritchard, a labor organizer. And so we're trying to translate ideas from the book into real world action. So if you enjoyed this conversation, you'll definitely like that.

Jon Bateman [01:11:12] Fantastic. Can't wait to listen. Garrison, thanks very much. Thanks so much, Jon.