FULL DISCUSSION: Google's Demis Hassabis, Anthropic's Dario Amodei Debate the World After AGI | AI1G

[0:03]Welcome everybody and welcome to those [0:05]of you joining us on live stream um to [0:07]this conversation that I have to say I [0:09]have been looking forward to for months. [0:12]Uh I had was lucky enough to ch to [0:14]moderate a conversation between Darede [0:17]and Deis Hassabos last year in Paris. Um [0:20]which I'm afraid got most attention for [0:22]the fact that you two were squashed on a [0:24]very small love seat while I sat on an [0:26]enormous sofa which was probably my [0:28]screw up. But I said at that point that [0:30]this was for me like, you know, chairing [0:32]a conversation between the Beatles and [0:33]the Rolling Stones. And you have not had [0:36]a conversation on stage since. So this [0:38]is, you know, the sequel, the the the, [0:40]you know, the bands get together again. [0:42]I'm delighted. You need no introduction. [0:44]Uh the title of our conversation is the [0:46]day after AGI, which I think is perhaps [0:49]slightly getting ahead of ourselves [0:50]because we should probably talk about [0:51]how quickly and easily we will get [0:54]there. And I want to do a bit of a sort [0:55]of update on that and then talk about [0:57]the consequences. So firstly on the [0:58]timeline Dario you last year in Paris [1:01]said we'll have a model that can do [1:03]everything a human could do at the level [1:05]of a Nobel laureate across many fields [1:07]by 2627. We're in 26. Uh do you still [1:11]stand by that timeline? [1:12]>> So you know it's always hard to know [1:14]exactly when something will happen but [1:15]but I don't I don't think that's going [1:17]to turn out to be that far off. So um [1:20]you know the the the mechanism whereby [1:22]imagined it would happen is that we [1:25]would make models that were good at [1:26]coding and good at AI research and we [1:29]would use that to produce the next [1:32]generation of model and speed it up to [1:33]create a loop that would that would uh [1:36]increase the speed of model development. [1:38]We are now in terms of you know the [1:41]models that write code I have engineers [1:44]within anthropic who say I don't write [1:45]any code anymore. I just I just let the [1:47]model write the code. I edit it. I do [1:50]the things around it. I think I don't [1:53]know. We might be six to 12 months away [1:56]from when the model is doing most maybe [1:58]all of what sues do end to end. And then [2:02]it's a question of how fast does that [2:03]loop close. Not every part of that loop [2:06]is something that can be sped up by AI, [2:08]right? There's like chips, there's [2:10]manufacturer of chips, there's training [2:12]time for the model. So it's, you know, I [2:14]I think there's a lot of uncertainty. [2:16]It's easy to see how this could take a [2:19]few years. I don't I I it's very hard [2:21]for me to see how it could take longer [2:22]than that. Um but if if I had to guess, [2:26]I would guess that this goes faster than [2:28]people imagine. And that that key [2:30]element of code and increasingly [2:32]research going faster than we imagine.

[2:36]That's going to be the key driver. It's [2:37]it's really hard to predict again how [2:40]much that exponential is going to speed [2:41]us up, but but something fast is going [2:44]to happen. So you demis were a little [2:46]more cautious last year. You said a 50% [2:48]chance of a system that can exhibit all [2:50]the cognitive capabilities humans can by [2:52]the end of the decade. Um clearly in [2:55]coding as Dario says it's been [2:56]remarkable. What is your sense of do you [2:59]stand by your prediction and what's [3:01]changed in the past year? [3:02]>> Yeah, look I I I I think I'm still on [3:04]the same kind of timeline. And I think [3:06]there has been remarkable progress. But [3:08]I think some areas of uh uh um kind of [3:12]engineering work, coding or so you could [3:14]say mathematics are a little bit easier [3:16]to see how they would be automated [3:18]partly because they're verifiable what [3:20]the output is. Um some areas of natural [3:22]science are much harder to do than that. [3:24]You won't necessarily know if the [3:26]chemical compound you've built or this [3:27]prediction about physics is correct. It [3:29]may be you may have to test it [3:31]experimentally and that will all take [3:33]longer. So uh I also think there are [3:35]some missing capabilities at the moment [3:37]uh in terms of like not just solving [3:39]existing conjectures uh or existing [3:42]problems but actually coming up with the [3:44]question in the first place or coming up [3:46]with the theory or the hypothesis. I [3:47]think that's much much harder and I [3:49]think that's the highest level of [3:51]scientific creativity and it's not [3:53]clear. I think we will have those [3:54]systems. I don't think it's impossible [3:55]but I think there may be one or two [3:57]missing ingredients. Um, it remains to [3:59]be seen how, you know, first of all, can [4:01]this self-improvement loop that we're [4:03]all working on actually close without a [4:05]human in the loop. I think there are [4:06]also risks to that to that kind of [4:08]system, by the way, which we should [4:09]discuss and I'm sure we will. But the [4:11]the but but that could speed things up [4:13]if that kind of system does work. [4:15]>> We'll get to the risks in a minute. But [4:16]one other change I think of the past [4:18]year has been a kind of change in the [4:20]pecking order of the race, if you will. [4:22]This time a year ago, we just had the [4:24]deepseek moment and everyone was [4:25]incredibly excited about what happened [4:27]there and there was still a sense, you [4:29]know, that Google Deep Mind was kind of [4:32]lagging open AI. I would say that now uh [4:35]it's looking quite different. I mean, [4:36]they've declared code red, right? Um [4:38]it's been quite a quite a year. So, talk [4:40]me through what specifically you've been [4:43]surprised by and how well you've done [4:45]this year and whether you think and then [4:46]I'm going to ask you about the lineup. [4:48]Well, look, I I think we were I was [4:50]always very confident we uh would get [4:53]back to sort of the top of the the [4:54]leaderboards and and the soda type of [4:56]models across the board because I think [4:58]we've always had like the deepest and [5:00]broadest research bench and it was about [5:02]kind of marshalling that all together [5:04]and um getting the intensity and focus [5:06]and the kind of startup mentality back [5:08]to the whole organization and it's been [5:10]a a lot of work and um but I think we're [5:13]and we're still a lot of work to do um [5:15]but I think you can start seeing the the [5:17]the the the you know the the kind of um [5:19]the progress that's been made in both [5:21]the models with Gemini 3 but also uh on [5:24]the product side with Gemini app getting [5:26]increasing uh market share. So I feel [5:28]like uh we're making great progress um [5:31]but there's a ton more work to do um and [5:33]you know we're bringing to bear Google [5:35]DeepMind's kind of like the engine room [5:37]of Google where we're getting used to um [5:39]shipping our models more and more more [5:41]quickly into the product surfaces. One [5:43]question for you Daria on on this aspect [5:44]of it because you've just saw you're in [5:46]the process of you know a new round at [5:48]an extraordinary valuation too. Um but [5:50]you are unlike them as a let's call it [5:53]an independent model maker and there is [5:55]I think an increasing concern that the [5:57]independent model makers will not be [5:58]able to continue for long enough until [6:01]you get to where the revenues come in. [6:02]Um it's made very openly about open AI [6:05]but talk me through how you think about [6:06]that and then we'll get to the AGI [6:08]itself. Yeah, I mean that you know I [6:09]think I think I think how we think about [6:11]that is you know as we've built better [6:14]and better models there's been a kind of [6:16]exponential relationship not only [6:19]between how much compute you put into [6:20]the model and how cognitively capable it [6:23]is but between how cognitively capable [6:25]it is and how much revenue it's able to [6:27]generate. So our revenues grown 10x in [6:29]the last three years from 0 to 100 [6:31]million in 2023 100 million to a billion [6:34]in 2024 and 1 billion to 10 billion in [6:36]2025. And so th those revenue numbers, [6:39]you know, I don't know if that curve [6:40]will literally continue. It would be [6:42]crazy if it did. Um, but those numbers [6:44]are starting to get not too far from, [6:47]you know, the sca the scale of the [6:48]largest companies in the world. So [6:50]there's there's there's always [6:51]uncertainty. You know, we're trying to [6:52]bootstrap this from nothing. It's it's a [6:54]crazy thing, but but I have confidence [6:57]that if we're able to produce the best [6:59]models in the things that we focus on, [7:02]um, uh, then I think then I think things [7:03]will go well. And you know, I I will I [7:05]will generally say, you know, I think I [7:07]think it's been a good year for both [7:09]both Google and Anthropic. And I think [7:11]the thing we actually have in common is [7:13]that they're you know, they're both kind [7:15]of kind of kind of companies that are, [7:17]you know, or the research part of the [7:18]company that are kind of led by [7:20]researchers who focus on the models who [7:23]focus on solving important problems in [7:25]the world, right? Who have these kind of [7:27]hard scientific problems as a as a north [7:30]star. and and and I think those are the [7:33]kind of companies that are going to [7:34]succeed going forward and you know I [7:37]think I think we share that between us [7:38]>> very much. Uh I'm I'm going to resist [7:40]the temptation to ask you what will [7:41]happen to the companies that are not led [7:43]by researchers [7:45]uh because I know you won't answer it. [7:46]But let's then go on to uh the [7:49]predictions area now and this we are [7:51]supposed to be talking about the day [7:52]after AI but let's talk about closing [7:54]the loop. This the odds that you will [7:57]get models that will close the loop and [7:59]be able to you know power themselves if [8:01]you will because that's the really the [8:02]crux for the the winner takes all [8:04]threshold approach. Do you still believe [8:07]that we are likely to see that or is [8:09]this going to be much more of a normal [8:11]technology where followers and catchup [8:13]can can compete? [8:15]>> Well, look, I definitely don't think [8:16]it's going to be a normal technology. [8:18]So, I mean, there are aspects already [8:20]that as Dario mentioned that it's [8:22]already helping with our coding and and [8:24]some aspects of research. The full [8:26]closing of the loop though, I think is [8:28]an unknown. I mean, I think it's [8:30]possible to do. you may need AGI itself [8:33]to be able to do that in some domains [8:35]again where there these domains you know [8:37]where there's there's more messiness [8:38]around them it's not so easy to verify [8:40]your answer very quickly um there's kind [8:43]of MP hard domains so as soon as you [8:46]start getting more and you know I also [8:47]include by the way for AGI physical AI [8:50]robotics working all of these kind of [8:51]things and then you've got you know [8:53]hardware in the loop uh that may uh [8:55]limit how fast the self-improvement [8:57]systems can work but I think in coding [8:59]and mathematics and these kind areas. I [9:01]can definitely see that working. And [9:03]then the question is more theoretical [9:04]one is what is the limit of engineering [9:06]and maths uh to solve uh the natural [9:09]sciences. Daria, you um last year, I [9:13]think it was last year that you [9:14]published Machines of Love and Grace um [9:16]which was a very I would say upbeat [9:19]essay about the potential that that you [9:22]were going to see unfold and you were [9:23]talking about you know a a what was it a [9:26]genius of data at country [9:29]data center I'm told that you are [9:31]working on an update to this a new essay [9:33]so you know wait for it guys it's not [9:36]out yet but it is coming out but perhaps [9:38]you can give us a sort of a sneak [9:40]preview of what a year later your big [9:43]take is going to be.

[9:44]>> Yes. So, you know, my take my take has [9:46]not changed. It has always been my view [9:48]that, you know, AI is going to be [9:50]incredibly powerful. I think Demis and [9:52]I, you know, kind of agree on that. It's [9:53]just a question of exactly when. Um, uh, [9:56]and because it's incredibly powerful, it [9:58]will do all these wonderful things like [9:59]the ones I talked about in Machines of [10:01]Loving Grace. It, you know, will help us [10:03]cure cancer. It may help us to eradicate [10:05]tropical diseases. It will help us [10:07]understand understand the universe. but [10:10]that there are these, you know, immense [10:12]and grave risks that, you know, not that [10:14]we can't address them. I'm not a doomer, [10:16]but but that, you know, we we we we we [10:18]need to think about them and we need to [10:20]address them. And I wrote Machines of [10:21]Loving Grace first. I' I'd love to give [10:23]some uh a sophisticated reason why I [10:25]wrote that first, but it was just that [10:27]the the positive essay was easier and [10:29]more fun to write than than the negative [10:31]essay. Um, so, you know, I finally spent [10:34]some time on vacation and I was able to [10:36]write an essay about the risks. Even [10:38]when I'm writing about the risks, um, I [10:40]I I try, you know, I I I'm like an [10:43]optimistic person, right? So, even as [10:45]I'm writing about these risks, I I I [10:48]wrote about it in a way that was like, [10:49]how do we overcome these risks? How do [10:51]we have a battle plan to fight them? And [10:53]and and the way I the way I framed it [10:56]was, you know, there's this scene from [10:58]Carl Sean's Contact, the movie version [11:00]of it, where, you know, they they kind [11:02]of discover alien life and this [11:04]international panel that's like [11:05]interviewing um uh you know, people to, [11:08]you know, to be humanity's [11:09]representative to meet the alien. Um uh [11:12]and uh one one of the questions they ask [11:14]one of the candidates is, you know, if [11:16]you could ask the aliens any one [11:17]question, what it would what what what [11:19]would it be? And one of one of the [11:21]characters says,"I would ask,"How did [11:23]you do it? How did you manage to get [11:26]through this technological adolescence [11:28]without destroying yourselves? How did [11:30]you make it through?" And and and ever [11:32]since I saw it, it was like 20 years [11:34]ago, I think I saw that movie. It's kind [11:35]of stuck with me. And that that's the [11:37]frame that I use, which is which is [11:39]that, you know, we we're we're we are [11:42]knocking on the door of these incredible [11:44]capabilities, right? the the ability to [11:46]build basically machines out of sand, [11:49]right? I think I think it was inevitable [11:51]that the instant we started working with [11:53]fire. Um uh but but how we handle it is [11:57]is not inevitable. And so I think the [11:59]next few years we're going to be dealing [12:02]with, you know, how do we keep these [12:04]systems under control that are highly [12:06]autonomous and smarter than any human? [12:09]How do we make sure that individuals [12:12]don't misuse them? Right? I have worries [12:14]about things like bioteterrorism. How do [12:16]we make sure that nation states don't [12:18]misuse them? That's why I've been so [12:20]concerned about, you know, the CCP, [12:22]other authoritarian authoritarian [12:24]governments. What are the economic [12:26]impacts? Right? I've talked about labor [12:28]displacement a lot. And and you know, [12:29]what what haven't we thought of which [12:31]which in many cases, you know, maybe may [12:33]be the the hardest thing to deal with at [12:35]all. Um, so, you know, I I'm I'm [12:37]thinking through how to address those [12:40]risks. you know, for for each of these, [12:42]it's a mixture of things that we [12:44]individually need to do as as leaders of [12:46]the of of of the companies and that we [12:49]can do working together. And then there [12:50]there's going to need to be some role [12:52]for wider societal institutions like the [12:54]like the government in in in addressing [12:56]all of these. But, you know, I I I just [12:58]feel this urgency that, you know, every [13:00]day, you know, there's there's all kinds [13:02]of crazy stuff going on in the outside [13:04]world, outside AI, right? Um but but you [13:07]know my my my view is this is happening [13:09]so fast and is such a crisis we should [13:12]be devoting almost all of our effort to [13:14]thinking about how to get through this. [13:16]>> So I can't decide whether I'm more [13:18]surprised that you a take a vacation b [13:20]when you take a vacation you think about [13:22]the risks of AI and c that your essay is [13:24]framed in terms of are we going to get [13:26]through the technological adolescence of [13:28]this technology without destroying [13:29]ourselves. So, I'm my head is slightly [13:31]spinning, but you then and I can't wait [13:33]to read it, but you you you mentioned [13:35]several areas that can guide the rest of [13:36]our conversation. Let's start with jobs [13:39]um because you actually have been very [13:40]outspoken about that and I think you [13:41]said that half of entry- level white [13:43]collar jobs could be gone within the [13:44]next one to five years. But I'm going to [13:46]turn to you Demis because so far we [13:50]haven't actually seen any discernable [13:52]impact on the labor market. Um, yes, [13:54]unemployment has ticked up in the US, [13:56]but all of the kind of economic studies [13:58]I've looked at and that we've written [14:00]about suggest that this is overhiring [14:02]post pandemic that it's really not [14:04]AIdriven. If anything, people are hiring [14:07]to build out AI capability. [14:10]Do you think that this will be as you [14:12]know economists have always argued that [14:15]it's not a lump of labor fallacy that [14:17]actually there will be new jobs created [14:18]because so far the evidence seems to [14:20]suggest that. Yeah, I mean I I think in [14:23]um the near term that is what will [14:25]happen. The kind of normal evolution [14:26]when a breakthrough technology arrives. [14:28]So some jobs will get disrupted but I [14:31]think new even more valuable perhaps [14:32]more meaningful jobs will get created. [14:34]Um I think we're going to see this year [14:36]the beginnings of maybe impacting the [14:39]junior level entry level child of jobs [14:41]internships this type of thing. And I [14:43]think there is some evidence I can feel [14:44]that ourselves maybe like a slowdown in [14:47]hiring in that. But I think that can be [14:49]more than compensated by the fact there [14:50]are these amazing creative tools out [14:52]there pretty much available for everyone [14:54]uh almost for free that if you know I [14:57]was to talk to a class of undergrads [15:00]right now I would be telling them to get [15:02]really unbelievably proficient with [15:04]these tools. I think to the extent that [15:06]even those of us building it, we're so [15:08]busy building it, it's hard to have also [15:10]time to really explore the almost the [15:12]capability overhang even today's models [15:14]and products have let alone tomorrow's [15:16]and I think that uh can be maybe better [15:19]than a traditional internship would have [15:20]been in terms of you sort of leaprogging [15:23]uh yourself to be useful uh in a useful [15:26]in a profession. So I think there's [15:28]that's what I see happening probably in [15:30]the next five years. Um maybe we again [15:32]slightly differ on time scales on that [15:34]but I think what happens after AGI [15:36]arrives that's a different question cuz [15:37]I think really we would be in uncharted [15:40]territory at that point.

[15:41]>> Do you think it's going to take longer [15:42]than you thought last year when you said [15:43]half of all white [15:45]>> colors? I have about the same view. I I [15:46]actually agree with you and with Demis [15:48]that at the time I made the comment [15:50]there was no impact on the labor market. [15:52]I wasn't saying there was an impact on [15:54]the labor market at that moment. Um, you [15:56]know, now I think maybe we're starting [15:59]to see just just the little beginnings [16:00]of it, you know, in software in coding. [16:02]I even see it within within anthropic [16:04]where, you know, I you know, I can look [16:08]forward I can kind of look forward to a [16:10]time where on the more junior end and [16:12]then on the more on the more on the more [16:14]on the more intermediate end, we [16:16]actually need less and not more people.

[16:17]And you know, we're thinking about how [16:19]to deal with that within anthropic in a [16:21]in a in a you know, sense in a sensible [16:24]way. Um I, you know, one to five years [16:28]as of six months ago, I would stick with [16:30]that. You know, if you kind of, you [16:32]know, connect this to what I said [16:33]before, which is, you know, we we might [16:36]have AI that's better than humans at at [16:38]everything in, you know, maybe one to [16:40]two years, maybe a little longer than [16:42]that. The those don't seem to line up. [16:45]The reason is that there's this there's [16:47]this lag and there's this replacement [16:50]thing, right? I I know the labor market [16:52]is adaptable, right? Just like you know [16:54]80% of people used to do farming you [16:56]know farming got automated and then they [16:58]became factory workers and then [16:59]knowledge workers. So you know there is [17:02]some level of adaptability here as well [17:05]right we should be economically [17:06]sophisticated about how the labor market [17:08]works but my worry is as this [17:10]exponential keeps compounding and I [17:12]don't think it's going to take that long [17:14]again somewhere between between a year [17:17]and five years it will overwhelm our [17:19]ability to adapt. I think I may be [17:20]saying the same thing Demis is just [17:23]factored out of that that difference we [17:25]have about timelines which I think [17:27]ultimately comes down to how how fast [17:28]you close the loop on CO.

[17:30]>> How much confidence do you have that [17:32]governments get the scale of this and [17:35]have are beginning to think about what [17:37]policy responses they need to have? [17:39]>> I don't think that that that it's [17:42]anywhere near enough work going on about [17:44]this. I'm I'm constantly surprised even [17:45]when I meet economists at places like [17:47]this that they're not more of uh [17:49]professional economist professors [17:51]thinking about what happens um and not [17:54]just sort of on the way to AGI but um uh [17:57]even if we get all the technical things [17:58]right that Dario was talking about and [18:00]the job displacement is one question [18:02]we're worried about the economics of [18:03]that but maybe there are ways to [18:04]distribute this new productivity this [18:06]new wealth more fairly I don't know if [18:08]we have the right institutions to do [18:10]that but that's what should happen at [18:11]that point there should be you know we [18:13]maybe in a post scarcity world. But then [18:15]there are even the things that keep me [18:16]up right now. There are even bigger [18:17]questions than that at that point to do [18:19]with meaning and um purpose and a lot of [18:23]the things that we get from our jobs not [18:25]just economically. That's one question. [18:26]But I think that may be easier to solve [18:29]strangely than uh what happens to the [18:31]human condition and humanity as a whole. [18:33]And I think I'm also optimistic we'll [18:35]come up with new answers there. We do a [18:36]lot of things today um from extreme [18:39]sports to art that aren't necessarily [18:41]directly to do with economic gain. So I [18:44]think we will find uh meaning and maybe [18:47]there'll be even more sort of [18:48]sophisticated versions of those [18:50]activities. Um plus I think we'll be [18:52]exploring the stars. So there'll be all [18:54]of that to to factor in as well for in [18:57]terms of purpose. But I think it's [18:58]really worth thinking now even on my [19:01]timelines of like five to 10 years away [19:03]that isn't a lot of time uh before this [19:05]comes. How big do you think is the risk [19:07]of a popular backlash against AI that [19:11]will somehow kind of cause governments [19:14]to do what from your perspective might [19:16]be stupid things? Because I'm just [19:17]thinking back to the era of, you know, [19:20]globalization in the 1990s when when [19:22]there was indeed some displacement of [19:24]jobs. Governments didn't do enough. The [19:27]public backlash was such that we've [19:29]ended up sort of where we are now. uh do [19:31]you think that there is a risk that [19:33]there will be a growing antipathy [19:36]towards what you are doing and your [19:38]companies in the kind of body politic? [19:40]>> Um I think there's definitely a risk. I [19:42]think um I think that's kind of [19:44]reasonable. There's fear and there's [19:46]worries about these things like jobs and [19:47]livelihoods. Um I think there's a couple [19:50]of things that I mean it's going to be [19:52]very complicated the next few years I [19:53]think geopolitically but also the [19:55]various factors here like we want to and [19:57]we're trying to do this with AlphaFold [19:59]and our science work and isomeorphic our [20:01]spinout company solve all disease cure [20:03]diseases come up with new energy sources [20:06]I think as a society it's clear we'd [20:07]want that I think maybe the balance of [20:09]what the industry is doing is not enough [20:11]balance towards those types of [20:13]activities I think we should have a lot [20:14]more examples I know Dary agrees with me [20:16]of like alpha fold like things that help [20:19]sort of unequivocal good in the world [20:21]and I think actually it's incumbent on [20:23]the industry and and all of us leading [20:24]players to show that more demonstrate [20:26]that not just talk about it but [20:27]demonstrate that um and but then it's [20:30]going to come with these other intendent [20:32]disruptions and um but I don't I think [20:34]the other issue is the geopolitical [20:36]competition there's obviously [20:37]competition between the companies but [20:38]also US and China primarily so unless [20:41]there's an international cooperation or [20:43]or understanding around this um uh which [20:46]I think would be good actually in terms [20:47]things like minimum safety standards for [20:49]deployment. I think Dario would agree on [20:51]that as well. I think it's vitally [20:52]needed. This technology is going to be [20:54]crossborder. It's going to affect [20:55]everyone. It's going to affect all of [20:56]humanity. Um actually contact is one of [20:59]my favorite films as well. So funny [21:01]enough, I didn't realize it was yours [21:02]too, Dario. But I I think um um you know [21:06]those kind of things need to be worked [21:07]through. Um and and if we can maybe it [21:10]would be good to have a bit of slow a [21:12]slightly slower pace than we're [21:14]currently predicting even my timelines [21:16]so that we can get this right society [21:18]but it that would require some [21:19]coordination that is I prefer your [21:22]timelines. Yes, I think I battle [21:24]concede.

[21:25]>> But but but Dario, let's turn to this [21:27]now because the one thing since we last [21:29]spoke uh in Paris, the geopolitical [21:31]environment has, if anything, I don't [21:33]know, complicated, mad, crazy, whatever, [21:36]whatever phrase you want to use. [21:38]Secondly, the US has a very different [21:40]approach now towards China. It's a much [21:42]more it's a kind of no holds bar, go as [21:44]fast as we can, but then sell chips to [21:46]China. Um, and that is so you've got a [21:50]different attitude towards the United [21:51]States. you've got a a very um strange [21:55]relationship between the United States [21:56]and and Europe right now geopolitically [21:59]against that. I mean I hear you talk [22:01]about it would be nice to have a CERN [22:02]like organization I mean it's a million [22:04]years from where we are from the real [22:06]world. So in the real world have the [22:08]geopolitical risks increased and what if [22:11]anything do you think should be done [22:13]about that and and the administration [22:14]seems to be doing the opposite of what [22:15]you were suggesting? Yeah, I mean, look, [22:17]you know, we're we're we're just trying [22:18]to do the best we can to, you know, [22:20]we're just we're just one company and [22:21]we're we're trying to operate in, you [22:22]know, the the environment that exists, [22:24]no matter how no matter how crazy it is. [22:26]But, you know, I think I think at least [22:28]my policy recommendations haven't [22:30]changed that, you know, not selling [22:33]chips is one of the, you know, one of [22:36]the one of the biggest things we can do [22:38]um to, you know, make sure that we have [22:41]the time to handle this. Um, you know, [22:44]you know, I said I said before, you [22:45]know, I I I prefer Demis' timeline. I [22:48]wish we had 5 to 10 years, you know, so [22:51]it's it's possible he's just right and [22:52]I'm just wrong, but but assume I'm right [22:54]and it can be done in one to two years. [22:56]Why can't we slow down to to Demis' [22:58]timeline? [22:59]>> Well, you could just slow down. Well, [23:00]no. The but but but the reason the [23:02]reason we the reason we can't do that is [23:05]is you know because we have [23:07]>> geopolitical adversaries building the [23:10]same technology at a similar pace, it's [23:13]very hard to have an enforcable [23:14]agreement where they slow down and we [23:16]slow down and and so if we can just [23:19]>> if we can just not sell the chips, then [23:22]this isn't a question of competition [23:24]between the US and China. This is a [23:26]question of competition between me and [23:28]Demis which I'm very confident that we [23:29]can work out. [23:30]>> And what do you make of the logic of the [23:32]administration which as I understand it [23:34]is we need to sell them chips because we [23:36]need to bind them into US supply chains. [23:40]So you know it's it's I I think it's I [23:44]think it's a question not just of time [23:46]scale but of the significance of the [23:48]technology right if this was telecom or [23:52]something then all this stuff about [23:54]proliferating the US stack and you know [23:56]wanting to build our you know chips [23:58]around the world to make sure that you [24:00]know you know this c you know the you [24:04]know these random countries in different [24:05]parts of the world you know build data [24:07]centers that have Nvidia chips instead [24:10]of Huawei chips. You know, I think of [24:12]this more as like, you know, it's a [24:14]decision. Are we going to, you know, [24:17]sell nuclear weapons to North Korea and, [24:20]you know, because that produces some [24:22]profit for Boeing. Um, you know, where [24:24]where we can say, okay, yeah, these [24:25]cases were made by Boeing, like the US [24:28]is winning. Like, this is great. Like, I [24:29]I I just, you know, that that analogy [24:32]should just make clear how I see this [24:34]trade-off that I just don't think it [24:36]makes sense. Um and and we've done a lot [24:40]of more aggressive stuff toward, you [24:41]know, toward towards towards China, [24:43]China and other players that that I [24:45]think is much less effective than this [24:46]this one this one measure.

[24:48]>> One more area for me and then I hope [24:50]we'll have time for a question or two. [24:52]The other area of potential risk that [24:54]doomers worry about is a kind of all [24:56]powerful malign AI. Um, and I think [24:59]you've both been somewhat skeptical of [25:01]the doomer approach, but in the last [25:02]year we have seen, you know, these [25:05]models showing themselves to be capable [25:07]of deception, duplicity. Uh, do you [25:11]think that do you think differently [25:13]about that risk now than you did a year [25:14]ago? And is there something about the [25:17]way the models are evolving that we [25:19]should put a little bit more concern on [25:20]that? [25:21]>> Yeah, I mean, you know, since since the [25:22]beginning of Enthropic, we've kind of [25:24]thought about this risk. I mean, you [25:26]know, our our our research at the [25:28]beginning of it was very theoretical, [25:30]right? You know, we pioneered this idea [25:31]of mechanistic interpretability, which [25:33]is looking inside the model and and [25:35]trying to understand looking inside its [25:37]brain, trying to understand why it does [25:39]what it does as it, you know, as as [25:41]human neuroscientists, which we actually [25:43]both have background in, um, try try to [25:45]understand try to understand the brain. [25:48]And I think as time has gone on, we've [25:50]we've increasingly documented the you [25:52]know bad behaviors of the models when [25:54]they emerge and are now working on [25:56]trying to address them with mechanistic [25:58]interpretability. So I you know I think [26:00]uh you know I I've always been concerned [26:02]about these these risks. I've talked to [26:04]Demis many times. I think he has also [26:05]been um concerned about these risks. I [26:08]think I have definitely been and I I I [26:11]would guess Demis as well although I'll [26:12]let him speak for himself skeptical of [26:15]of dumerism which is you know we're [26:17]doomed. there's nothing we can do or [26:19]this is the most likely outcome. I think [26:21]this is a risk. This is a risk that if [26:24]we work all work together, we can [26:26]address we can learn through science to [26:29]properly, you know, control and and [26:31]direct these creations that we're [26:33]building. But if we build them poorly, [26:36]if we go, you know, if if if we're all [26:40]racing and we go so fast that there's no [26:42]guardrails, then I think there is risk [26:43]of something going wrong. So I'm going [26:45]to give you a chance to answer that in [26:46]the context of of a slightly broader [26:48]question which is over the past year [26:50]have you grown more confident of the [26:53]upside potential of the technology [26:55]science all of the areas that you have [26:57]talked about a lot or are you more [26:59]worried about the risks that we've been [27:00]discussing [27:02]I've been working on this for 20 plus [27:04]years so we we already knew look the [27:06]reason I've spent my whole career on AI [27:08]is is the upsides of solving basically [27:11]the ultimate tool for science and [27:13]understanding the universal around us. [27:14]I've I've sort of been obsessed with [27:16]that since a kid and and and building AI [27:18]is the you know should be the ultimate [27:20]tool for that if we do it in the right [27:21]way. The risks also we've been thinking [27:23]about since the start at least the start [27:25]of deep mind 15 years ago and um we kind [27:28]of sort of foresaw that if you got the [27:29]upsides it's a dual purpose technology [27:31]so it could be repurposed by say bad [27:34]actors for harmful ends. So we've needed [27:35]to think about that all the way through [27:37]but I'm a big believer in human [27:39]ingenuity. Um but the question is having [27:41]the time and the focus and all the best [27:45]minds collaborating on it to solve these [27:48]problems. I'm sure if we had that we [27:50]would solve the technical risk problem. [27:52]It may be we don't have that and then [27:53]that will introduce risk because we'll [27:55]be sort of it'll be fragmented. There'll [27:57]be different projects and people be [27:59]racing each other then it's much harder [28:00]to make sure you know these systems that [28:02]we produce will be technically safe. But [28:04]I I feel like that's a very tractable uh [28:07]problem if we have the time if we have [28:09]the time and space. I want to make sure [28:11]there's one question gentlemen. Keep it [28:13]very short because we've got literally [28:14]two minutes. [28:16]>> Thanks for Hello. [28:18]>> Yeah. No speak. [28:19]>> Thanks very much. I'm Philip, co-founder [28:21]of StarCloud building data centers in [28:22]space. Um I wanted to ask a slightly [28:25]philosophical question. The sort of [28:27]strongest argument for doomerism to me [28:29]is the firmy paradox, the idea that we [28:30]don't see intelligent life in our [28:31]galaxy. I was wondering if you guys have [28:33]any thoughts. Yeah, I've thought a lot [28:34]about that. That can't be the reason [28:35]because we we we should see all the AIS [28:38]that have So, just so everyone know the [28:41]idea is well, it's sort of unclear why [28:43]that would happen, right? So, if if the [28:45]reason there's a Firmeny paradox, there [28:46]are no aliens cuz they get taken out by [28:48]their own technology. We should be [28:50]seeing paper clips coming towards us [28:52]from some part of the galaxy. And [28:54]apparently, we don't. We don't see any [28:55]structures. Dyson sphere is nothing [28:57]whether they're AI or natur or sort of [28:59]biological. So to me um there has to be [29:02]a different answer to FMY powders. I [29:03]have my own theories about that, but [29:04]it's out of scope for the next minute. [29:06]But um you know I I just feel like uh [29:09]that that I my prediction my feeling is [29:12]that we're past the great filter. It was [29:13]probably multisellular life if I would [29:15]have to guess. It was incredibly hard [29:17]for for biology to evolve that. Um so [29:20]we're on you know there isn't a comfort [29:22]of like what's going to happen next. I [29:24]think it's for us to write as humanity [29:26]what's going to happen next. [29:26]>> This this could be a great discussion [29:28]but is out of scope for the next 36 [29:29]sessions. But what isn't 15 seconds each [29:32]what when we meet again I hope next year [29:34]uh the three of us which I would love uh [29:36]what will have changed by then [29:38]>> I well I think the biggest thing to [29:41]watch is this issue of AI systems [29:44]building AI systems how that goes whe [29:47]that whether that goes one way or [29:49]another that that will determine you [29:52]know whether it's a few more years until [29:55]we get there or or if we have you know [29:58]you know if if we have wonders and and a [30:01]great emergency in front of us that we [30:03]have to face.

[30:04]>> AI systems, building AI system. [30:05]>> I agree on that. So, we're we're keeping [30:07]close touch about that. Um but also I [30:09]think um outside of that, I think there [30:11]are other interesting uh uh uh ideas [30:13]being researched like world models, [30:15]continual learning. These are the things [30:16]I think that will need to be cracked. If [30:18]self-improvement doesn't sort of deliver [30:20]the goods on its own, then we'll need [30:22]these other things to work. And then I [30:24]think things like robotics may have its [30:25]sort of breakout moment. But maybe on [30:27]the basis of what you've just said, we [30:29]should all be hoping that it does take [30:30]you a little bit longer and indeed [30:31]everybody else to give us [30:32]>> I would prefer that. I think that would [30:34]be better for the world. [30:35]>> Well, you guys could do something about [30:36]that. Thank you both very much.