Alexandr Wang: Building Scale AI, Transforming Work With Agents & Competing With China

[0:00]Since we recorded this Lite Cone episode [0:01]with Scale AI CEO Alexander Wang, Meta [0:05]has agreed to invest over $14 billion in [0:09]scale, valuing the company at $29 [0:12]billion. Alex has also announced he will [0:15]lead Meta's new AI super intelligence [0:17]lab. Our conversation you're about to [0:20]hear covers the history leading up to [0:22]this investment. From scale's early days [0:24]at YC to its integral role in the [0:27]training of foundational models. Let's [0:30]get to it. The AI industry really [0:32]continues to suffer from a lack of uh [0:37]very hard evals and very hard tests that [0:40]show really like the frontier of model [0:43]capabilities. The biggest thing is you [0:46]just have to really really really care. [0:49]When you interview people or when you [0:50]interact with people, you can tell [0:51]people who are just sort of like phone [0:53]it in versus people who sort of like [0:55]hang on to their work. It's like the so [0:58]incredibly monumental and forceful and [1:00]important to them that they they do [1:02]great work. Very exciting time to to see [1:05]the how the frontier of human knowledge [1:06]expands. [1:08][Music] [1:15]Welcome to another episode of the Light [1:18]Cone. Today we have a real treat. It's [1:21]Alexander Wang of Scale AI. Jared, you [1:24]worked with uh Alexander way back in the [1:27]beginning actually. Uh what was that [1:28]like? What year was it? Put us in the [1:30]spot. Yeah, Alex. I mean, most of what [1:32]we want to talk about today is like what [1:34]Scale is doing now because like the the [1:36]current stuff is like so so awesome and [1:38]so interesting since Scale got started [1:40]at YC. I thought it just seemed [1:42]appropriate to start all the way at the [1:43]start. And um you it's funny uh Diane [1:46]and I were at MIT last month talking to [1:48]college students and like of all the [1:50]founders, the one that they like most [1:52]look up to and like want to emulate is [1:54]actually you. Like everybody wants to be [1:56]the next Alexander Wankers. Everybody [1:57]knows the story of how you like dropped [1:59]out of MIT and and ended up starting [2:01]scale, but they don't know the real [2:03]story. And so I thought it'd be cool to [2:05]go back to the beginning and just talk [2:06]about the real story of how you ended up [2:08]dropping out of MIT and starting scale. [2:10]So before I went to MIT, I worked at um [2:12]Quora for a year. And so this is 2015 to [2:16]2016 or no sorry 2014 to 2015 was when I [2:18]worked as a software engineer and this [2:20]was already at a point in the market [2:22]where ML engineers as they were called [2:24]or like machine learning engineers uh [2:26]made more than software engineers. So [2:28]that was already like the market state [2:30]at that point. I went to these summer [2:31]camps um that were that were organized [2:34]by um by rationalists the rationality [2:37]community in San Francisco. So um and [2:40]they were for precocious teams but they [2:41]were organized by um uh many people who [2:45]have become pivotal in the AI industry. [2:47]So one of the organizers is this guy [2:49]Paul Cristiano who um used to uh who's [2:53]the inventor of RHF actually and now he [2:55]run or he's a research director at the [2:57]US AI safety institute. He was at [2:59]opening for a long time. Um Greg [3:00]Brockman came and gave a speech at one [3:02]point. Eleazar Yudkowski came and gave a [3:04]speech at one point and actually I was [3:06]very like when I was I don't know must [3:08]have been uh 16 I was exposed to this [3:11]concept that like potentially the most [3:13]important thing to work on in my [3:15]lifetime was AI and AI safety. So [3:18]something I was exposed to very early [3:19]very early on. So then when I went to [3:20]MIT I was started MIT when I was 18. I [3:24]like studied AI quite deeply. That was [3:27]most of what I did in the sort of day [3:29]job. and then um uh kind of got antsy [3:33]applied to YC and then the idea was kind [3:36]of like okay how could initially was [3:37]like okay where can you apply um sort of [3:40]like AI to things and this was um in the [3:44]era of chat bots which is like crazy to [3:47]think about actually um that there was [3:49]like this like mini chatbot bubble boom [3:51]yeah yeah yeah 100% in uh in 2016 um [3:56]which is uh which was I guess spurred by [3:59]magic right? Or or some of these apps [4:01]and and uh Facebook had a big vision [4:03]around chat bots. And anyway, there's [4:05]this little mini chatbot boom. So, the [4:06]initial thing that we wanted to work on [4:09]uh and um was was chat bots for doctors, [4:13]right? [4:14]Which is like a funny idea because do [4:17]you guys know anything about doctors? [4:19]Yeah. No, not at all. um like basically [4:21]no I it was just sort of like oh doctors [4:23]are a thing that sound expensive and so [4:26]and I think it was like I think it's [4:28]like indicative of like I mean I don't [4:29]know you guys see this all the time but [4:30]I feel like most of the times young [4:33]founders like first 10 ideas are like [4:36]always first of all they're very [4:37]mimemetic so they're probably like [4:38]there's a lot of like the same ideas [4:39]over there's like a dating app there's [4:41]like some something for like you know [4:43]social life you know there's the same [4:45]ideas um and then I think that like I [4:48]think young people have a very poor [4:49]sense sense of alpha like what are what [4:51]are the things that they're actually [4:52]like going to be uniquely positioned to [4:54]do and I think you know most young [4:56]people don't have a sense of self so [4:58]it's you know it's not clear so when we [5:00]were in YC we were roommates with uh [5:02]with um with another YC uh company and [5:07]we were sort of like um we were sort of [5:09]observing this like this like chatbot [5:12]boom ahead of you know that was [5:14]happening at the time um and but it was [5:16]very clear that like um chat bots if you [5:19]wanted to build them, and this is funny [5:20]to say in retrospect, required lots of [5:22]data um and required lots of like human [5:24]elbow grease um to be able to get them [5:26]to work effectively. And so like just [5:29]like kind of off the cuff at one point [5:31]was like, oh, like what if you just did [5:34]that? What if you just did the data and [5:36]the the like language data and the the [5:38]human data so to speak for the chatbot [5:40]companies? We were also very lost, by [5:42]the way. I think you probably remember [5:44]we we were we were quite lost mid batch. [5:47]um uh and like many YC companies I think [5:50]and so then we um switched to this like [5:53]concept I think the you know the initial [5:54]idea was like API for um for human tasks [5:57]or something along those lines and uh [5:59]and one night I was just like trolling [6:01]around for domains scaleappi.com was [6:04]available and then we just bought it we [6:07]launched it I think a week later we [6:09]product hunt yeah I remember the the [6:11]product hunt page is still live I was [6:13]reading it last night and I remembered [6:16]the tagline line. It was an API for [6:17]human labor. Like that that that that's [6:19]my recollection of sort of like the like [6:21]distilled insight that you had was like [6:22]what if there is an a what if you could [6:24]call a human with like an API? Yeah. And [6:26]that was I mean I think it was like 3 [6:28]days for us to put up the landing page. [6:31]It launched on product hunt. I think [6:32]this idea captured some amount of [6:36]imagination of the like of the startup [6:39]community at the time because it was [6:41]sort of like this weird form of futurism [6:43]where you have like humans delegated [6:46]like APIs delegate to humans in this in [6:48]this interesting way. It's like an [6:49]inversion of the Yes. Yeah. Humans doing [6:52]work for the machines instead of the [6:53]other way around. Yeah. Yeah. Yeah. It's [6:55]funny because the the initial phase, you [6:57]know, we sort of we just worked with all [6:59]these engineers who reached out to us [7:01]from um from that product hunt which was [7:03]a real grab of use cases, but then that [7:05]was enough for us to raise money at the [7:07]time and like you know uh and to get [7:09]going and then a few months after that [7:12]uh we it became clear that like [7:14]self-driving cars was actually the first [7:17]major application that we needed to [7:19]focus on. And so there were many uh very [7:22]big decisions I would say in the first [7:23]like year or so of the company. One [7:25]thing that was curious is at that point [7:27]there were other solutions that were [7:29]already the game in town like mechanical [7:31]turk from Amazon was sort of the thing [7:32]that people were using but you ended up [7:34]capturing this whole other set of people [7:36]that didn't know about it and you had a [7:38]way better API and you kind of won.

[7:43]Yeah. It was not clear at that point [7:44]because you probably were compared a lot [7:46]with mechanical turd. Yeah. So, [7:47]Mechanical Turk was definitely the sort [7:49]of like um the concept in most people's [7:52]mind at the time. I mean, it was just it [7:54]was kind of one of these things where I [7:55]think a lot of people had heard about [7:56]it, but anyone who had used it knew it [7:57]was just awful. [7:59]And so, it's like whenever you're in a [8:01]space and that's kind of the like that's [8:02]like the thing. It's like people mention [8:04]a thing, but it sucks, that's usually [8:06]like a pretty good sign. Um, and so that [8:08]was that was enough to give us like [8:10]early confidence. But then I think the [8:11]thing that like really I would say the [8:14]the um the thing that was as actually [8:17]fundamental to the success of the the [8:19]company was actually focusing on this [8:21]like on this like seemingly very narrow [8:23]problem of of self-driving cars. I think [8:26]that um you know I remember very early [8:29]on when it was maybe like six months [8:31]after we were out of YC basically um [8:33]there was another YC company Cruz that [8:35]that had reached out to us on our [8:37]website and sort of like in the blink of [8:39]an eye they became our largest customer [8:41]and they found you just from your launch [8:43]or yeah just yeah I think maybe even [8:46]Google like I it's not even totally [8:47]obvious but just yeah vaguely from our [8:49]launch and vaguely it was actually an [8:50]XYC founder that uh was working at [8:52]Cruise that reached out to us so maybe [8:54]some YC mumbo jumbo. [8:58]We're a ketti. Yeah. Uh, who knows? The [9:01]world works in mysterious ways, but and [9:03]so they grew very very large. So then [9:05]early on we made this decision and I [9:07]remember we we we um went to our lead [9:09]investor at the time and you know we had [9:11]this conversation. It's like hey [9:13]actually we think we should probably [9:14]just focus on this self-driving thing. [9:16]You know it was actually a very [9:17]interesting conversation because the [9:19]reaction was like oh that's just like [9:21]obviously way too small a market. um [9:23]then like you you know you're never [9:25]going to build like a gigantic business [9:26]that way. Um and we were like we think [9:29]it's probably a much bigger market than [9:32]than you think it is because there's [9:35]like you know all these self-driving [9:36]companies are getting crazy amounts of [9:38]funding and the automotive companies are [9:40]doing huge programs in self-driving and [9:42]it clearly is the future. Like it feels [9:43]like something that that um that should [9:46]exist and so we're like if we focus on [9:47]it we think we can build like build the [9:49]business much more quickly. And it's [9:50]funny looking back because both things [9:52]are true. It is both true that it [9:54]enabled us to build the business to be [9:55]to get to scale pretty very quickly and [9:58]it is also true that that was not a big [10:00]enough market to sustain a gigantic [10:02]business. The story of scale in many [10:04]ways is like this progression of like [10:07]how do you continue you know AI is this [10:09]incredibly dynamic space. Um, lots of [10:11]things are constantly changing and um, a [10:13]lot of I think what um, what we pride [10:15]ourselves on at the company is how we've [10:17]been able to um, continue building on [10:20]and and um, contributing to this very [10:22]fastmoving industry. When did you uh, [10:24]become much more aware of the scaling [10:27]laws because you know uh, one of the [10:30]interesting facts that sort of emerged [10:32]is that uh, you're a little bit the [10:34]Jensen Hang of data. I think that in [10:37]self-driving [10:39]um scaling laws were not really a thing. [10:42]Um because and the fundamental the the [10:44]biggest reason actually was that like [10:45]one of the biggest problems in [10:46]self-driving is that your whole [10:48]algorithm needs to run on the car and so [10:50]you're very constrained by the amount of [10:51]compute you have access to and is [10:54]available to you. So like a lot of the [10:57]engineers and a lot of the companies [10:58]working on self-driving never really [11:00]thought about scaling laws. They were [11:01]just all thinking about like, okay, how [11:02]do you keep grinding these algorithms to [11:04]be better and better better that are [11:06]like small enough to fit onto these um [11:08]onto these cars? But then we started [11:10]working with OpenAI in 2019. This was [11:13]like GPD2 era. Um and I would say like [11:17]GPT1 [11:19]GPT was sort of like this curiosity. [11:22]GPD2. Um, I remember OpenAI like they [11:26]would have a booth at these like large [11:27]AI conferences and they would like, you [11:29]know, their demo would be to allow [11:31]researchers to like talk to GPD2 and it [11:34]was like [11:35]mildly like it was it wasn't like [11:37]particularly impressive, but it was like [11:38]kind of cool. is like kind of this thing [11:40]and then um I think by GPG3 [11:44]uh it was sort of this like that's when [11:46]the scaling loss clearly um you know [11:49]felt very real and that was I mean I [11:52]think GPD3 was 2020 um so which is [11:55]actually like long before before the [11:57]world caught on to what was happening [11:59]did did you know as early as 2020 did [12:02]did you have a strong inkling that this [12:03]was really going to be like the next big [12:05]chapter of scale or not until chat GPT [12:07]took off was was 35 or was it four? I [12:11]think that like um in 2020 I think it [12:14]was clear that scaling laws were going [12:16]to be a big thing, but it was still not [12:19]totally obvious. you I remember this [12:21]like interaction, you know, I I got [12:24]early access to GPD3 and then it was [12:27]like in the playground and then I I was [12:29]like playing with it with a friend of [12:31]mine and uh I I told the friend of mine, [12:33]"Oh, you can like talk to this model." [12:35]And during the conversation, um uh my [12:39]friend got like visibly frustrated and [12:41]angry at the AI, but in a way that [12:43]wasn't just like, "Oh, this is a dumb [12:45]like toy." It was like it was in a way [12:46]that was like somewhat personal. And [12:48]that's when I was I realized like whoa, [12:49]this is like somehow qualitatively [12:51]different from anything that had existed [12:52]before. I feel like it was passing the [12:54]touring test at that point. Kind of. It [12:57]was like semblances. Yeah, semblance. It [12:59]was like sort of like the the the [13:01]glimpses of it potentially passing the [13:03]touring test, right? But I think the [13:04]thing that really [13:06]um caused the recognition of I would say [13:09]generative AI, which is still even the [13:11]term in some ways, it was really Dolly I [13:13]think that that um that convinced um [13:16]that convinced everyone. But I think I [13:19]think my my personal um journey was like [13:22]GBD3 [13:24]like was like highly interesting and [13:26]then and so it was like one of many bets [13:28]at the company and then in 2022 [13:33]over the course of Dolly and and then um [13:36]and then later chatbt and you know um [13:38]GP4 etc. and we worked with open eye on [13:41]instruct GBT which is kind of the [13:42]precursor to chat GBT. it became very [13:44]obvious that that was like the at the [13:45]farm moment for the for the company and [13:47]for frankly the world. That's when we [13:48]saw it as well with the big shift in [13:50]companies because it was that 3.5 moment [13:53]release end of uh 2022 and we started [13:56]seeing a bunch of companies and smart [13:57]people changing directions and pivoting [14:00]their companies in 2023 and that was [14:03]that moment this dynamic that you [14:05]referenced which is kind of the you know [14:07]scales the NVIDIA for data kind of [14:09]thing. Um I think that became quite [14:14]obvious. Um I would say GPD4 really was [14:19]the moment where it was like it was like [14:21]wow this is like like scaling laws are [14:23]very real. The need for data will [14:25]basically you know grow to consume you [14:29]know all available information and and [14:32]um and knowledge on that humans have. [14:34]And so um it was like wow this is this [14:37]is like this like astronomically large [14:39]opportunity. Yeah for seemed like the [14:41]first time it was something that you [14:43]could uh get to not hallucinate [14:45]basically ever you could actually have a [14:47]zero uh hallucination experience in [14:50]limited domains and which is we're still [14:52]sort of in that regime even at this [14:54]point. You know, the classic view is [14:56]that if it's hallucinating, you're not [14:58]giving it the correct data in the prompt [15:00]or context or uh you're trying to do too [15:03]much in one step. Yeah. I mean, I think [15:05]I think like the reasoning paradigm is [15:08]is is has a lot of lags and it's [15:10]actually been interesting this last era [15:12]of the of model improvement because um [15:16]uh the gains are not really coming from [15:18]pre-training um which is so so we're [15:21]like moving on to a new scaling curve of [15:23]of reasoning and reinforcement learning [15:25]but it's it's like shockingly effective [15:28]um and and I think that you know it's [15:31]the the analogies between like AI and [15:34]and Mors law are pretty clear which is [15:35]like you know you'll get on different [15:37]like technical curves but like if you [15:39]zoom way out it'll just be feel like [15:40]this like smooth improvement of models.

[15:42]One of the things that uh has been [15:43]popping up with some of the like really [15:46]big well-known rappers is they're [15:48]getting access to full parameter [15:50]finetunes of the base models especially [15:52]the frontier base closed source models. [15:55]Is that like a big part of your business [15:56]or you know something that people are [15:58]sort of coming to you for just like [16:00]these verticalized full parameter [16:02]fine-tuned like data sets? Yeah, I think [16:05]this is going to be a like blueprint for [16:08]the future, right? So right now I mean [16:09]like the total number of large scale [16:12]parameter fine tuned or reinforcement [16:14]fine-tuned models is like still pretty [16:15]small but if you kind of think about it [16:17]like that like one version of the future [16:21]is that every firm's core IP is actually [16:26]their specialized model or their own [16:28]fine-tuned model and just in the same [16:31]way that like you know today you would [16:34]generally think that the co the the uh [16:36]IP of most tech companies is their [16:38]codebase [16:39]um in the future you would generally [16:41]think that their their their specialized [16:43]IP might be the model that powers all of [16:45]their all their internal um workflows.

[16:48]And what are the special things they can [16:50]add on top? Well, they can add on um [16:52]data and environments that are somehow [16:54]specific very very specific to the [16:57]day-to-day problems or information or [16:59]challenges or business problems that [17:00]they see um on a day-to-day level. And [17:03]that's the kind of like really gritty [17:06]real world information that you know [17:09]nobody else will have because nobody [17:10]else is like doing the same the exact [17:12]same business motion as them. There's a [17:13]lot of weird tension in that though. Um [17:16]I remember uh friends of ours from one [17:17]of the top model companies came by and [17:20]they were like hey do you think YC and [17:22]YC companies would give us their evals [17:25]so we could train against it? And we [17:27]were like no dude what are you talking [17:29]about? Why why would they do that? [17:31]Because that's like their moat. And then [17:32]I guess now that based on this [17:34]conversation, it's actually I mean eval [17:36]are pretty important as a part of RL [17:38]cycles. And then even the eval are not [17:40]really uh the valuable part. The [17:42]valuable part is actually the like [17:44]properly fine-tuned model for your data [17:46]set and your set of you know sort of [17:48]problems. Yeah, it's like these Lego [17:50]blocks, right? If you have the data and [17:51]you have the environments and then you [17:53]have the you have you know a base model, [17:55]you like you know can stack those on top [17:57]of each other get get a fine tuned model [17:59]and obviously the eval are important. [18:00]This is some of the tension and this is [18:02]basically you know in a nutshell the [18:04]sort of like um does AGI become a Borg [18:07]that just sort of like swallows the [18:08]whole economy in like you know has one [18:10]firm or do you still have a specialized [18:12]economy? My belief generally speaking is [18:14]that you you still do have a specialized [18:17]economy like the like these models are [18:20]platforms but the like like alpha in the [18:23]modern world will be determined by you [18:26]know to what degree you're able to sort [18:28]of like encapsulate your business [18:30]problems into data sets or environments [18:32]that are then conducive towards building [18:35]like you know differentiated models or [18:37]differentiated AI capabilities. Yeah, [18:38]that's why asking for eval was so crazy [18:40]to me because it's like okay you get the [18:42]evals the base model is way better and [18:45]then not you know now all your [18:46]competitors have exactly uh the same [18:49]thing that used to be your advantage. I [18:51]think we will undergo a process in AI [18:54]where we learn what the bright lines [18:57]are, right? I mean, I think that like [18:59]it's like very obvious and intuitive to [19:02]tech companies that they should not give [19:03]away their codebase and they should not [19:05]give away their database. Like they [19:06]should not give away their data, they [19:07]should not give away their codebase. The [19:08]analoges of that in a you know highly AI [19:11]fueled economy I think will identify [19:13]over time but are yeah the evals your [19:16]data your environments etc. I think you [19:18]have a very uh techno optimistic view of [19:20]what the future is going to be with how [19:24]jobs are going to be shaped. Can you [19:26]talk more about that? Because I think [19:27]you hinted at it before. It's going to [19:29]be more specialized. It's not that all [19:31]these jobs are going to go away, right? [19:33]First off, it's undeniably true that [19:36]we're we're uh at the beginning of an [19:39]era of like a new a new way of working [19:42]like like you know this there's this [19:44]term that people have used a long time [19:45]which is like the future of work. Well [19:48]uh we are like entering the future of [19:50]work or the certainly the next era and [19:52]so work fundamentally will change but I [19:54]do think um humans own the future and we [19:58]we are we are like uh we have a lot of [20:00]agency actually and a lot of a lot of [20:03]choice in how this sort of like [20:05]reformatting of of work or how the [20:07]reformatting of sort of like workflows [20:09]ends up playing out. You know, I think [20:10]you kind of see this play out in uh in [20:13]coding right now. And I think coding in [20:15]some ways is is really the sort of like [20:18]um case study for other fields and and [20:21]other you know other areas of work where [20:23]sort of the the initial phase is this [20:26]sort of like assistant style thing where [20:29]um you know you're kind of doing your [20:31]work and then the models are kind of [20:32]like assisting you a little bit here and [20:34]there and then you go to a you know the [20:36]sort of like cursor agent mode kind of [20:38]thing where you're you're like um [20:40]synchronously asking the the models to [20:43]like carry out these workflows and [20:44]you're sort of like you're you're [20:46]managing like one agent kind of or [20:48]you're sort of like you're kind of like [20:49]pair programming with a single agent and [20:51]then and then now with like codecs or [20:54]other systems like it's it's very clear [20:55]the paradigm is like oh you have this [20:57]like you have this like swarm of agents [20:59]that you're going to deploy on like all [21:00]these various tasks and you're just [21:01]going to like sort of like you know dep [21:04]like um give all these tasks and you'll [21:06]have this sort of like um this this [21:08]cohort of of agents that are sort of [21:10]like you know doing this work that you [21:12]you think is appropriate. [21:13]And that last job um uh has a has a [21:17]semantic meaning in the in the current [21:19]workforce. It's a manager. You know, [21:20]you're basically managing this sort of [21:22]like this set of agents to do um actual [21:26]work. And so and and I think that like [21:28]AGI or you know AGI or doomers or [21:31]whatnot like they take this view that [21:33]like oh even this job of like managing [21:35]the agents will just be done by the [21:36]agents. So like humans will be taken out [21:38]of the of the process entirely. But our [21:41]belief, my personal belief is that you [21:43]know this is um management is very [21:47]complicated. Um management is also like [21:49]more about like what's the vision that [21:51]you have and what's the sort of like [21:52]what's the like end result you're aiming [21:54]towards and those will be fundamentally [21:56]I think like you know we have a human [21:59]demand and human desired driven economy.

[22:01]So those will be driven by humans. And [22:03]so I think the terminal state of the [22:06]economy is just is largecale humans [22:09]manage agents in a nutshell. I have a [22:11]funny story where um founder friend of [22:13]mine is trying to promote uh one of his [22:16]you know junior employees but they're [22:18]really really smart and they're working [22:20]on the agent infrastructure and then he [22:22]was like hey do you want to like you [22:25]know I'm looking for someone who could [22:26]step into management. You've never [22:28]managed people before. or do you you [22:29]know if we hired some people uh under [22:32]you like how would you feel about that [22:34]and uh this you know uh mid20some really [22:37]smart you know sort of do he's just like [22:40]he's an engineer and he's like why would [22:42]I do that like just give me like more [22:44]compute like you know the model like [22:47]look at what just happened to the model [22:48]literally like last month and you know I [22:50]didn't have to do anything it just [22:51]started doing things that it couldn't do [22:53]a month ago why would I want to manage [22:55]people like just give me like I will [22:57]just manage more agents for and it's [22:59]fine. Okay. So, what are the unique [23:01]things that that um that humans will do [23:04]over time? I mean, I think this like [23:06]this like element of vision um is very [23:08]important. This element of like kind of [23:10]like debugging or sort of like um fixing [23:13]when things go wrong. Like most of a [23:17]manager's job, speaking as a manager, is [23:20]is just like putting out fires, dealing [23:22]with problems, dealing with like like [23:23]issues that come up. Like I think [23:25]intuitively, you know, I the idealistic [23:28]manager job seems like this very cushy [23:29]job because you're like, "Oh yeah, all [23:31]the other people do all the work and I'm [23:32]just sort of like I just vaguely [23:34]supervise and then the reality is [23:36]obviously like highly chaotic." I think [23:37]people often jump to this like, you [23:39]know, extreme reality where it's like, [23:41]oh yeah, these like, you know, you're [23:43]just going to manage the agents and [23:44]you're going to sort of like live this [23:46]like, you know, kind of Victorian life [23:48]where all your problems are solved. But [23:49]but no, I think it's still going to be [23:51]pretty complicated like getting agents [23:52]to like coordinate well with one another [23:53]and like coordinating the workflows and [23:55]and and debugging the issues that come [23:57]up like these are still complicated [23:58]issues and you know having seen what [24:01]happened in self-driving which was more [24:03]or less that like you know it's easy to [24:04]get to 90% very very hard to get to 99% [24:08]I think that like something similar will [24:09]happen as with large scale agent [24:11]deployments and that like you know final [24:13]10% of accuracy will be like you know [24:16]will require a lot of work. Yeah. Even [24:17]for uh self-driving cars right now, [24:19]there's the remote assist for all these [24:21]super etch case. So there's still a [24:24]human at the end managing the car. Yeah. [24:26]And the ratio, by the way, I mean um the [24:29]companies don't publish them, but I [24:30]think the ratio is something like five [24:32]cars to to one teley operator um or or [24:36]maybe even less than maybe three cars [24:38]per teley operator. So um the ratio is [24:41]like you know much lower than people [24:44]think. I think that like humans are much [24:45]more involved even in self-driving cars [24:47]than I think most people appreciate. I [24:50]mean, which if you put it in that [24:51]perspective, I think it's still very [24:53]optimistic. It's just the output of [24:55]getting rides instead of doing in [24:56]today's world, if you're a Uber driver, [24:58]you just do one car. In this world, you [24:59]can do five cars, right? Well, you have [25:01]to believe for this like for an [25:03]optimistic version of the future where, [25:05]you know, unemployment is still low, [25:06]etc. You just have to believe that [25:08]humans are like almost insatiable in [25:10]their desire and their demand. um and [25:13]that like you know prices will go down, [25:15]things will become you know uh the the [25:18]economy will become more efficient and [25:20]we'll just like want more. And I think [25:22]this has been a pretty reliable trend [25:24]for like the history of humanity is that [25:26]like you know um we have somewhat [25:28]insatiable demand. Um, and so I have I [25:32]have like conviction that like you know [25:33]the economy can kind of get as efficient [25:35]as it needs or as it like can get like [25:37]hyper hyperefficient and then human [25:39]demand will just like continue to sort [25:41]of like fill the bucket. Yeah. In the [25:42]20th century uh you know when you said [25:45]computer maybe early 20th century people [25:48]didn't think of like a computer as it is [25:50]today. They thought of a human being [25:53]that would sit in front of a punch card [25:54]tabulator and that was like what a [25:57]computer was doing. I mean title. It was [25:59]literally that was a real person's job. [26:02]And then of course now today it's like [26:04]where are all the computers? Well, [26:05]they're actually real computers now. I [26:07]don't know. It's that was the Apollo [26:08]mission. It was a bunch of uh people [26:11]just crunching numbers with the [26:12]trajectories of uh of the Apollo and [26:14]that was it because the uh computer that [26:17]went on the uh rocket is actually was a [26:19]microcontroller with I think only like [26:21]single digit hertz. It was like very [26:23]tiny amount of computations. It was just [26:25]humans doing it. Totally. and and even [26:27]this like I mean I think the concept of [26:30]being a programmer is somewhat is like [26:32]highly esoteric um in the sense that [26:34]like oh you're like writing the [26:36]instructions for these like machines to [26:38]just like you know just continue do [26:42]repetitively and in some ways it's like [26:43]the leverage boost that all humans will [26:46]get is like similar to the leverage [26:47]boost that like programmers have had [26:49]historically for a long time I think a [26:51]like a lot of people in Silicon Valley [26:52]say this like the the the closest thing [26:55]to alchemy in our world preai, let's [26:58]say, is programming because you sort of [27:00]like you can do something that uh [27:02]creates like like an infinite there's [27:04]these infinite replicas of whatever you [27:07]build and they can sort of like run an [27:09]infinite number of times and um and I [27:12]think the entire human workforce will [27:14]soon see that kind that large of a [27:18]leverage boost which is extremely [27:20]exciting because I think that like [27:22]programmers are are are um have like [27:25]benefited Ed over the past few decades [27:28]from this like unique perch where they [27:29]they have like you know one 10x or 100x [27:33]engineer can like can build something [27:35]like absolutely incredible and like very [27:37]very valuable and like very um uh [27:40]shockingly productive and all of a [27:42]sudden I think like like humans in all [27:45]trades I think will like gain this like [27:46]level of leverage. Alex, I'm curious to [27:48]return to a point that you made earlier [27:50]about like how scale has kept [27:51]reinventing itself. If you had to like [27:53]describe the arc of scale like what's [27:55]what's what's the story and what were [28:00]the turning points? Our initial business [28:01]was all around um you know producing [28:03]data um you know generating data for [28:06]various AI applications and primarily [28:08]self-driving car companies right for for [28:09]the early years it was really like [28:11]you're saying you're really focused on [28:12]on that. Yeah for the first like three [28:13]years fully focused on that. One of the [28:15]properties of focusing on that business [28:17]of building that business is over time [28:19]you know we had this like obligation to [28:22]really like get ahead of most of the [28:25]waves of AI if that makes sense because [28:27]you know for AI to be successful in any [28:30]vertical area it needed data and so like [28:33]our demand for our our products would [28:36]preede a lot of times the actual sort of [28:38]like evolution of AI into those [28:40]industries. So, you know, as an example, [28:43]we started working with OpenAI on [28:44]language models in 2019. Um, we started [28:47]working with the DoD on government AI [28:49]applications um and defense AI [28:52]applications in 2020. This is like long [28:54]before I think the you know recent sort [28:56]of like dronefueled um you know AI uh AI [29:01]craze in the in the Department of [29:02]Defense. we started working with [29:04]enterprises long before um there was [29:06]sort of like this uh you know the recent [29:08]sort of like larger waves around uh [29:09]enterprise AI implementation. So um [29:12]almost uh uh sort of systemically or or [29:16]intrinsically we've had to uh basically [29:19]build ahead of the waves of AI. I think [29:22]this actually quite similar to Nvidia. [29:24]you know, whenever like Jensen gives his [29:26]annual presentations about, you know, [29:28]Nvidia and its two trends outlook, like [29:30]he always is so ahead of the trends. Um, [29:33]and that's because he has to get there [29:34]on the trend before the trend can even [29:36]happen. That's I think been one um one [29:40]way in which our businesses continue to [29:41]adapt because AI is like this, you know, [29:43]it's this this like it's the fastest [29:45]moving industry I think ever um in the [29:47]history of the world. And so you know [29:49]that each each turn um each evolution uh [29:52]has been has moved incredibly quickly. [29:54]The other thing that that happened late [29:56]2021 early 2022 um we started working on [30:00]um applications and so we started [30:02]building out uh AI based applications [30:05]and now u more much more so uh agentic [30:08]workflows and agentic applications um [30:10]for enterprises and government [30:12]customers. And this was an interesting [30:14]evolution of our business because [30:15]because historically like our core [30:17]business is highly operational. You [30:19]know, we build this like data foundry. [30:21]We have all these processes to produce [30:23]data. Um it's a very operational process [30:25]that involves like lots of humans and [30:26]human experts to be able to produce data [30:29]with quality control systems in place. [30:31]That highly operational business um and [30:34]the success of that business is what [30:35]created the momentum for us to you know [30:38]sort of dream about building an [30:40]applications business. when we went into [30:42]it, [30:43]uh, I had studied other businesses that [30:46]had basically successfully um, added on [30:51]very different businesses and what are [30:53]sort of like the unique traits or or why [30:56]do some of those work and one of them [30:58]that is probably the most interesting [31:00]um, I think is like the most singular in [31:02]modern uh, modern business history is [31:06]um, Amazon building AWS. You know, if in [31:09]2000 you had written a short story that [31:12]said that like, you know, this large [31:14]online retailer would build this like [31:16]largecale cloud computing rent to server [31:19]business. Like it would seem like [31:21]nonsensical. I remember when they [31:22]launched AWS in 2006, Amazon stock went [31:26]down because all the analysts thought it [31:28]was such a terrible idea. It never been [31:30]done before. It just like it doesn't [31:32]seem related at all to their core [31:33]business. um it has it's like this like [31:36]weird thing but the sort of like wisdom [31:38]of that was I think twofold. I think [31:40]like first um and uh from talking to [31:44]people who are like there at the out you [31:45]know the sort of like the genesis moment [31:47]of this business like one thing probably [31:50]the most important thing was that they [31:51]had conviction that that the the sort of [31:54]like underlying business model of AWS [31:56]would basically be this like this like [31:57]infinitely large and growing market like [32:00]that market would would literally grow [32:02]forever. there would be like this like [32:04]exponential of the amount of compute [32:05]that needed built up needed to be built [32:07]up in the world and um if you did that [32:10]there was like sufficient cost of you [32:11]know cost advantages from economies of [32:13]scale I think like startups you know you [32:15]kind of like [32:16]um uh you kind of have to like switch [32:19]modes at a certain point where like [32:20]early on you're trying to go for very [32:23]very narrow markets like almost the [32:24]narrowest markets you can and then [32:26]you're just trying to like gain momentum [32:28]and then sort of like slowly grow out [32:29]from those hyper narrow markets and then [32:32]um at some point you if you like have [32:34]ambitions to be a hundred billion dollar [32:36]company or more then you have to sort of [32:37]like switch gears and say where are the [32:39]infinite markets um and how do you build [32:41]towards those infinite markets and so um [32:44]this was sort of like uh the moment [32:46]where we realized that and and the [32:48]simple realization was that every [32:50]business and every organization was just [32:52]going to have to reformat their entire [32:53]businesses um with AIdriven technology [32:57]um and now obviously like agent driven [32:59]technology and that would just be like [33:01]Over time that would swallow the entire [33:03]economy and so it was like another one [33:05]of these like okay that's an infinite [33:07]business to build out AI applications [33:09]and AI deployments for large enterprises [33:11]and governments. I think a lot of people [33:13]don't realize that you guys are in the [33:14]middle of this transformation. They [33:16]still think of scale as the data [33:17]labeling company but like if you fast [33:19]forward 10 years do you think most of [33:23]scale will actually be the agent [33:26]business? Yeah, it's it's growing much [33:28]faster at this point. And I think it it [33:30]it's an infinite market. So the crappy [33:32]thing about most markets is that they [33:34]have like a pretty shallow S-curve. Um [33:36]but then you know you look at [33:37]hyperscalers or like you know these like [33:40]mega cap tech companies and they just [33:41]have like these like ridiculously large [33:43]markets. So you really want to get into [33:45]these these these like um infinite [33:47]markets. So our strategy so far has been [33:49]to focus on building use cases for you [33:52]know focus on a small number of [33:54]customers and um and be quite selective. [33:56]So we work with you know the number one [33:58]pharma company in the world the number [33:59]one telco in the world the number one uh [34:01]bank the number one um healthcare [34:03]provider um and we work a lot with the [34:06]US government you know the department [34:07]department of defense and and other [34:09]government agencies and um the whole [34:12]thing is like how do we take a very [34:13]focused approach towards building um [34:16]stuff that resemble you know real [34:19]differentiated AI capabilities and all [34:20]this I think sounds somewhat tright but [34:22]but um we have this multiund million [34:26]business in building all these [34:27]applications. By my account, I think [34:29]it's it's one of the largest AI [34:31]application businesses um in the [34:33]industry. Certainly what our investors [34:34]tell us and it's fueled by our [34:37]differentiation in the data business [34:38]because our belief fundamentally is that [34:42]um kind of what we talked about before [34:44]the the end state for every enterprise [34:47]or every organization is um some form of [34:51]specialization um imbued to them by [34:54]their own data. Our day jobs [34:56]historically have been producing highly [34:58]differentiated data for you know these [35:01]like largecale model builders in the [35:02]world and then we can apply that wisdom [35:05]and that capability and those [35:06]operational capabilities towards [35:08]enterprises and their unique problem [35:09]sets and um and give them specialized [35:12]applications. Honestly like it kind of [35:14]sounds like palent here at the like most [35:16]zoomed out level if you sort of like [35:18]squint and that you're a technology [35:20]provider. We're like a technology [35:22]provider to like the most, you know, [35:25]some of the largest organizations in the [35:26]world um with a focus on data. Yeah. Um [35:29]and I think the key difference is like, [35:31]you know, Palunteer um has built a real [35:34]focus around these data ontologies and [35:37]um and really solving this like messy [35:39]like data integration problem for [35:41]enterprises. Um and then our whole [35:44]viewpoint is like what is the like most [35:46]strategic data that will enable [35:48]differentiation for your AI strategy and [35:50]how do we like generate or harness that [35:53]data from within your enterprise towards [35:55]developing that. I guess you will end up [35:57]being pretty big competitors in another [35:59]5 10 years but for now like it's [36:01]basically so green field honestly. I [36:03]mean I think it's an infinitely large [36:04]market the other so you might not ever [36:06]meet actually which is interesting. [36:07]Yeah. Yeah. I I think in practice now we [36:09]actually like frankly we work we're more [36:12]partnered with Palunteer than than [36:14]competitive with them. Yeah. Um and uh [36:16]well that's because the problems at [36:18]these giant organizations are actually [36:19]so massive and intractable that they'd [36:22]throw up their hands. It's like they [36:24]have no shot at ever hiring people who [36:25]could possibly solve the problem. Uh but [36:28]a company like Scale or a company like [36:30]Palunteer can actually hire kind of the [36:32]same kind of people who would apply to [36:34]YC actually. It's kind of like there's [36:36]this Yeah. I don't know the the through [36:38]line in my head right now is realizing [36:39]like you know there's plenty of capital [36:42]and then the limiting agent is actually [36:46]really great technical smart people who [36:49]uh are optimistic and actually work [36:52]really hard. There's like not enough of [36:55]those people. That's true for the world. [36:56]And by the way, I think one of the cool [36:58]things about um agents as we were [37:01]talking about before is that like all of [37:02]a sudden those people get near infinite [37:04]leverage. So, um I think we're going to [37:07]I think that bottleneck gets exploded [37:09]now hopefully um due to due to AI.

[37:11]Again, I I think you know just like how [37:14]in cloud AWS is the largest by far, but [37:16]there's so many other cloud providers [37:18]that actually are all at like like it's [37:20]not a winner take all kind of business [37:22]per se and it doesn't have to be. Yeah. [37:24]Exactly. And and and I think that um [37:26]it's just too big of a market to even be [37:28]close to winner takes all. like I just [37:30]there's no single organization that [37:32]could have the organi um operational [37:34]breadth to be able to to swallow the [37:36]whole market. Talking about uh [37:38]operations, you clearly are living in [37:41]the future which is super cool. I'm sure [37:43]you're running scale with all these [37:46]agents and tools already to make it very [37:49]efficient. Could you share some of the [37:50]things that you're doing internally as a [37:52]company and agents you're adopting so [37:54]you can do more with less people? You [37:56]know, we saw this early because uh when [38:00]when the model developers were starting [38:02]to develop agents and starting to [38:04]develop using reinforcement learning [38:06]like actual you know like reasoning [38:08]models where the the models could [38:10]actually like really do end toend [38:13]workflows. We were uh responsible for [38:16]producing a lot of the data sets that [38:18]enabled um the agents to get there and [38:21]then we saw just like how effective that [38:23]that training process is. I think that [38:25]like the efficacy of reinforcement [38:26]learning for um for agent deployments is [38:30]like is pretty insane. So then once we [38:33]realize that we realize like okay if you [38:35]can actually like you know turn um [38:38]existing human-driven workflows into [38:43]environments and and data for [38:45]reinforcement learning. Um then you have [38:48]this ability to convert these like human [38:51]workflows into human workflows um [38:53]especially ones where you're like okay [38:55]with some level of fault faultiness and [38:56]and okay with a certain level of [38:58]reliability you can convert those into [39:01]um into agentic workflows. So there's [39:02]all sorts of like you know agent [39:05]workflows that that happen in our hiring [39:08]processes and happen um in our quality [39:11]control processes and happen to sort of [39:13]just like automate away certain like [39:15]data analyses um and data processes as [39:18]well as like various like sales [39:19]reporting like it's sort of like [39:21]embedded at you know every major org of [39:24]the company. Um and the whole thing is [39:26]like um it's just like mindset like can [39:28]you identify these like very repetitive [39:30]human workflows and basically like [39:32]undergo this process where you convert [39:33]that into data sets that enable you to [39:35]build automation tools. What do these [39:37]data sets actually look like? I mean for [39:39]browser use is like is it an environment [39:41]and then you know here's a video of a [39:44]human being going through this process [39:46]of like filling out this form and decide [39:48]like yes no on this uh drop down or [39:51]something. I mean you know what's a [39:52]concrete example just for the audience? [39:54]One of the processes that we go through [39:56]is like you know you you um you'll take [39:59]a sort of like full packet of a from a [40:02]candidate and you'll like want to [40:03]distill that into like you know a brief [40:06]of some sort that sort of like gives all [40:08]the salient details about that candidate [40:10]for like decision by a sort of like [40:12]broader committee. Um and these kinds of [40:14]cases you know broadly speaking like [40:16]deep research plus+ kind of things are [40:19]like the lowest hanging fruit. It's just [40:21]sort of like can you take these [40:22]processes that like more or less look [40:24]like you know you have to like click [40:25]around a bunch of places and pull a [40:27]bunch of pieces of information and then [40:28]blend them together and then p produce [40:30]some analysis on top of that like that [40:32]process that fundamental like [40:33]information driven sort of like analysis [40:35]process is the easiest thing to to drive [40:38]via workloads and the kinds of data you [40:41]need are just like you know um uh you we [40:45]call them kind of environments but [40:46]usually it's just like what is the task [40:48]what is the the full um sort of like [40:50]data set that's necessary to conduct [40:52]that task and um what is like the rubric [40:54]for how how you conduct that [40:56]effectively. Do you need RL and [40:58]fine-tuning when like prompt engineering [41:00]and metaprompting seems so good? I think [41:03]that yeah I mean I think I think [41:04]prompting I mean as the malls get better [41:06]prompting will get better but like [41:07]prompting gets you to a certain level [41:09]and then reinforcement learning gets you [41:11]beyond that level. And um actually this [41:14]is a good point. I I think that like [41:15]probably most of the time in our in our [41:17]business it's mostly prompting that just [41:19]is like works really well. I mean that's [41:21]the weird thing is like oh shoot you [41:23]don't have to crack open the models and [41:24]then frankly like the next models are [41:27]going to be so good and then the evals [41:29]are mainly about picking which model or [41:30]you know at what point do you switch to [41:32]the next one. I do think startups need [41:34]basically like a strategy for how they [41:37]like will um walk up the complexity [41:41]curve so to speak. Like you need to like [41:43]you you know whatever product or [41:45]business you build like needs to like [41:47]really um benefit from like the ability [41:49]to like race up this complexity curve [41:51]which is the broad broader curve of [41:53]capability of the models. I mean you you [41:56]actually created this uh leaderboard [41:58]that has a lot of these super hard tasks [42:01]that are trying to go into this next [42:02]curve of uh reasoning. Can you tell us [42:04]about it? One of the things that we [42:05]built um in partnership with the center [42:07]for safety is humanity's last exam. It [42:10]was a funny name. I think unfortunately [42:12]there will be yet another exam beyond [42:14]it. But you know the idea was how like [42:17]let's effectively work with you know the [42:21]the the smartest scientists in the field [42:23]and you know um you know we worked with [42:25]many very brilliant professors but also [42:27]very many like individual researchers [42:29]who are like quite brilliant um and we [42:31]just collated and aggregated this data [42:33]set of what the smartest researchers in [42:35]the world would say the hardest [42:37]scientific problems they've worked on [42:39]recently are. they solved them or they [42:41]sort of like came to the right, you [42:42]know, they were able to solve the [42:43]problems, but they're sort of like the [42:44]hardest problems that they're aware of [42:46]and know of. I was curious how you came [42:48]up with these problems. So, each of the [42:50]professors contributed new problems. So, [42:52]these are not these are problems that [42:53]have never appeared in any textbook or [42:55]any exam ever. They just like came out [42:57]of their brains and they like typed up [42:59]like a new problem like from scratch. Am [43:00]I understanding this right? Yeah. Yeah. [43:01]And the the general guidance was like, [43:03]you know, what has come up recently in [43:06]your research that you think is like a [43:08]is a particularly hard problem, right? [43:10]The problems are stupidly hard [43:12]incidentally. They're like insane. I [43:13]don't know if you guys have looked at [43:15]these problems. They're totally crazy. [43:17]Yeah. It's totally crazy. And by the [43:18]way, like they cannot be searched on the [43:20]internet. It's like you need to have a [43:22]lot of a lot of expertise and actually [43:25]think about them. Yeah. For quite long [43:27]time. Yeah. They require a lot of [43:29]reason. I'm recently like uh right now [43:31]so we have a time limit where the models [43:33]um can only think for I think it's 15 [43:35]minutes or 30 minutes and one of the [43:37]most recent requests from one of the [43:38]labs was like can you please increase [43:39]that time limit to like a day so that [43:42]the model has like up to a day to think [43:44]about the um to think about the [43:46]problems. Um but yeah no they're they're [43:49]deviously hard problems unless you have [43:51]expertise in the specific problem you [43:53]probably don't have a chance of getting [43:55]it right. Um but even this evaluation [43:57]like I think when we first launched it [43:59]um you know and this was just earlier [44:01]this year uh the the best models were [44:04]scoring like 7% 8% on it. Now the best [44:07]models score north of 20%. It's moved [44:10]really really quickly and I think you [44:11]know I think uh do you think we're going [44:13]to get a benchmark saturation for this [44:14]one as well? I think eventually yeah [44:16]it'll it'll be saturated and then we [44:18]have to move on to new evaluations. I [44:19]mean I think the like uh the the the [44:22]saving grace for the naming was that it [44:24]is the last exam. The new eval will be [44:27]sort of like real world tasks, real [44:28]world activities which are sort of like [44:30]fundamentally fuzzier and more [44:31]complicated. Have you solved any of the [44:33]problems yourself, Alex? I know I I know [44:35]you were a competitive math person for a [44:36]long time. Yeah. Yeah. The I mean the [44:38]math problems require a lot of they're [44:40]like very deep in the fields. I think uh [44:43]I was I managed to get a handful but [44:45]like most of them are like hopeless. Um [44:48]yeah, I looked at the ones that the [44:49]models can solve and so [44:52]so that was that was one of the evals [44:53]and we we've produced a number of other [44:55]evaluations but um but yeah, I think [44:57]that like the the in the AI industry [45:00]really I think um continues to suffer [45:03]from a lack of uh very hard evals and [45:08]very hard tests that show really like [45:12]the frontier of of um model [45:14]capabilities. And these eval when you [45:16]get when you build an eval that sort of [45:18]like becomes popular in the industry, it [45:20]has this like deeper effect which is [45:22]that that's all of a sudden the like [45:24]northstar and the yard stick that that [45:26]researchers are trying to um optimize [45:28]for. And so it's it's actually this like [45:30]very gratifying activity. You know we [45:31]built humanity last exam. Um you know [45:35]most of the like all the model providers [45:37]um you know will always report their [45:38]their their their results. There's like [45:40]tons of researchers who are really [45:42]motivated by by doing a good job. I mean [45:45]it's it's uh and and the models are [45:47]going to get you know deviously good at [45:48]like you know frontier research [45:50]problems. I guess Sam's starting to talk [45:52]about you know that sort of stage four [45:53]innovators of AGI is coming and you know [45:56]that's the prognostication for the next [45:59]year. Do you think that's you know [46:00]correct the next 20 12 to 24 months is [46:03]like really the moment that literally [46:05]new scientific break breakthrough um is [46:07]coming from the operation of reasoning [46:09]and these models. I mean, I think it's [46:11]super plausible, you know, in fields [46:12]like biology, and this is probably one [46:14]of the ones that comes up the most, but [46:15]there's like there's probably intuitions [46:17]that the models have about biology that [46:19]humans don't even have um because it's [46:22]just, you know, they have this like [46:24]different form of intelligence, right? [46:25]And so, you'd expect there to be some [46:27]areas and some fields where the models [46:31]um have have some fundamental deep [46:33]advantage versus humans. And so, I think [46:36]it's like very realistic to expect in [46:37]those fields. Biology I think is sort of [46:39]like the clearest one for me. Kind of [46:41]already happened for chemistry. Last [46:42]year the Nobel Prize went to uh the [46:45]Google team Dez and John Jumper with [46:48]Alpha Fold. Yeah, exactly. That was like [46:51]a huge jump. Before that there was like [46:52]this competition where um they were [46:55]trying to get more protein fold [46:58]structures that were going to get solved [46:59]and it was like abysmal and Alphafall [47:03]destroyed it. It's a strange time to be [47:06]a scientist, but an exciting time for [47:08]science. There's this um uh short story. [47:12]It talks about this future where like, [47:14]you know, there's uh effectively AIs [47:17]that are like that are conducting all [47:19]the frontier of R&D research and um and [47:22]scientists, you know, what scientists do [47:24]is they just sort of like look at the [47:26]discoveries that the AIs make and sort [47:27]of like try to understand them. Yeah. I [47:29]mean I think that like very exciting [47:30]time to to see the how the frontier of [47:33]human knowledge expands and then I mean [47:35]I think that'll be great because in [47:36]areas like um in biology will fuel [47:40]breakthroughs in medicine and healthcare [47:41]and and all these other and all these [47:43]other things and then the majority of [47:44]the economy will will chug along you [47:46]know giving humans what they want. China [47:48]open sourcing or Deepseek open sourcing [47:50]their models is like another very [47:52]interesting question like how does that [47:53]play out and um and there's this awkward [47:56]sort of thing that you know the best [47:57]open source models in the world now come [47:59]out of China I mean that's sort of this [48:00]like awkward reality uh to contend with [48:03]and what do you think we can do to just [48:05]make sure that it's the American models [48:06]that are ahead or you know is that [48:08]written in the stars or you know [48:10]something tells me that's not the [48:12]simplest explanation for me about why [48:14]the Chinese models are so good is is [48:16]espionage I I think that there's um [48:19]there's a lot of secrets in how these [48:22]frontier models are trained. Um and when [48:24]I say secrets, they you know it sounds [48:26]more interesting than they are, but [48:27]there's just a lot of task and [48:28]knowledge. There's a lot of like you [48:29]know tricks and small um and intuitions [48:33]about where to set the hyperparameters [48:34]and like you know ways to make these [48:36]models um work and to get the model [48:38]training to work. the Chinese labs have [48:41]been have been able to move so quickly [48:42]and accelerate and make such fast [48:44]progress. Um whereas some even like very [48:47]talented US labs like have made progress [48:49]less quickly. And I just purely think [48:51]it's because you know a lot of the the [48:53]secrets about how to train these models [48:56]um you know those secrets leave the [48:58]frontier labs and make their way back to [48:59]these Chinese labs. Um I I think the the [49:03]only way to model the future is that [49:04]China has pretty advanced models. Um, [49:08]you know, the Solace right now is [49:10]they're not the best models. Um, they're [49:12]sort of like a half step behind, let's [49:14]say. Um, but, uh, but it's tough to [49:17]model what'll happen when it's sort of [49:18]truly neck and neck. We're very behind [49:20]on energy production, which is just pure [49:23]regulation, like that could be fixed in [49:25]2 seconds, but, you know, hasn't been [49:26]yet. That's a huge problem. I mean, if [49:28]you look at, you know, not that the past [49:30]will be a predictor of the future. If [49:31]you look at what US total grid [49:34]production looks like, it's like looks [49:35]flat as a pig. And if you look at um you [49:38]know Chinese uh aggregate uh you know [49:41]grid production it's like you know it's [49:42]doubled over the past decade. It's just [49:44]like it's just this like straight up I [49:46]saw that and it's astonishing. It's I [49:48]mean that's just a policy failure. China [49:50]just you know the vast majority of that [49:52]is coal and coal's growing in China and [49:55]um in the United States uh actually [49:57]renewables have grown a lot but [49:59]renewables trade off against the uh the [50:01]sort of fossil fuels. So we've sort of [50:03]like done a done a transition of our of [50:05]our um energy grid whereas they're just [50:09]continuing to compound. Let's say we [50:11]have this issue on power production but [50:13]we're we're advantage in chips. I think [50:15]like net net we will come out ahead on [50:16]compute. Um if you look at data I mean [50:20]this goes towards a lot of the questions [50:21]you've been you've been asking about but [50:22]like I mean I think China is like [50:24]fundamentally very well positioned on [50:26]data. Um it's weird to say because [50:28]obviously like you know we help all the [50:30]American companies with data in China [50:31]they can ignore copyright or other [50:34]privacy rules and and they can sort of [50:36]um you know build these large models [50:38]without abandon. And then and then the [50:39]second issue is that um there's actually [50:42]large scale government programs in China [50:44]for data labeling. Um there are uh you [50:47]know seven data labeling centers um like [50:51]in various cities that have been started [50:53]up by the government itself. There's [50:55]large scale subsidies for um for AI [50:58]companies to use data labeling a voucher [51:00]system. In fact, there's like college [51:01]programs because you know one of the [51:03]interesting things is in China like [51:05]employment is such a large national [51:06]priority that they like you know when [51:08]they have a strategic area like AI [51:10]they'll like figure out okay what are [51:12]all the jobs and they'll like create [51:13]these like funnels to um to to create [51:16]those jobs. And then we're seeing this [51:18]in robotics data too where like there's [51:20]the already in China there are like [51:22]large scale factories full of robots [51:24]that just go and collect data. Um and uh [51:27]and strangely enough like even a lot of [51:30]US companies today actually rely on data [51:33]from China in training these like [51:35]robotics foundation models. Long story [51:36]short, I think China likely has a data [51:38]an advantage on data and then the [51:40]algorithms um you know the US is is on [51:44]net much more innovative but if [51:46]espionage continues to be a reality then [51:48]like you know you're basically even on [51:49]algorithms. So, um, so it's hard to [51:52]model, but I think that probably like, [51:54]you know, it's like 60 40 7030 that the [51:57]United States like has like an [51:59]undeniable continued advantage, but [52:01]there's like a lot of worlds where China [52:03]just like catches up or potentially even [52:06]overtakes. I mean the the scary thing [52:08]for me is you know watching Optimus or [52:10]YC has uh some robotics companies like [52:12]Weave Robotics and you know we look at [52:14]those things the software can be as good [52:17]or better than anything coming out of [52:18]China but when it comes to the hardware [52:20]it's like bomb cost over here 20,000 [52:22]30,000 bucks like you can't you know we [52:25]can't even make like high precision [52:26]screws over here and then over there the [52:29]same m the same robot the embodied robot [52:31]could be made for like I don't know 2 [52:33]3,000 $4,000 right it's like you just [52:35]walk down a street in Shenzen Zen and [52:37]like they they got it, you know, and so [52:39]how do you compete against that at sort [52:40]of that at a state level? The degree to [52:43]which China is incredible manufacturing, [52:47]I mean that's a that's a very big [52:48]problem. Um and it relates to defense [52:52]and national security. It's a [52:54]fundamental issue uh because on some [52:57]level defense and national security will [52:58]boil down to which countries have more [53:01]things that like can deter conflict or [53:03]can can go into a you know can can shoot [53:06]other things down. Yeah. I don't think [53:07]it's going to be fighter jets and [53:09]aircraft carriers anymore. I mean it's [53:11]probably going to be you know this micro [53:13]war of it's like hyper micro. It's [53:16]drones and embodied robots and I mean [53:18]Yeah. Exactly. Drones embodied robots. [53:20]Cyber warfare. the um cold war era um uh [53:26]philosophy of like you know you build [53:27]like bigger and bigger bombs. Um it's [53:29]like the exact opposite of that. It's [53:31]actually like it's like the [53:32]fragmentation and uh and and move [53:35]towards sort of like you know smaller [53:37]more nimble attraable resources. Um is [53:39]the is the that that's like one of the [53:42]big picture trends I would say. Um and [53:44]then the other big picture trend is just [53:45]what we believe which is uh the move [53:48]towards uh agentic warfare or agentic [53:51]defense which is basically you know if [53:53]you if you actually mapped out the what [53:57]warfare looks like today or like what [53:59]like the um you know the actual process [54:02]of a conflict um you know if you look at [54:05]Russia Ukraine or other conflict other [54:07]conflict areas like the decision-making [54:09]processes are driven are remarkably [54:12]manual and human driven. And it's just [54:14]like all these all these like very [54:16]critical battle time decisions are made [54:19]like with very limited information [54:21]unfortunately um uh in these like very [54:24]manual workflows. And so it's very clear [54:26]that that um if you used AI agents, you [54:29]would have perfect information and you [54:31]would have uh immediate decision-m and [54:34]so that you know it's we're going to see [54:36]this like huge shift towards um [54:39]agentdriven uh warfare and agent-driven [54:42]um conflict and it has the potential of [54:45]turning these conflicts into these like [54:46]almost incomprehensibly fastm moving uh [54:49]kinds of scenarios. And that's something [54:52]that you guys are actively working on, [54:53]right? Can you is there anything that [54:55]you can talk about? I assume some of it [54:57]is classified, but yeah. Yeah. So, one [55:00]of the things we're doing is we we're [55:01]building this uh this system called [55:03]Thunder Forge um with uh the Indopacific [55:06]Command um out in out in Hawaii. It's [55:08]responsible for the sort of Indopacific [55:10]region and it is the flagship DoD [55:13]program for um using AI for military [55:16]planning and and operations. So, we're [55:18]basically doing exactly what I said. We [55:20]are we take the h the existing human [55:22]workflow the military works in a what's [55:24]called a doctrinal way or they're sort [55:26]of like governed by the doctrine of this [55:29]like you know very established military [55:30]planning process and you just convert [55:33]that into you know a series of agents [55:36]that work together um and and conduct [55:38]you know the exact same task but it's [55:39]just like all agent driven and then all [55:41]of a sudden you you turn these like um [55:44]very critical decision-making cycles [55:46]from you know 72 hours to 10 minutes And [55:50]it kind of like changes it from um you [55:52]know you know when you play chess if you [55:54]play chess versus a human they have to [55:56]spend all this time thinking um you know [55:58]you you know it's sort of this like slow [55:59]game and if you play chess against a [56:01]computer it's just like these immediate [56:02]moves back and it's like this sort of [56:04]like unrelenting form of of warfare. I [56:07]mean some of it is like the being able [56:09]to see the chain of thought immediately [56:10]was is the most powerful thing. Yeah. [56:13]Like cuz is you know I don't want the [56:15]answer. I want to see how you got there. [56:17]And then actually seeing the reasoning [56:19]itself was so powerful. I mean that's [56:21]actually why the um launch of that first [56:24]deepseek was way more interesting [56:26]because uh I think 01 had come out but [56:29]they hid the uh the reasoning and it's [56:31]like no the reasoning is actually a [56:33]really important part of it and the only [56:34]reason why they hid it was they didn't [56:36]want other people to steal it which they [56:38]did anyway. I think that that's that's [56:40]another like um interesting thing about [56:43]this space which is that um you know so [56:47]far you could really model as like [56:48]there's like advanced capabilities um [56:51]and you can try to keep those secret and [56:52]you can try to keep those closed but [56:54]they open over time kind of no matter [56:56]what you do. Well, I mean clearly Alex [56:58]you've done a lot of incredible things [57:01]and transformed your company multiple [57:03]times and you have all these deep matter [57:05]expertise in many areas. you're clearly [57:08]hardcore. Is there advice for the [57:10]audience to be more like you? You know, [57:12]I think that the the the biggest thing [57:15]is um you just have to really really [57:17]really care. Um and I think it's like a [57:21]a folly of youth in some ways that um [57:24]that when you're young like almost [57:26]everything feels like, you know, so [57:28]astronomically important that you just [57:29]like you try immensely hard and you care [57:31]about every detail. You know, everything [57:34]uh matters just way more to you. And I [57:36]think um and I think that that trait is [57:39]really really important and um you know [57:42]it's like just in varying degrees for [57:43]different people. So I wrote this post [57:45]many years ago called hire people who [57:46]give a and it really is pretty [57:49]simple. You notice I noticed, you know, [57:51]when you interview people or when you [57:52]interact with people, you can tell [57:53]people who are just sort of like phone [57:55]it in versus people who sort of like [57:57]they like hang on to their work as like, [58:00]you know, it's like it's like so [58:02]incredibly monumental and forceful and [58:05]important to them that they they do [58:07]great work and it sort of like eats at [58:09]them when they don't do great work and [58:10]when they do great work, they're sort of [58:11]so satisfied with themselves. And so [58:13]there's sort of this like um the [58:15]magnitude of of care. And one of the [58:18]greatest indicators of like a just like [58:20]how much I enjoyed working with people [58:22]or like frankly how successful they were [58:23]at scale was really just this like what [58:26]is what you know to what degree of their [58:29]what degree their soul is invested in [58:30]into um uh into the work that they do. [58:34]And so I think that that you know if you [58:36]were to pick one thing that that [58:38]probably is the sort of like unifier in [58:40]some way. It's like, you know, um I care [58:42]a lot, uh I care a lot about every [58:45]decision we make at the company. Um you [58:47]know, I still review every hire at the [58:49]company. You know, I I we have this [58:50]process why where I approve or reject [58:53]literally every single hire at the [58:54]company. Um uh and and so I care [58:57]immensely and then this and then like I [58:59]work with all these people who care [59:00]immensely and then that enables us to [59:02]really sort of like we um we feel much [59:06]more deeply what happens in the business [59:07]and as a result we sort of like uh you [59:10]know we'll change course more quickly [59:11]we'll learn more quickly um we will [59:14]we'll take our work more seriously we'll [59:15]adapt more quickly and I think that [59:16]that's been quite important to the to [59:18]the success that we've had. Alex, you [59:20]were telling me a story recently that [59:22]stuck with me about how like quite [59:24]recently even even when Scale was a very [59:26]large company, you were personally hand [59:28]reviewing all like the data that was [59:30]being sent to partner companies and [59:31]being like basically like the final [59:33]quality control like you know like you [59:35]know that data point is not good enough.

[59:37]Yeah, exactly. I think a lot of founders [59:40]would probably um would probably uh you [59:43]know agree with this but um what your [59:46]customers feel and when your customers [59:48]are happy and sad like it really like [59:50]gets to you and so when you have when [59:52]you have unhappy customers it's like [59:53]it's like personally very painful thing [59:55]broadly speaking you know we have this [59:58]value at our company um quality is [1:00:00]fractal um and and I do believe that [1:00:03]like high standards sort of like um they [1:00:07]trickle down within an organization and [1:00:10]um you know it's very rare that you see [1:00:12]an organization where like where like [1:00:15]standards um increase as you get lower [1:00:18]and lower down in the organization. You [1:00:20]know most of the time when people [1:00:21]realize their manager or their [1:00:22]management manager or their like [1:00:24]director or whomever don't really care [1:00:25]then they sort of like you know that [1:00:27]that removes the sort of like the like [1:00:30]deep desire to to need to care. Um, and [1:00:33]so it's like incredibly important that [1:00:34]that high standards um, and and this [1:00:37]sort of like this deep um, uh, sort of [1:00:40]care for quality is like this is this [1:00:43]like um, deeply embedded sort of um, [1:00:46]tenant of the entire organization. [1:00:48]Founder mode, man. Founder mode, man. We [1:00:52]got to have you back. Thank you so much [1:00:54]for spending time with us. With that, [1:00:56]sorry we're out of time, but we'll see [1:00:58]you next time. [1:01:00][Music] [1:01:02]Heat. Hey, heat. Hey, heat. [1:01:07][Music]