Jeff Dean's Last Conversation Before Leaving His Job: I Underestimated AI and Seen the Entrepreneur's Only Way to Live

Source | InfoQ

Compilation | Yuqi

A year ago, Google's chief scientist Jeff Dean predicted at the AI Ascent 2025 summit that by 2026, there could be AI systems that can work around the clock and are as capable as junior software engineers.

A year later, six days ago, he admitted in an interview with YC that he had underestimated how fast AI is progressing. The model's ability to handle complex tasks grew much faster than he had anticipated at the time. So, according to Jeff Dean, how fast will AI move forward in the future? How can startups survive in an era where generic models continue to expand the boundaries of capabilities?

Early this morning, this interview had a different weight. Jeff Dean announced that tomorrow will be his last day at Google. After working at Google for 27 years, this legendary engineer, known as the “programmer among programmers” in Silicon Valley and deeply involved in the construction of Google's system architecture and AI technology, co-founded Discovery Loop with long-term partners Sanjay Ghemawat, Oriol Vinyals, and Quoc Le, a public welfare company focusing on cutting-edge research in machine learning, science, and engineering. Google will continue to work with them as a founding investor and cloud computing partner.

According to WIRED, this startup idea actually only surfaced a few weeks ago. In order to retain this core team, Alphabet CEO Sundar Pichai also tried to persuade them to “not lose the job card” during many meetings. But in the end, a few people decided to leave the big company system in exchange for the fun, speed, and freedom that only a startup can have.

Screenshot from: https://x.com/JeffDean/status/2085035498222002595/photo/1

Jeff Dean wrote in his farewell letter that he saw Google grow from a company of just 25 people to a tech giant with more than 190,000 employees. Today, Google has 13 products with over 1 billion users. From search, email, translation, and video to large-scale computing, autonomous driving, and AI systems, the technology he participated in building has spanned almost the entire evolution of Google.

And one of the main reasons that prompted them to leave was precisely inertia, which is difficult for large companies to get rid of. As Oriol Vinyals said, within large organizations, driving any radical change requires overcoming layers of resistance; they want to do something different. What's interesting is that until now, the new company hasn't even had time to recruit people or rent an office. As for who will be the CEO, after a short pause within the team, everyone has their eyes on Jeff Dean — “I think it's me.” he said.

As a result, this interview, published on the eve of Jeff Dean's departure, is like a focused judgment on the next stage of AI as he stands at a turning point in his career. On the program, he and YC partner Diana Hu discussed the paradigm shift in AI from “model centered” to “context engineering,” the huge opportunities that inference hardware is emerging, and how entrepreneurs can find a real living space worth sticking to in an age where generic models are becoming stronger and more applications may be directly incorporated by models.

This article is based on a video compilation of this interview, edited by InfoQ.

Too long to read the edition

Q: Last year you said 2026 would have AI with capabilities close to those of junior engineers. A year has passed, does this prediction punch you in the face?

A: That's pretty accurate, but I've underestimated one thing: the model's ability to handle increasingly complex tasks is growing much faster than I expected. Moreover, this ability is spilling over into fields other than coding, and Agent-based systems are starting to really take off.

Q: What are the bold predictions for 2027?

A: The deep learning system will implement a fully automated problem decomposition and automated experiment cycle: split the problem into sub-problems, run experiments automatically, integrate the results, and obtain an improved system. And this doesn't just apply to machine learning; it can be used in any field of science and engineering with measurable goals.

Q: When Google loaded the search index into memory in 2001, what was the “memory loaded” moment in 2026?

A: Specialized inference hardware with high performance and low energy consumption. Reasoning is the key to making the Agent system reach more people, and dedicated hardware can bring 50x latency improvements and 30-80x energy efficiency improvements. What was eliminated back then was the physical delay in disk addressing; what is being eliminated today is an extra step in general computing. But the stakes are higher this time around, because the Agent requires thousands of consecutive deductions, with each delay accumulating.

Q: What is the biggest misconception people have about AI right now?

A: Everyone thinks the Agent is a temporary task that runs for an hour or two, but with a strong enough underlying model, you can keep the Agent running for days or even weeks.

Q: What did the math of that napkin that gave birth to TPU actually calculate?

A: The voice recognition effect skyrocketed in 2013, and we settled the bill: if every user spoke to the phone for three minutes every day, the CPU couldn't support it at all, so TPU was created. When Transformer came out a few years later, it just became the foundation.

Q: In the AI era of delayed numbers, which number should I remember best?

A: Doing one calculation consumes about 1 pJ, but moving the data into the processor consumes 1000 times that amount of energy. This gap determines that you have to do batching, process many samples together, and divide the cost by “one thousand times divided by the batch size.”

Q: How should founders choose the fields they want to work in?

A: Use the current strongest general model to test the field you want to work on. If the model's success rate is 0% or 1%, that's a good sign; if it's 20%, it's a dangerous sign, indicating that abilities have developed, and the next generation model will soon be covered.

Q: When the Agent has written all the code, what skills become scarce?

A: Taste, know what problems to ask your agent to solve. The first is to rely on experience in solving a large number of problems; the second is to write down the ten things you think may be important in the next 12 months; the third is to do crazy thought experiments, such as “If a transistor fails 20 times a day instead of once every million years, how should the system be designed?”

Diana: At the AI Ascent conference in May of last year, you said that by 2026, there might be an AI system that can work around the clock and is close to that of a junior engineer. How close are we to that prediction now?

Jeff: I think the model has improved a lot in dealing with Agent-based long-running coding tasks. Now it seems obvious that they are indeed quite capable, and depending on your precise definition of “junior engineer,” this prediction now seems quite accurate.

Diana: So what were you underestimating at the time?

Jeff: I think the ability to handle increasingly complex tasks is growing much faster than I expected. Also, in fields other than coding, these Agent-based systems are starting to really gain prominence, and I think this will be an important trend in the future.

Diana: What do you think the predictions for the 2027 edition will be?

Jeff: I think you'll see more automation of machine learning systems themselves. Specifically, it allows the machine learning system to improve its ability by running a large number of experiments, break down the problem into sub-problems, run these sub-problems in a tight automatic test cycle, and then integrate the results to obtain an improved system from this fully automated problem decomposition and automated experiments. And this applies not only to machine learning, but also to other fields of science and engineering. Basically anything with measurable goals can make a lot of progress today.

Diana: In 2001, Google search was still running on hard drives. You and Sanjay settled accounts and realized that at some point in time, the entire search index would finally fit into the memory of every running computer. You made that radical decision to basically work with Sanjay to bring a new set of memory-based, non-harddrive-based versions of search to production within a few days, and it's what made Google what it is today.

History is always repeating itself, so now in 2026, what is that “ready to fit in memory” moment? What is the turning point where everyone here should think and design?

Jeff: The situation is a little different, but what I'm saying is that you'll see more and more high-performance, low-power inference hardware systems appear. Because now everyone realizes that reasoning is the key to making these Agent systems reach more people. Latency is really important, and hardware specialization is the core way for you to outperform general-purpose computing devices like GPUs or CPUs in terms of energy efficiency and latency.

Diana: So you mean we probably don't have to wait any longer?

Jeff: Right, imagine if you could improve latency 50 times, what could you do?

Diana: There are about 6,000 people here. Which of their assumptions about AI are actually wrong?

Jeff: I think people probably haven't fully realized the potential for how long the Agent system can run. Everyone always thinks that an Agent is just a temporary task running for an hour or two. But in fact, in some problem areas, with powerful enough underlying models, you can keep these agents running for days or even weeks to actually solve complex, long-term problems. This is by no means just a “wait a little longer” difference; it is a qualitative change.

Diana: So which missions did you actually run for a few weeks? What are you letting the Agent solve?

Jeff: For example, you could have the Agent implement a new version of the software, rewritten in a different programming language, which might have better security or performance characteristics. It can really do this seriously, and it does it quite reliably.

Diana: Everyone has always known that you're particularly good at “napkin math.” One story about you is that when Google's speech recognition first went live in 2013, you made an estimate: if every Google user spoke to the voice recognition system for three minutes a day on their phone, the size of the entire server would have to double, and the cost was frighteningly high. So you built a custom chip, which is the origin story of TPU.

Jeff: The situation at the time was that our deep learning-based speech model started to work very well, but the computational cost was much more expensive than the old speech system, but it cut the error rate in half. This is equivalent to 20 years of progress in speech recognition, which was achieved in a few months. We only adjusted the model a bit, expanded the scale slightly, and obtained better data. So we started to worry: if voice recognition were to improve dramatically, people would use it more often.

So that rough calculation is actually about: if people start using voice recognition more frequently to dictate emails and talk to their phones, what will happen? After doing the math, we discovered that running with a CPU wasn't enough at the time, so we created TPU, a chip specially designed for low-precision dense linear algebra, which is the core of almost all machine learning algorithms today.

Although it can't do anything other than calculate linear algebra and can't run Chrome or Word, that doesn't matter to machine learning reasoning. The chip came out a few years later, and was 30 to 80 times more energy efficient and 20 to 30 times lower in latency than the CPUs and GPUs of the same generation.

Diana: You definitely couldn't have anticipated at the time that TPU would become so fundamental when the Transformer architecture came out. After all, Transformers only appeared a few years after TPU was invented.

Jeff: That's exactly why we made it a general linear algebra system in the first place. We know that machine learning algorithms are still evolving and cannot be overly specialized, but they also need to be specialized to obtain huge performance benefits. Therefore, we made a large multiplication unit, high-speed memory, and high-speed interconnection, and later TPU was able to efficiently collaborate hundreds or thousands of chips on the same problem. In this way, it has been iterated from generation to generation until today.

Diana: If anyone wants to be a future founder, what kind of “napkin math” should be considered to create something as influential as TPU?

Jeff: I think, think about what problems and bottlenecks you've seen in your field of thinking, and then ask yourself: Is there a very different way of thinking that can improve the solution by an order of magnitude or even two orders of magnitude? Sometimes you're just squinting at a problem and not being anchored in “how was this problem solved today,” but instead thinking about how you would solve it from the first principle, and you can come up with some really good ideas that people never even thought of.

Diana: Many years ago, you wrote a very famous list called “Latency Numbers Every Engineer Should Know”, which lists the time required for various system operations: how long does it take for a cache to go missing, how long does it take to find a disk, how long does it take for a network package to travel from California to the Netherlands... This list later became the bible of countless distributed systems engineers. Now it's time to update the AI version.

Jeff: If you look at what is most important in today's AI systems, you first need to know how much bandwidth is between the accelerator's main memory to on-chip memory to the multiplication unit. You need to know how much energy it consumes to perform a multiplication operation, and you also need to know how much interconnection bandwidth between the chips, and how many chips you can connect to under this bandwidth. Going further, when you need to communicate with 10,000 chips instead of 500, how much will the network bandwidth be reduced. These numbers are the underlying constraints that really determine how you think about problems; they directly determine what kind of thinking you use to solve what kind of problems.

Diana: As you said, now the yardstick for measuring everything is energy. Doing a calculation or math operation consumes about 1 pJ, but the energy consumption for moving data and doing data I/O is 1000 times that.

Jeff: Exactly. That is, the short path of moving data from the HBM on the accelerator to the processor so that it can actually be calculated, the energy consumption is three orders of magnitude lower.

Diana: This gap actually quietly determines which products can be made and which algorithm routes can be followed. Since data mobility is so expensive, if the founder came to you and said he had a “model problem,” but in reality it was essentially an energy issue or data I/O issue. How can you tell the difference between the two?

Jeff: The 1,000-fold difference you just mentioned actually shapes a lot of decisions in machine learning. If it weren't for this thousand-fold gap, you wouldn't need to do batching at all. However, the reality is that you have to combine many samples and many tokens to dilute the cost of data movement. Instead of paying a thousand times the cost, you pay the energy consumption of “one thousand times divided by the batch size”. Batching isn't friendly for low latency scenarios. Therefore, these hardware-level energy consumption considerations profoundly influence every decision we make when building the upper layer system.

Diana: A very specific example is how to train a model, divide the data set into batches and run epochs. Many people think this is a model problem, but in reality this is essentially a system data I/O problem, right?

Jeff: Yes, you have to assemble batches to make the hardware run more efficiently. Ideally, you'd hate to train with a batch size equal to 1, but that's so inefficient, so now everyone uses a big batch.

Diana: I've heard you have a habit of disappearing for a while and then coming back with a great plan. So is it possible for you to spend a few weeks to completely complete the training with a batch size equal to 1?

Jeff: Honestly, I've been thinking more about this lately. Reasoning is an interesting question because you really want extremely low latency, training doesn't necessarily require that low latency, and reasoning is exactly the least useful area of batching. The core is to keep data handling to a minimum, and then think about extremely low precision computations. Maybe there is no need to support so many different types of precision formats. If you have a good answer about what kind of accuracy you need, then simply solidify it into the hardware, don't care about anything else.

Diana: I'm reminded of a core analogy a famous computer scientist told me: the entire AI process is essentially a huge compression problem. Because in order to losslessly compress data and then recover it, you basically have to really understand it first.

Jeff: Right, if you really understand the data, you should be able to compress it very well.

Diana: The Transformer architecture is exactly one compression method that has proven to be particularly easy to use.

Jeff: I owe it to my colleagues.

Diana: In the past, when we talked about AI advancements, we often meant that the model itself got better. However, it has become increasingly clear in recent years that it is not just model size, number of parameters, or amount of data, but peripheral capabilities such as search, tool call, memory, and agent tools, which are converging into what people call “context engineering.”

Jeff: The model is really just a piece of the puzzle in the whole system. What you need to do is build a complete system that can solve really interesting problems. This system requires a model that knows how to use various tools, knows how to retrieve relevant information, and maybe still preserves historical information retrieved in the past when solving similar problems, and then put this information into the context of the model.

The beauty of context engineering is that this information is extremely clear to the model. Unlike training data, trillions of tokens are mixed together to form a soup with hundreds of billions of parameters, and everything is mixed together. The context directly seen by the model is far more clear than that vague training data.

Next, we also need to understand what tools can be used, what tools can help the model solve the problem at this stage, how to break down a complex problem into a series of tool calls, and even try various paths to see which one works, and then evaluate the results. This is the arrangement of complex agents and multi-agent systems, and it will become more and more important.

Diana: How do you become a master of context engineering?

Jeff: The best way is to directly use these models, harnesses, and tools to solve real problems. Use it and you'll see where the model failed. Then you'll find that you often don't need to adjust model parameters, which is difficult to do externally, but you can make the model perform better by writing better reminders and designing a more clear context.

Diana: Can you give us an example of doing your own context engineering? What skills have you written, or equipped with tools, that have drastically changed your workflow?

Jeff: Take what I did with Sanjay a few weeks ago; we often have to do some performance optimization work on very low-level libraries. Google has an internal microbenchmark library that you can use to measure how long different operations take, such as how many nanoseconds it takes to eject an element from a data structure. These data structures may run on Google's millions of processes, so it's actually important to ensure their high performance.

We wrote a skill to teach the model how to perform a series of operations in a specific order: first run a micro benchmark to measure the current performance, then modify the code, then run the benchmark test again to verify the improvement effect, then measure the cache usage, and then continue iterating. In this way, it can improve itself; in fact, we just gave it the methods we would use as humans in a form that the model can use.

Diana: In other words, do you have a skill that, if someone else gets it, can optimize performance like Jeff Dean? The whole world is going crazy to rob this thing; for some people, it's worth unlimited money.

Jeff: Actually, we published a document called “Performance Hints” a few months ago, written by Sanjay and me. It was a 30-page document covering a variety of performance optimization tips. Some people have summarized it and fed it to various models, and it has been found that the model's ability to infer code performance issues has indeed become stronger.

Diana: Everyone in this room is probably building Agents, and they've all seen their agents suddenly run off at step 30 or 40. What do you think are today's bottlenecks? Is it a context evaluator, or is it because it's an open loop system and errors keep piling up?

Jeff: Of course we want the Agent to run for a long time, because that's how we can solve increasingly complex problems. But as you've observed, they sometimes don't work after ten tool interactions. Sometimes it's because the model is trying to do something it's inexperienced with. Once the model deviates from the training distribution, the performance drops dramatically like most machine learning models. The farther away from the comfort zone, the higher the probability of failure.

There are several ways to deal with it: one is to provide the model with skills and tips so that it can follow a familiar bright path as much as possible; the second is to use a multi-agent system, let multiple agents try different solutions, and then use another model or agent to evaluate which solutions are promising. This essentially uses inference time computation to search for possible solution spaces, which can greatly improve the reliability and performance of running Agent processes over a long period of time.

Diana: How do you implement this kind of workflow internally?

Jeff: We have all kinds of harnesses and a whole set of skills.

Especially in Google's internal development environment, we've written skills to teach agents to use internal tools to review code, measure performance, and pull log files. These skills allow the basic model to do the job well through the correct skill definition even if it has not been trained to use our proprietary system.

Diana: What's unique about Google is that you co-design the full stack from the processor to the product. Which layers will Google continue to cultivate and accumulate advantages over time? Where can a small team of two or three still win?

Jeff: Obviously the Google and Gemini model and hardware infrastructure, what we're looking for is to build a generic model that can do almost everything. But in many cases, this means we can't focus as much on a particular area. A well-designed interface, combined with a model and a set of skills, or a dedicated model that isn't part of the general model system, may have significant advantages, because you can create delightful, high-precision, high-quality products for a field you really love. I think this is where two or three people can gain an advantage when they make something they are really excited about.

The power of generic models is indeed growing more and more widely. So you have to think clearly: what you're doing now will last a long time, or will the cutting-edge model be better than you in the next 6 or 12 months? Or is it something they won't be able to do in two or three years? Judging this time window is what you need to weigh the most when deciding what to do.

Diana: So how do you determine that exactly? How can the founders deduce which areas are worth doing?

Jeff: The most important thing is to choose something you're super excited about, want to build, and that you think will be useful to the world. Second, use the current strongest generic model to test the problem domain you want to do. Let's first take a look at how the current models are performing in this field. If they can't do it at all, that's probably a good sign. If they're able to do part of it, but they're not doing it very well, that's probably a red flag. Because this shows that this ability has begun to appear in models, it is likely to get better with more training data and larger models. So what you're looking for is a field where the model success rate is 0% or 1%, not a 20% field.

Diana: So how do you find these 0% fields? Are they essentially something outside of a training set? What kind of question patterns meet this standard?

Jeff: Sometimes it's because the product you're making has access to a specific type of data, and the underlying generic model can't. For example, if you make a tool to help users organize personal information, the model itself doesn't have permission to access the user's personal data. They can also be extremely difficult problems. If you get the right training data, you can train a more specific model than a generic model, and you can actually do this in a very economical way; training a segmented model for a specific problem doesn't require that much computing power.

Diana: Seen this way, there are actually two paths. The first path is a bit delicate. Your Google slogan is “Organize Global Information.” That field is basically already covered, but the direction of “organizing your own information” is still completely open. And on the second path, you just mentioned specific models for certain fields. Can you talk specifically about which fields?

Jeff: Look at my colleagues' work on AlphaFold, which is a model dedicated to protein folding. It was very successful. It really handled that field very well. It's not a generic model, but a very specific one. In other fields, such as materials science and chip design, this kind of thinking can also play a huge role. You can use highly accurate but niche models to do things that are difficult to do today.

Diana: As you've said before, the key to managing a group of agents is to write a clear design document or spec. How are people good at this?

Jeff: When you work with an Agent, the more you know what you want, the Agent has more guidelines and rules, and an outline of what it's trying to accomplish. And if you don't specify too many things, the Agent will have to infer what you mean. In many cases, it may infer something different than you think. So we told computer scientists right from the start, to clearly state what the software you're writing is trying to accomplish before you write it. Now, the importance of specifying what you want has increased. Because before, you'd give it to a very smart person who might have context or ask you follow-up questions. While agents can do this sometimes, a clear spec is a good idea.

For example: One use of a coding agent that works very well is that you can have today's model translate software from one computer language to another, which works very well, because in that case you actually have an extremely detailed spec, and you have the entire software to explain what the system is supposed to do. So if you have something implemented in Python, you want a Go implementation, which is something today's models seem to be very good at. Because it can take all the tests in Python, make sure they pass in the Go version, translate the tests into Go, compare the behavioral differences between the two implementations until there is no difference, and it's very efficient because that spec is so clear.

Diana: If everyone learned to run hundreds of agents at the same time, and all the code was written by agents, what skills would become scarce?

Jeff: I think it's really great taste. Knowing what problem to have your agent solve is the core of the research problem.

A researcher can have all the tools and techniques, but most of the battle is: what questions are you going to spend your time on? If you pick the right problem and successfully solve it, it's far better than if you execute a boring problem brilliantly. This kind of “what to do” high-level intelligence is extremely important, I think, and the model is unlikely to be very good at it. Therefore, in the future, people will be at the helm of a large number of AI-assisted computations to accomplish great things. But the essence of “what you want your model to do” is what you should really focus on.

Diana: In this age of Agent programming, the word taste was frequently mentioned. But it sounds so metaphysical, how do you make it specific? How to cultivate taste?

Jeff: This is a real problem. In many cases, taste doesn't have an objective standard that can be measured. I think part of it comes from experience. Having solved a large number of different problems in the past will teach you what kind of problems might be interesting in the future, or what kind of things can just be achieved by combining previous methods. Another approach: write down a bunch of things you think might matter in the next 12 months. Maybe you picked one and did it, but came back 12 months later to evaluate: Which of those other things really mattered? Which have been made by others in the world? Which ones haven't been touched yet? This can provide a large sample for developing your own taste.

Diana: I think there's a third way, the one we talked about earlier, to do a really crazy thought experiment.

Jeff: Oh yeah, that's another good approach. Sometimes, don't take for granted what most people take for granted. For example, I did a crazy thought experiment with my colleagues a few days ago: over the past 60 years, the entire silicon chip design and manufacturing industry has done a lot of work to make transistors smaller and smaller, and the error rate is getting lower and lower. Because our assumption is that every chip manufactured should be exactly the same as the other, you don't want any bits to be reversed. However, on a macro scale, we don't make this assumption when building large-scale distributed systems; we use unreliable components to build reliable distributed file systems. A single disk may be broken, but your data should be safe. So here's an interesting thought experiment: what would happen if you tried to build a system with a transistor that could go wrong 20 times a day instead of only making a mistake every million years? That would be a very different design point, and could present very interesting possibilities on the manufacturing side.

Diana: That's crazy. It's starting to resonate with neuromorphic computation or the way the human brain works.

Jeff: Exactly. Signals in the brain aren't particularly reliable from one place to another. So in the brain, when there is really important information to be conveyed, there are multiple channels to ensure that the information can be delivered.

Diana: Are there any crazy assumptions you've thrown away and ended up building an impactful system?

Jeff: TPU is a great example of customizing hardware for a problem area when it didn't seem as important as it is today.

The origin of MapReduce is another good example. I and Sanjay and some of my colleagues have written a lot of hand-written parallelized code before, and did a lot of checkpoints to make sure it was robust enough to run on hundreds or thousands of machines. But that code is often mixed up with simple things you want to do, such as “I want to see the content of all the pages and then calculate the mapping of the URL to the page language”, but it's overwhelmed by lots of parallelization and reliability code. We thought of functional programming training and realized that we could squint at these problems, abstract MapReduce as an abstraction layer, put checkpoints and reliability mechanisms in the underlying library, and build everything on top of it.

Therefore, MapReduce was born from the thought experiment “If we squint, can we find a large number of questions suitable for this abstraction?”

Diana: Now that AlphaChip is laying out chips, AlphaEvolve is proposing solutions, evaluating, and maintaining effective solutions. It looks like you're building a system that can self-synthesize — AI is building AI.

Jeff: More broadly, this is actually the foundation of the scientific method: proposing experiments, implementing experiments, evaluating experiments, and obtaining results. There are now more and more problem areas. You can run not just a few experiments, but a large number of experiments, because you can automate this cycle and make the cycle delay extremely low, which will allow us to handle a large number of different problem areas in engineering tasks such as science, engineering, machine learning, model design itself, and chip design. If you can automate these, and have an orchestration framework, you can break down high-level goals into sub-problems. Each sub-problem is an automated exploration cycle, and then the orchestration framework then integrates the solutions to the sub-problems into an overall solution, which will have a huge impact and will accelerate machine learning, accelerate science, and speed up engineering.

Diana: It sounds like many fields, as long as there are good evaluators or fields that can be formally verified, are suitable for AI self-improvement.

Jeff: Right, many times your evaluator needs to be faster. For example, my colleagues did some work in the field of quantum chemistry about ten years ago. If you want to know the properties of a molecule, you can generate a molecular configuration, and then want to know what properties it has, you can run a very computational density functional theory simulator, which can take a whole night of calculation to give an answer.

But what my colleagues did was: take the output of a large number of simulated runs, the molecular configuration of the input, and the output of an expensive simulator, and then train a neural approximator to replace the simulator. They created a verification device that was 300,000 times faster than the original emulator and was almost as accurate.

It's completely changed the way you do science; you can sift through 10 million candidates while eating lunch instead of spending six months putting together enough computing resources to run all the simulations. In many fields, there is huge room for faster verification models.

Diana: So what areas of problems are you particularly looking forward to with this hypervelocity science method?

Jeff: Obviously machine learning itself is one. Can I improve a model from myself by running lots of experiments? Today, the way large research teams improve the model is: people come up with some ideas, run a series of small-scale experiments to see what works, then scale up, evaluate, and integrate the most promising ones into new formulations.

But there's no real obstacle stopping this from becoming a more automated cycle. The model itself decides what to explore, or at a high level, gently pushes “why not try a new model architecture idea that includes this”, then it runs a lot of experiments to see what works and integrates faster. Essentially, what you want to optimize is the number of findings per unit of calculation input.

Diana: In 2014, you and Geoff Hinton and Oriol Vinyals wrote a paper on distillation, using a large model to train a smaller, more efficient model. Now this has become a technique everyone in the industry uses, but this paper was rejected by NeurIPS.

Jeff: Right. I don't blame the Program Committee because often a paper receives three reviews, and some people take a look and say “it's unlikely to have a significant impact.” But when we wrote this paper, we saw that this is a super important question, because we know that making cheaper and more capable models from larger models is something we want to do urgently. We want to serve models to more and more people in many fields such as voice and vision.

But sometimes the reviewers might not have that experience, and they might not be considering large-scale AI services, but rather thinking, “Is this a fundamental development?” So it doesn't matter if you get rejected once in a while. We put it on arXiv, people read it, used it, everything was fine.

We're actually using distillation technology to make our flash models, distilled from larger pro models, which is why our Gemini flash models are so competitive at the same scale and speed.

Diana: When you joined Google in 1999, it was just a 20-person startup. If you were to transfer young Jeff Dean from then to now, with your skills, what would you do? Will you join a cutting-edge lab or start a company?

Jeff: The most important question for me is: do you really care about what you're doing? If you could make progress with a group of colleagues you love to work with, could this change the world in a positive way? What you should strive for is to have a positive impact on the world, work with people you love, work hard, and do your best.

As for joining a cutting-edge lab or starting a small company of two or three people, it was a different experience.

In large organizations, you have lots of great colleagues, lots of interesting questions, and you already have a platform where your work can influence many people around the world.

And as a small startup, you have to have something you're passionate about, and take a lot of risk with this particular problem, but it can also be very rewarding.

Whichever path you take, ask yourself: if I fix this, and the best results happen, will the world be much better? The world still says “oh, that's cool, but it doesn't matter”; it's not worth your time.

Diana: You've been the mentor and manager of many engineers, and you've built huge systems. How do you work with smart people, how do you find smart people, and what are your experiences?

Jeff: You always want to find someone who has really great skills in the field you need, but you also want to find someone you enjoy being with because you spend a lot of time solving very difficult problems with them. You want people with low self-esteem, team spirit, and skills that complement you.

I always think that in a small team, others know things I don't know, and I also have skills that others don't have, which is super fun because you're working together to build something that no one of you can do alone. In the process, you actually gained a ton of new knowledge, new skills, and so did they.

You should think of your engineering or research career as a tool belt full of technology; you always want to add new tools to it, because you never know when you'll run into a problem, and adding more tools makes it more likely that the problems you encounter in the future can be solved by you.

Diana: I'm pretty sure someone in this room will eventually build something as impactful as the MapReduce, TPU, distillation, etc. you made. What problem would you like them to study?

Jeff: I'm particularly excited about new methods of hardware, or more efficient inference hardware. I think there might be radically different machine learning algorithms that are much more data-efficient than the methods we use today. Today's models are probably seeing a thousand times as much data as an 18-year-old can see. But an 18-year-old is flat or even better at many things than the cutting edge models that have seen more data.

So can you think of a more data-efficient system that can continuously learn from your actions? Continuous learning is a very interesting direction, and multi-agent interaction is also interesting. Creating ways for people to have better conversations and help people around the world get to know each other based on their interests can all be very interesting. There are so many cool things in the world, and we should all try to make even cooler things happen.

Twitter:https://twitter.com/BitpushNewsCN

Compare the TG exchange group:https://t.me/BitPushCommunity

Compare TG subscriptions:https://t.me/bitpush