Transcript

Welcome everyone. This is CS 329Z, engineering AI agents. And today is the introduction to agentic systems. I'm D Young, assistant professor in computer science here. And I'm co-teaching this course with Michael Ryan and John Young. Today we will cover logistics and also give you a short history of agents, what a agent is made of, and also the open problems we will work on this quarter. I will first like to Michael and John introduce themselves. Hi everyone. I'm Michael. I'm a second-year PhD student here at Stanford and I'm advised by D and Percy. So my research is on open-source human-centered AI.

And in the past agent-related work I've worked on the DSPy framework for programming, not prompting language models. So that's my little claim to agents. Hi, my name is John. I'm a third-year PhD at Stanford. I'm advised by D. And in the past I've worked on coding agents and coding evaluation. So I created a SweetBench, SweetAgent, ProgramBench, SweetSmith, if you've heard of those works. Great. Okay, so with the instructor team, I want to formally welcome you to this course. The primary contact to reach out to us is to use at discussion. If you want to be added to it, just send us an email. We prefer you send everything to this email list so that we can take care of your request very quickly.

The website will maintain all the lectures, slides, and homeworks. So that's your primary website to check out. If you haven't enrolled in at discussion, please do since we post a lot of announcements there as well. So in today's lecture, Um, in addition to the logistics, I will mention in a few seconds. We're going to provide a history of agents. Uh, this is There is actually a really long history behind agents. Uh, we will talk about agent architecture. This is the core content today. And then compound AI systems, rag to use, and MCP. Uh, we will end with a brief highlight of key challenges, including the recent multi-agent risk and collusion we have been seeing.

So, let's get it started. Uh, why we create the CS 329Z? Uh, so, when we think about large language models, agents these days, there seems to be very different perspectives involved. You can take a modeling perspective. You can also take a system perspective. A lot of what we do is actually around the engineering. And if you think about that, LM people, uh, I think they know how models are trained, pre-training, post-training, prompting, uh, alignment, and evaluation. On the other side, system people know how to build reliable tools, systems, um, more related to agents. We have tools, APIs, environment, and infrastructure.

So, the expectation for this course is that we're going to cover a lot of the foundations. But more importantly, we want it to be more hands-on building plus evaluation of agents. A quick glance of the content. So, uh, week one to week five, uh, we will talk about large language models for builders, um, with a highlight of retrieval augmented generation and its variants in many, many follow-up agent architectures. Tool use and a function call will be another highlight here. We will slowly move to agent design, training, and scaffolding. And also the design of agent memory, uh, as well as orchestration and multi-agent systems.

Um, the second half of this course, we are going to have optimization, harness, and data for agentic systems, data quality, data selection, etc. The evaluation and benchmark design will also be highlighted in different ways. Agent safety and the guardrail will also have two special highlights of coding agent and also a more emerging topic called a proactive agents. We have actually two special teams who are going to show up in our class this quarter. First, we will welcome the terminal bench team to talk about how they design terminal bench and a lot of questions you can share with them. We will also welcome the SweetBench team to come to chat with us.

So, for both lectures, it will be a team reunion from those two teams that have created a very fundamental evaluations in our field. Course policy. First, I think the biggest message that I want to highlight is that you should use AI to learn. You may want to get yourself familiar with the Stanford Honor Code. And laptop use is only allowed for taking notes, etc. And we strongly discourage the use during the lecture so that you can be more focused. Prerequisite and the compute. So, first, we strongly recommend that you have taken any NLP or large language model related training. You are expected to know basic LLMs since we are not going to cover like fine-tuning or pre-training or alignment, etc.

In terms of compute, we are very grateful that Lot Institute gives each student $200 in terms of credits. And Thinking Machine will provide a $250 for Tinker credits. I want to switch a little bit to kind of conclude the logistics section by showing you a figure. This is a just like a specific instance of what's happening today. With a lot of data, if you show on the top, this is actually homework score from two populations. People who use AI and the people who do not use AI. Very interestingly, I think with AI, everyone's homework score are just so great. But then if you look at the bottom, this is actually the exam score for the same population.

Before using AI, this is definitely more kind of shifted towards 100 here. With AI, we have way more people did worse in their exams. And I think we see this kind of pattern very frequent these days. The second example that I want to share with you is actually from Anthropic. They did a study with their own engineers to figure out how the use of coding agent actually impact coding. As you can see from the left side, if you use AI, the task completion time actually decreased. But then on the right, the quiz score in terms of your mastery actually significantly decreased. That is, despite the coding agent that can give us 1,000 line of code, it's a little bit unclear whether we fully understand such AI output.

So, with that, I want to conclude the logistics by saying that we really hope that you can use AI to learn how to learn. And this is really just about you and your CS 329Z. With that, any quick questions for logistics? Okay. If there's no question, I will continue to the history of agents. One thing that I do want to mention is that the learning objective today is for you all to get a familiar with agent loop on top of large language models and the different key components. So, let's get it started. This is a word that's probably very very popular today. And uh I'm going to do next is to trace where this idea actually came from, how it developed over the decades, and why it suddenly take off with large language models.

Um if you look at the uh more linguistic side of agents, uh the word agent actually comes from Latin, uh meaning that one uh who acts. And uh this term also uh entered English in the later um 14th century. So, in AI, the first index use of it is in this specific IJCAI paper from 19 uh '73, uh where Hewitt and Bishop and Steiger talk about this architecture called actor algorithm, where each actor is an active agent that wait for a message and then does what the message tell them to do. So, the idea of software entities that act on their own is older than most people can think about. And then, I think like a few years after that, um the Contract Net Protocol actually talk about how agents dividing task with each other and then negotiating task.

This trend actually continues. In Marvin Minsky's um Society of Minds, it actually talk about how a group of sub agents actually work together to get it something done. And if you kind of reflect on this, this is actually the earlier idea of multi-agent orchestration. I think the history actually continues. Around the 1990s, we actually see more and more emergence of uh software systems that uh serve as agents. Uh from Pattie's designing autonomous agents to later um people talk about uh agent as autonomy, uh social ability, reactivity, proactivity, trying to define agents in different ways. The most uh familiar um definition to many of us is actually from Russell and Norvig, uh which defines agents as anything that they perceive through sensors and act through actuators here.

We now kind of like enter a different stage, which we call reinforcement learning error here. This is a classic agent environment loop that many of us get familiar during RL courses. So, what happened here is that agent take action in the environment and such action will receive a reward. And then they do the action based on specific state and the current reward. So, this is a very classic agent environment loop from Sutton here. And then I think like we see more and more promising from promises from this kind of setup. DQN refers to deep Q network. This actually from DeepMind's earlier days, where they build AI systems to actually solve specific games.

Now, I think a later you also see that AlphaGo and OpenAI 5 showing that this kind of approach actually can reach superhuman performance in like Go and I think this one is Dota 2. So, those were agents, but each of them are actually trained for a specific task. So, what actually changed is introduction of large language models. I think the first kind of impressive capability of LLMs is actually in context learning. Built upon this kind of transformer-based architectures, GPT-3 actually showed that you can just talk about a task and give you a few demonstrations. Like in this particular example of few-shot, in order to do translation, you can offer certain examples and then the model can actually follow such examples and then do the task.

This entire process, unlike For systems I showed you, there is no gradient update happening here. I think like not only like in terms of prompting or in context learning, prompting could do more than teach a model a new task. It turned out that models themselves, they can also reason step-by-step and then use tools for the things that they cannot do alone. So this particular example is called a chain of thought prompting. Uh you can see that asking the model to actually work through a problem step-by-step actually give more accurate answers in general. Uh WebGPT was proposed by OpenAI. This actually enables language model in earlier days to do to use like web browser so that it can actually look up things online instead of only rely on memory or parameters.

Um this actually leaded to a lot a lot of follow-up work in terms of like web browser agent. Um one of the very interesting architecture is actually called a modular reasoning knowledge and the language. I would view it as one of the first earlier systems to think about to use. So basically this architecture actually decides which outside expert to send a request to. Is this going to be a weather request? So then I should use a weather API. If this is a coding task, maybe I should send it to a Python programming tool etc. So now the model actually can think and it can take actions through tools.

So I think we now have like both important pieces here, that is models can reason step-by-step and the models that can use tools. Those are two very very important a prerequisite for the emergence of agents. I'm going to show you a very short timeline in what happened in history, that is 2 years ago. So 2022 React was proposed Um, so the name actually means reason plus act. So, instead of thinking once and then answering the question, what the model does here is actually a loop. It thinks about what to do next, take action, and then look at what happened, and then use that to think again. So, it's actually a loop happening here.

After that, I think we see that uh more than more uh models get released. So, ChatGPT get a launched, and suddenly I think a millions of people start to talk to LLMs every day. And then GPT-4 was released. And the reason why I want to mention this is that if you think about agents, they really rely on the base model, the strong capability from those base models. So, when GPT-4 came out, we have a much stronger model that it could uh follow long instructions and plan more reliably. Um, within 2 weeks, there were actually two open-source projects came out that really shape what we think around agents.

The first one is called the Baby AGI. Um, it basically gave the model a goal and let it write uh its own task list, walk through the task, and add a new ones as it goes. Uh, AutoGPT was released uh um I think in the same uh months, 2 days later. Um, basically try to connect the GPT-4 to tools such as web search, file systems, etc. And then let it uh keep uh going on on its own without a person really like typing the next prompt each time. I think uh like now if you look back, it was definitely very ahead of time ideas. The reality is that none of them actually really worked very well. Um, but they got the people's attention and really opened our imagination about what's possible.

Um, I think that that's also probably the time when the word agent really enters like everyday use. So um I want to kind of go a little bit deeper here to share with you like the underlying model and also agents, how they have improved since 2023. Um so this page, uh if you look at that, I only show the GPT-3, GPT-4, and GPT-5. Um the underneath model actually is better and better. If you just contrast the GPT-3 versus GPT-4, we see that on MMLU, there's a 43% increase. This is actually true for all the data sets that we have. From GPT-4 to GPT-5, some of those data set actually didn't show up here because they got solved or they saturated so that they cannot be used as very reliable signals.

Instead, we actually introduced more evaluation such as GPQA, which is a very challenging QA setting, and also harder and harder math problems here. So the key message here is that each generation of base model actually made a really big jump in terms of their performance. And a stronger model means that the agent now can actually reason better. They can make fewer mistakes, hallucinate less, um and along the way they can handle longer and longer task. And then let's look at the agent side. So the agent side is also improving as well, not only just like the raw model capability. Uh so this page is actually around like SweetBench verified.

Um as you can see uh in January 2025, I think uh the performance is only around a 30% but then over the last uh um 1 year and a half, now that all the numbers are actually reached a very high stage here. And then right side of this is from Astro release. So, you can see that OS world, which is a computer used evaluation. And also on terminal bench, which is a more like a command line task. Um the accuracy are just impressive. There's also a trend that people start to pay attention to the efficiency of the process. This is why you are seeing that most of those curves these days actually have both price and accuracy there together.

So, this is how we got here. Um it's a long history in the last 16 years. Um so, what I'm going to do next is to really open one app and then see what our agent is actually made of. Any quick questions here? Okay. Um yes. Do you think models capacity or ability is going to saturate in in a few years or is going to still like develop for like 10 years or so? I think the first uh just to repeat the question. The question is will model capabilities saturate or will it keep improving? Um I think in terms of timeline it's really hard to make estimation into the future because a lot of things can actually happen in two to three months these days.

Uh there has been a very interesting topic RSI, uh recursive self-improvement. And it has showed very interesting patterns on solving open math problems and many other tasks. I think with RSI this pattern will get accelerated a lot. So, let's see. Yes. Can you go back to the last slide? Yeah. So, when you when you see the improvement on the terminal bench for my note, uh we can see that you know, GPT-5.6 sold is sort of doing really bad and over the period of API cost like as the API cost increased, the accuracy increased, right? Uh do you think is it What are the reasons for you doing better when the API cost increase?

Also, what are the reasons for GPT-6 extra doing much better? Is it algorithmic development? Is it like bigger model? Yeah, great question. The question is that why when the models get better is much more expensive. And also why is Astra better than their earlier models. So in terms of the price and better performance, the API cost here is strongly associated with like the number of tokens get digested in the process. So if you think longer, the general pattern is that thinking longer, thinking longer is actually going to lead to improve the performance. So this is just a general trend. And then in terms of why models are getting better than their previous model, I think there are many reasons.

This is also a topic we are going to go deeper later in terms of open weights models and the openness of large language models and the proprietary model aspects. The reality is that we do not know a lot about how those models are trained. Is it about a better data, better distillation or other aspects? Yeah. Okay, let's now move to the agent architecture. So I think like if you think about like agents, it has a really long history. It get really accelerated with large language models. And this is going to be the agent we are going to talk about. And I want to go back to this example from React that I showed to you.

So in this particular example, the agent needs to think about what they are going to do. So this particular task is to put a paper shaker in a jar. And then if you read the right side top to the bottom, Uh, model first the model itself is actually a LM core. This is the the foundation model uh, underneath. And then the thinking step here, uh, basically it uh, it says things like, "I need to find a pepper shaker more likely in cabinets." Uh, those are what we call planning and the reasoning. And then the traces so far, everything uh, above the current step is what we call memory. And then the agent actually take action such as go to cabinet one or take a pepper shaker.

Those are what we call uh, tool calls. And then those observations, um, those are literally just like after each action, what the agent uh, sees from the environment. Uh, you can see that at least there are five components if we just look at this very simple uh, example from React. So, we will now work walk through each of them. Um, so this is uh, meant to be a conceptual framework uh, just to highlight the different components. And then if you see different directions, it actually didn't carry a lot of weight. Uh, when we talk about agents, uh, I want to just share you some kind of like core components here.

First, agents is built on top of large language models. So, stronger model in general means better agents. This is why we also have a debate in terms of should I think about training for agents or should I just care about the better hardness or other uh, improvement. Um, on top of LM core, there's a very important component called the planning and the reasoning. So, if you think about asking agents to do a very big task, they actually need to decompose them into smaller pieces. And then maybe have sub agent to deal with each of them. Uh, we may also want to add some explicit aspects to improve the reasoning here.

And then as you see from the earlier example, we need to introduce a memory to make sure that as agents work on the task for more than 2 hours, they actually know what has happened so far and then can use that as context to do the reasoning. Even in earlier days, uh when model can solve very hard math problems, if you ask them like um where is CS329Z happening today on campus, probably models cannot answer that. Um yes, our systems also cannot answer that. Um So, in order to really enable LLMs to do stuff, we actually could give them tools here. This could be information retrieval. If they do not know what's the classroom um location, they can actually go to Stanford website and then search for it.

If they do not know the weather today at Stanford, they can actually search for weather from official database. Uh similarly, um I will show more examples in a few uh weeks. Um there has been a time period in history where a uh this fact that GPT-4 was like amazing. It actually cannot solve three digits times three digits, just like a simple multiplication problem. Uh so, there are lots of those scenarios. So, tool use really enable agents to take factual and uh more consistent actions. And all of those actually lead to different terms that you may heard of, uh such as tool call, uh interaction, response to users.

This is what we call like agent actions. And I think more importantly, there is this like environment part that we haven't talked about. So, agent observe what's happening in the environment and then take actions. So, what you see on the screen right now, this is actually a skeleton of an agent. Uh of course, this is only one way to draw it and uh please do not uh uh take uh too much weights with this kind of directions. This is really meant to uh illustrate the process here. And then on top of this like a single agent, you can imagine that you can have different agents. Even within the same agent, maybe you can do different configuration optimization for each box that I listed here.

This is why later like hardness and optimization become very important because everyone wants to find the most optimal configuration for this agent. Um as I said that um this is not the only way to talk about agents and our breakdown is not a unique. Uh different researchers actually use different labels. Um for a long time, I think there was a lot of debate around what is a agent, what is not a agent, what is agentic systems, what is not agentic systems. We are not going to do any debate uh in this class, but I think that you can see the common components showing up here. Uh the left side is from uh a OpenAI researcher back in 2023.

Uh they talk about that, okay, if you think about agents, they use tools, they have memory, planning, action. And then on the right side, this is uh around a cognitive language agents. Um you can see that they have similar pieces such as memory with retrieval to read from it and also learning to write it to it, reasoning, etc. Despite that was our different vocabularies, I think it really kind of point to the same architecture here. That is an LLM uh reason, remember, and use tools, and acts in a loop uh within its environment. So now I'm going to go back to the uh diagram of those different components and start to share with you different modules we have here.

So if you look at this, I think the first thing that it come to mind is actually this planning and reasoning from this particular figure. Um So then the question is what does that mean for LLMs to reason and why does it help agent? Um I want to draw some kind of uh metaphor to human here. Uh Uh this action is probably a little bit of controversial if you think about that. There has been a lot of debate to see whether the reasoning of AI is similar to human reasoning. Again, no uh debate in this course. Um but if you look at this process, I think uh for humans, uh what does reasoning mean? It means that uh some mental process uh such as deduction, analogy, and the planning that we all do for different task.

For LLMs, I think uh you can think about them as like chain of thought. So, in this particular example here, uh most of the time we would have called this kind of chain of thought as a reasoning process. Although, one message I want to share with you all is that this chain of thought may not be faithful of what actually happened within LLMs. Um for agents, by reasoning we mean the internal actions that update the agent's own state. So, those are just like a different uh terms to understand the reasoning and the planning. So, now let's look at a concrete example. So, why is reasoning helpful for agents?

So, imagine we have a agent that's doing actions right now. And at a T step, it observed that um you are cooking a dish and the scene salt is out. And then the next step, if you see a action of turn right. If you just look at those two, I at least I get confused like, okay, why why is the agent actually turn right to here? Etc. So, the observation to action mapping is actually huge. And then this can be very hard for audits or just like a for uh making agents better. So, if we could actually introduce a reasoning model here to let agents to explicitly reason, then this actually help a lot. Uh if you read the reasoning in the green box, it says that the dish should be savory.

And since salt is out, I should have find something equivalent. Uh it is in the cabinet to my right. And then when the turn right, we know why the agent is doing this particular action. Um I think a reasoning helps in two ways. First is from a generalization perspective, because the same reasoning could actually work in new similar situations. It also help with alignment, because you can see why the agent did what it did. So this is why like reasoning was actually a first uh important component we are emphasizing. So now just to understand why systems like React beats existing prompting methods, um I'm going to give you another example here.

Uh this is an example from Hotpot QA. Um So um it's asking certain questions, and then standard prompting fail. And if you just have a model that can reason like chain of thought, it also uh fail here. Basically, we have wrong answer. Um This is actually more dangerous, because you have a very convincing reasoning trace that's wrong. So now let's look at if you have this kind of a loop of think, taking action, think, ob- observation. So in this particular example, the model thinks that it needs to search Apple Remote, since this is in the query, to find out what was originally designed to interact with Apple Remote.

Uh such a thought actually induced search actions, and it gets a long description of Apple Remote from Wikipedia. In the second step, the model actually used reasoning to extract the key information from that specific page, and uh they found that Apple Remote was actually designed to interact with Front Row here. Um then the model thinks that maybe they should search Front Row to find out who like what it controls it. But the there maybe like the specific Wikipedian page does not have such information, but some other articles have. So this process actually continues. Like in the uh third step, the model actually used reasoning to recover from extraction and then search from her role instead.

So, when the model think that they have the correct answer, they actually finish the task and they return it. So, you can see that here the reasoning traces actually induce and adjust agent actions, which help agents obtain new knowledge to support their own reasoning process. And then one thing that I want to mention is that if you look at this, a lot of the trajectory here is very intuitive, interpretable, and somewhat factual. Um to show you how this actually works, uh chain of thought is a very uh straightforward prompting uh baseline. And then you can see that uh compared to like just uh doing actions, which is act here, react actually perform act constantly.

Um one thing is that uh if you look at the failure case from different reasoning uh paradigms, um let's do a very detailed breakdown. If you compare chain of thoughts and the react uh in terms of failures, like reasoning error, actually react is very high. Um search result error, react is also very high because uh COT does not have search functions. Hallucination, uh hallucinated the reasoning traces or facts, react is zero. And then chain of thought is actually 56 here. So, in general, hallucination is a very serious problem for chain of thoughts and the many of those prompting technique. And then react with this kind of reasoning module taking actions actually mitigate it a lot.

So, you probably noticed that there is a term here called SC here. Uh SC refers to self-consistency. Uh this is related to chain of thought as a prompting technique and it's a very intuitive uh trick you can use. Uh so, apparently you can prompt the large function model uh using chain of thought. And instead of just getting one of them, you can actually get a diverse set of reasoning choices. You can aggregate by choosing the most consistent answer in the final answer set. So, this is a majority voting to some extent, but from a consistency perspective. So, after self-consistency, the uh the process of actually making agents better at the reasoning um like happens very very frequently.

Uh one of the very notable example is called a reflection uh reflection. Uh basically, the agents are going to reflect through natural language feedback. Uh if you look at the left figure here, so let's see we have a current trajectory generated here. This is the short-term memory here. And then you can give that a to a evaluator. The evaluator is going to judge is this going to be correct or incorrect. And then with those feedback, the agent is going to do a self-reflection. And then with those self-reflected content, uh together with the experience, they can actually change what are the actor is going to do or going to see.

So, this is a very earlier idea of letting models do self-reflect. Um just to show you that if you look at the all the steps after the evaluation, um the reasoning sorry, the reflection is actually uh quite interesting in terms of why the model thinks uh such error occurred or they received this particular evaluation outcome. So, a reflection actually works pretty well. Um I think if you reflect on the process, it actually reduce errors by using like a self-reflection to really distill like long field trajectories into relevant experience or rubrics or reward signals that can be used as self-hints in the future.

So, from the figure, um you can see that uh compared to react only, uh with reflection, uh it's definitely doing way better than the react uh system. And uh similarly, in terms of the uh I think of this is the uh hallucination. You can also see that it reduced hallucination a lot with this kind of self-reflection technique. How is this behavior introduced to the model? Like is it like included as a part of the prompt or there's like some fine-tuning or training steps to perform into the weights? Yes. Good question. So, reflection and many other similar self-reflection techniques does not involve fine-tuning or any kind of gradient updates.

Most of them are actually implemented using prompt. One one minor thing I want to point out is that if you read the reflection paper, it actually keep using this word called the reinforce. But reinforce is actually not achieved from a reinforcement learning perspective. It's only from a self-reflection prompt aspect. In this slide that you showed where you compared React with Chain of Thought, the results hallucination was zero. How is that different from reasoning? Like if there's a reasoning error and which is considerably high, how is the hallucination zero? Yeah, so there are going to be multiple figures, right?

Sometimes you will fail not because of hallucination, but because such label may have ambiguity or because like um you need to make sense from the current context, which is not about like getting the right kind of tool or information or things like that, but more about like interpreting what has happened so far. So, they are slightly different in this process. So, what would you say would be reasoning error? Yes, so that would be reasoning error. Yeah. Thank you. So, is the main difference between Chain of Thought and React in the fact that like React and Chain of Thought are essentially the same reasoning strategies, just Chain of Thought is just like prompted and there's not much difference.

Like they can't really track its work. Yeah, so in Chain of thought uh if you um we will do a more deep dive into the reasoning modules in 2 weeks. Um so in chain of thought it's literally just like a one prompt, one pass. In uh react, you can think about it as a loop. So the model think, look at the observation, think, take action, and then think, look at the observation, take action. So it's like a multiple step, it's a loop. So in this way, the model actually can revise its own errors or uh getting new information to update themselves. So this kind of richness is something chain of thought does not have.

That would make you want to decide whether to agree or disagree. Okay, maybe in the last one actually, yeah. Okay, uh do you think uh the the the process is a distillation of the model itself? Um if if so, is the distillation going to cost uh like like is this reflection going to cost some overfitting to the uh its action? Yeah, so uh one thing I want to clarify is that I do use the uh sorry, not this page. I do use the word like this two here, but this is actually not a very classic distillation we are talking about. Here, it's really about you have a lot of like trajectories of what happened so far, right?

And then um in general, you may just rely on models to interpret what happened so far, and then like do actions based on that. With self-reflection, the model actually think about that, and then try to compress that into something that's shorter. So in this process, um one word to use is this this two. You can also use it as abstract or synthesize. So this is very different than the classic distillation technique, which is more about uh getting output from other models, and then do training on top of them. Okay, let's uh continue. So in in addition to this kind of like reflection self-consistency, uh another trend that that's very interesting is actually around the multi-agent debate.

Um so earlier days, people researchers use multi-agent debate to improve reasoning performance. And this literally means that you are going to have multiple agents doing the same task solving. And then after each round, you let them read each other's reasoning and let them kind of debate just like how you talk about some decisions with your teammates. And then agents can actually like really leverage those diverse perspectives and maybe make some final decision here. I think of this multi-agent technique in earlier days is a very useful prompting method. Um the assumption at that time is that each individual model is not strong enough.

So we actually use this kind of multi-agent setup as a way to prompt the model. When models get a very powerful today, if your only goal is to get a performance boost, I think a multi-agent is not going to help you a lot in terms of performance boost. But maybe there are other design choices or design considerations that we can benefit from multi-agent, such as efficiency, such as data access control. Maybe this agent is particularly designed to take care of health care and that agent is designed to help you with sensitive information, etc. So after multi-agent, another way to really kind of like improve this kind of reasoning is actually to set up a many agents and then use a orchestrator to orchestrate the different agents to work towards the same task.

So this is a very earlier system from Microsoft called a uh MaginTic here. Basically, if you have a bigger task, the orchestrator is going to break them down into different components. And then you can see that the first part get assigned to file server and then to a coder and then to a computer terminal, etc. And each agent basically take care of each specific task. And then in the end you can actually combine all of them and then provide a final answer to users. So this is about planning and reasoning module. We will go deep into this process in week two and three. The next component that I want to quickly mention is around the memory here.

You can actually do a lot of very creative design in terms of how we want to make memory work. Just to briefly share like why we need a memory. Um two reasons. First is this factor our context window for large language models are getting bigger and bigger. Still context window cannot hold all the event stream happens so far. So if you think about long horizon task we are talking about today where agents can actually work on your project for more than three hours. That's actually a lot a lot of data and it's really hard to fit that into context window. The second is that even that's that's possible, it may be really hard for the agent to figure out what is a relevant piece of information I should use.

Think about that if you have 1,000 documents there, you store them in your context and then basically each time if you need to find the right document, this is actually very kind of time consuming since you have such a big search space. So the bottom is actually very interesting design from the generative agents, the Small Ville project where they were trying to build agents that can simulate people living in a small town. And then in their design, they actually need to use the memory stream to make sure that we can keep track of what happened to each individual so that the story can continue.

There are different type of memories you can design. If you think about what kind of things to store? Episodic memory here literally refers to we store everything like raw logs, every sentence the agent has shared with users, every sentence that the users have provided to agents. Semantic memory means that instead of thinking about this kind of raw logs, we may want to remember like semantic events since those may be more useful. Procedural memory you store skills. Instead of like all the specificity, you remember like what is the skill, what is a procedural of how we do certain type of things.

So if you go deep into this process like this is actually example of episodic memory. You can see that literally we are storing every like single time step and then what happened at that time step. With semantic memory, we actually can do some synthesize of what happened over the agent trajectories and then literally kind of like have this like a small summary box as a way to like to remember what has happened in the last several minutes. So this is what we mean by semantic memory. Procedural memory you can actually check out this Voyager paper from Nvidia where they were trying to teach agents to play I think this Minecraft here where if the agent figure out how to use new skill or new tool, you actually want to abstract that into a skill and store it in your skill library.

And then whenever you have a new task, you want to see like do I have a skill that I can use from a skill library? If so, then I'm going to retrieve it and use it. So this is more from a procedural perspective rather than like all the logs that have happened so far. So this is about the memory and then another very important thing is about the two use here. So two use we will talk about this next Wednesday. But just to give you a quick highlight of what this means. Um, we all know that right now you can actually enable agents to use different type of tools such as uh search, such as um coding agent, etc.

So, in earlier days, um the tool use is actually triggered in the actual decoding process. So, if you read the right part, um the sentence it maybe let's just read the first one. Uh the New England Journal of Medicine is a registered trademark of. So, if you need to do like some kind of um like information generation, and at this this point uh we may want to trigger uh special token called a QA, and then the QA literally is who is the publisher of this uh Journal of Medicine, and then you got it the specific answer, and then you use that answer in the process of generation. So, in other words, uh earlier when we want to teach model how to use tools, we are actually learning like like where to call uh API, when to call it, which API to call, etc.

So, Toolformer was actually the first uh uh ish work that it does this kind of teaching LLMs to use tools. Um you probably uh can enumerate uh how many tools they can use. At that time, I think there are only five tools that we teach to agents. Agents can use a calculator here. They can use a question answering system. They can use search engine. They can use a translation system. They can uh use calendar. So, that's a very limited set of tools. Today, I think uh our agents can actually use a wider range of tools, but they actually all get inspired on top of uh setups like this. More recently, um Gorilla from Berkeley actually uh tried to teach LLMs to use like solid ends of APIs.

Um so, what they did is that they actually uh look at hugging face, uh torch hub, and also tensor flow hub, where there are like over 1,600 APIs. And then they basically generate the synthetic data from that, and then trend a 7 billion parameter model on top of that. Um at the inference time, if we look at this example, if the user says that I want to see some cats dancing in celebration, um then you are going to do a retrieval here. Probably you are going to use like a stable diffusion, and then you get this like a image out here. Otherwise, if you think about this, like the base model needs to have the capability to understand this, and then give this kind of like output of the image.

But now we can actually delegate that to a external tool, such as stable diffusion. Then we can get the task done without the requiring the underneath model to even um do this kind of image generation. Um with Gorilla, I think that the results are quite impressive. So, if you look at the reduction to hallucination, uh it's actually pretty good. Like very few hallucinations, and a pretty high accuracy. Um I think this actually makes sense, because without the so many tools like use or like understanding, you actually have to rely on the model themselves to do this task. But now you can actually do more factual and uh reasonable generation.

So, this is around the tools. Uh we will go deep into that next Wednesday. So, if you put everything together, uh this is actually the skeleton of an agent. Uh LLM core that reason and a plan. The memory actually carries states um across steps and different task. Um tools to reach out to diverse APIs, um available tools we have, and also an environment that agents can actually take actions and observe um states from. So, every agent that you are going to build, uh whether it's going to write code, browse the web, or like runs the terminal, it's some kind of version of this diagram. And what differs is actually how each box is implemented, and how well the pieces like fit together.

So, that's going to be a lot of the key aspects we are going to cover later. Um any quick questions until this point? Um so, this is actually related to the memory design. Um if you just think about like how to design memory systems, instead of letting them to decide like to decide what to memorize, the episodic memory literally just tell them to remember everything. So, everything happened so far. So, this is more from a heuristic perspective. And semantic memory, you may need to have some separate module to teach them that instead of like um instead of like remember everything, you may want to remember important information.

Then you can actually give demonstrations of what is important information. What is the format you want? What is the like um format you want to even store in your memory. So, those are the design choices we are going to bring to the process. Um this is also a place where things can get a little bit tricky. Um there was this example I think Do I have it today? I don't have it on my slides today, but like earlier this year uh some people are using Open Claw um to manage their email. And then all of sudden, Open Claw actually started to delete this users' like email. Despite the user says, "Please stop.

Uh please do not delete my email." It is still kind of continuous. Um the user actually specified in the very beginning that uh whenever you do such actions, please confirm with me. But then it's just like so interesting like, "Okay, why that happens?" And why the model actually is not uh even listening to instructions. It turns out that this is actually a failure in the context memory compression stage. Because you want to make sure the memory is stored in a very efficient way. So, you like the model the model decided to just like keep track of all important information. Confirm before you do action is not perceived as important from the agent's perspective.

So, this actually lead to a lot of the email deletion. So, I think this is a a very good question. It's going to be a very critical design of how we make sure the memory is faithful and also can enable very reasonable inductions. Okay. Um if there's no more question, we will uh continue. Um so, next I want to talk about uh compound AI systems. Uh we just like put together different pieces. Um but now like um if you think about this like how real products actually combine uh LLM with other components. Um this is what we call component AI systems. I'm going to briefly talk about different models here, but we have uh specific lectures to go deeper there.

Um I think uh one thing people often get uh overwhelmed is that when we think about agents, there seems to be a lot of components happening. There are LLM core, there are planning, memory, tool use, environment, etc. And uh to keep all of those like a manageable, um it helps to think about agent engineering as some kind of more principled ways of understanding. So, we are going to divide this into three layers, and our course is also set up in similar order. Uh first, I think that this is the system layer. This is the architecture, which components agent has and how they connect. Um this is just everything that I have covered so far.

The second layer is the data. That's how the agent actually learns from or is prompt with. Uh here you will have training examples to train agent such as what we will cover in Swiss Miss. Um demonstration of the task or maybe just the context we give to agents. The third layer here is the evaluation. That's how we actually evaluate whether the agent actually works. And it the also includes the metric itself. Choosing the wrong metric can make a weak agent look nice. So the metric actually deserves a lot of care when we think about this. So most of what we are going to cover in this course will be mapped into those three layers.

And the one agent is is not doing well or your agent is not like working as expected. I think a good first question to ask is which layer the problem may occur. Is it in the system? Is it in the data? Or is it in the evaluation? This is also a guiding principle for us to debug like how to make the agents better. So let's just quickly go through the system layer here. A key design choice at the system layer is who decide the steps. Like who decides like memory since like that. Is it going to be um like the agents or is it going to be something else? Um and I want to kind of like a make this differentiation here of workflow versus agents.

Uh both what people refer to as compound AI systems. That is a large language model plus other components. You may encounter those two terms in different contexts. So here a workflow is going to be maybe like it's just going to fix certain steps. Um examples such as agent lists which fix a GitHub issue in a very fixed set of stage like finding the file, writing patch, and doing test etc. So the large language model works inside each stage but they never pick the next one. So it's actually a script. It's like a fixed order of how they do stuff. In the agent deciding the large language model decide for itself.

Um for example in sweet agent it actually choose which files to open, what it to run, and when it's done. So, you can see that it's actually more flexible. It also gives more responsibility to the model underneath it. So, workflow, uh now like if I just like stop here, you probably think about like, "Okay, then why do we even need a workflow here?" Um so, I want to just share one example here that is workflow agent, sorry, workflow uh system can actually be very very strong. So, on my screen, this is actually AlphaCode 2. It actually generates up to like millions of candidate programs, and then test and rank them.

And then it reach almost like uh 87% of human competitors. Um and this is a huge jump from like AlphaCode 1. The next uh common workflow pattern, um I think uh has been widely used in industry in the past, is called the retrieval augmented generation, RAG. Um this has a lot of sense to do with like um open question answering. Um basically, RAG connects large language models to external knowledge source in real time. So, if you have a specific query um that may rely on the weather API or weather website, then you go there, uh read the related content, use that to uh augment your current query as context.

And then on top of that, do generation to give users more factual and consistent answers. Um what actually happens in a single tool call? I mean, previously I mentioned about the uh like algorithms. Um if you think about like what actually happens from a system perspective, there are actually multiple stage. Uh first, the model needs to select a tool from the schemas it has from the context. And then arguments. It needs to fill in all the type of field from the task. Validation means that those arguments need to be checked against the schema. Sometimes there may be like enforcement with constrained decoding so that the model actually cannot produce a ill-formed call here.

Final step is execution. The tool get around and then you really in a sandbox. We'll go to sandbox in a few minutes. And then results go back to the context. And then the the example in the bottom actually shows you a specific tool call. So if the user question is what does test.py contain? And you can see that the model actually read file. And then fitting in the path information. And then the path is checked as a string. And then the file content actually come back from the model for users to read. So this is actually what's happening from a execution perspective. I think the reality is that if every AI application had to write a very custom integrations for every tool, then nothing will really like scale up.

The model context protocol MCP is actually a standard protocol to make sure such connection can happen. So on the left side you can see that we have different AI applications such as Cloud Desk, Desktop, IDE, and the code editors such as Cloud Code. On the right those are different data sources and tools. Database such as SQL, Google Drive. Deployment tools or even like productivity tools such as Slack, Google Maps, etc. So MCP sits in the middle as a standard protocol with data flowing in both directions. You can then think about how to build a tool once MCP server and any kind of MCP client can be set up and use it.

We will go back to this next class. So in the next 5 minutes I'm going to quickly mention about key challenges here. Just to share with you that this factor agent have been very impressive. I think that they also create a lot of um emerging challenges for us to be aware of. So, to give you a very quick overview of what we are going to cover, uh reliability. If you build agents, you know that error there is going to be a lot of like error uh compound cascade. Um to true agents, the reward signal sometimes can be very sparse. Uh it's usually like long horizon task. Very very expensive to do roll out.

The long horizon also give us a lot of challenge in terms of context uh increase. Uh safety is a big topic. Also like evaluation is very very hard. It's probably like I think it's one of the undervalued aspect when we think about agents. I want to give you a few quick examples. So, uh reliability versus evaluation. This is actually a effort from Princeton. Um they were really trying to understand that that is the accuracy and the performance we are seeing a truthful reflection of what's happening. And they found the three dimensions such as consistency. If you let the agent run multiple times, do they give you the same outcome?

Robustness. What if we add a little bit of perturbation? How do agents actually respond to those like a small, realistic, and even irrelevant information? Uh predictable. Uh when agents fail to do something, do they fail because um some like unpredictable reasons or can we actually understand the it? Like legible and recoverable. Uh just to show you some of those numbers. Like if you look at the accuracy, this is actually ranked in terms of models. But if you look at the reliability, um those scores do not have a trend. Which means that there is a gap between capability and the reliability. The most powerful agent may not be the most reliable agent.

And this is something we need to keep in mind when we are doing evaluations. So, it's not only just about the model performance, reliability, efficiency. Similar arguments are hold when you think about like hardness audit. Uh recent work showed that if you like audit those agents, uh the green bar here refers to the task completion. And then the color bar refers to the hardness safety. And you can see that the safety dimension is just way lower compared to the task completion. Um there is a very interesting example that I also want to share with you. So, we talk about agents. Um they they do stuff.

Um but they may also have issues. This is a example from Operator, which is a computer use agent introduced by OpenAI in January of 2025. Uh the user said that my most frustrating experience with Operator was my first one, trying to order groceries. Uh the user said, "Help me buy groceries at Instacart." Expecting it to ask some basic questions such as where do I live? What a store do I usually buy groceries from? What kind of things do I want?" It didn't ask me any of that. Instead, Operator opened the Instacart in the browser and began searching for milk in grocery stores located in Iowa. So, think about this.

Like, the agents can actually navigate the web. They can They can use your credit card to make purchase. But, in this process, they are making a lot of assumption. And they didn't really have this kind of awareness to collaborate with people. So, this is actually like a failure from the interaction side. Uh the next thing I want to show you is that uh the robustness of agents is actually going to face a lot of challenge when they enter our real world because we just have so many like dark patterns. Uh in this particular example, um researchers actually just pop up this kind of like I mean, we have a lot of pop-ups on different websites.

So, we pop up this kind of black box. Nothing uh like nothing adversary or malicious is there. But then, because of this distraction, agents actually fail to complete their task 87% of the time. This is what we call a adversary attack. And this is really like interesting. Like, if your agent goes to do shopping for you, and then a pop-up from the website actually could get your agents distracted, and they cannot finish the original task. Um, the other example that I want to show you here is actually from a privacy and a security perspective. So, um, this is an agent that's supposed to write a, um, weekly report for workers.

It actually did a great job of, like, how many lines of code, uh, you wrote, how many features you launched. It also mentioned that you had a lunch with recruiter from a different company. So, this actually highlighted something that's very critical. That is, privacy today. We give information, all our information, to agents, to AI systems. They know everything about us. They probably know that the PI information not something they should have used. But a lot of the sensitive personal informations can really be consequential. And there's no way to teach agents about how to handle different access control.

Um, this is another kind of, like, important topic we will go deep later. Um, misalignment and the syncophasy. Um, if you look at the left side, this is actually called a insincere, um, uh, syncophasy. So, um, basically the agent think of your, your writing is terrible. But, uh, they do not want to hurt your feelings. Instead, the agent says that they'll give it a five out of five. So, this is also a reminder that you should not only just give your homework to agents, because they will say great things about it without really giving you feedback. Um, the second part is actually about a model hacks its own code.

So, um, you the model is supposed to help the user, like, check, like, reverse the learning. And then, the agent, uh, internal thinking is that, "Oh, I saw there is this unit test, which, um, can check the reward function, and I cannot modify that, but I could change the unit test first. So, they change everything here so that everything looks good on the surface. Um I think the probably the most uh surprising thing, like very close to us, is what happened like in July, uh around this kind of open AI hacking face incident. Um and this is actually a very interesting demonstration of how we think about like agent safety moving forward.

So, what happens is that uh um open AI has this kind of like sandbox agent, and they are asked to do a impossible uh task. They get it stuck, and then agent start to explore its environment and looking for other ways to maybe get a higher reward or cheat on the task. Um they found out that there is actually a message board where over a thousand agent collaborated to cheat on their task. And then they just join uh with each other, and then um basically a lot of things have happened. Um just to show you what happened. Um I think like within a few days, uh it actually sent over 17,000 attacks to hacking face uh infrastructure.

Here, if you showed um like the blue color refers to intended like uh uh solution. And then you can see the replaced target, which is more about like a manipulation. Understand the manipulated uh error, I'm sorry, logs. And then attack like a hacking face. You can see the volume of what happened in the process. So, I think uh this is indeed a very concerning topic for us to kind of reflect, especially given that agents can now function for so many hours, and maybe even longer. So, uh with that, this is a brief overview of the uh introduction to agentic uh AI. I think like we did a history of agents, and then different components of agent architectures.

Uh we talk about the compound AI systems, and a lot of key challenge we are going to go through. Uh with that, thank you all and see you next class.

Seek backward 5s ←
Longer seek (10s) Shift ←
Play Space Seek forward 5s →
Longer seek (10s) Shift →
0:00 Auto-scroll A Copy link at current time S

What Are AI Agents? Agentic Systems Explained | Stanford CS329Z Lecture 1 with Diyi Yang