Build Enterprise AI Agent from Scratch |

[Ch 1] Introduction to AI Agents — Why Do We Need Them?

Tue, 07 Apr 2026 00:00:00 +0000

Before you write a single line of LangGraph, you need a clear mental model of what an AI agent actually is — and more importantly, what it isn’t. This chapter builds that foundation.

What Is an AI Agent?

An AI agent is a system that uses a language model to decide which actions to take, executes those actions (via tools), observes the results, and then decides what to do next — repeating this cycle until a goal is reached.

The key word is decide. Unlike a simple LLM call that takes input and produces output in one shot, an agent has a control loop. It can take multiple steps, use external tools, and adapt based on intermediate results.

The Spectrum: Chatbot → Pipeline → Agent

Most AI systems fall somewhere on this spectrum:

Type	Control	Steps	Tools	Example
Chatbot	Static	1	None	FAQ bot, direct LLM reply
LLM Pipeline	Fixed	N (predetermined)	Optional	Summarize → Translate → Format
AI Agent	Dynamic	N (decided at runtime)	Yes	Research → Plan → Write → Verify

Fig 1: The spectrum from stateless chatbots to dynamic agents

A chatbot is a single LLM call: user sends a message, LLM replies. It has no memory between sessions and no ability to use external tools.

An LLM pipeline chains multiple LLM calls in a predetermined order. The steps are fixed: you always summarize, then translate, then format. There’s no branching — the flow doesn’t change based on what the model produces.

An AI agent decides its own next step based on the current state. It might call a search tool, decide the result is insufficient, call it again with a refined query, then write an answer. The number and order of steps are not known in advance.

The Agent Loop (ReAct)

The dominant pattern for AI agents is ReAct (Reason + Act), introduced by Yao et al. (2022). The agent alternates between two phases:

Reason — the LLM thinks about what to do given the current context
Act — the agent executes an action (tool call) and receives an observation

This cycle repeats until the LLM decides it has enough information to produce a final answer.

graph TD A([User Input]) --> B[LLM: Think\nWhat should I do next?] B --> C{Tool call\nneeded?} C -- Yes --> D[Execute Tool] D --> E[Observe Result] E --> B C -- No --> F([Final Answer\nto User]) style A fill:#4CAF50,color:#fff,stroke:none style F fill:#2196F3,color:#fff,stroke:none style D fill:#FF9800,color:#fff,stroke:none style B fill:#9C27B0,color:#fff,stroke:none

Fig 2: The ReAct agent loop — Reason and Act until a final answer is reached

The loop terminates when the LLM produces a regular message (no tool call). In practice this means the LLM’s output is inspected after each step: if it contains a tool call, the system runs it and feeds the result back; if not, the loop ends.

Real-World Use Cases

Agents shine when a task requires dynamic decision-making with external information or actions:

Use Case	Why It Needs an Agent
Automated report generation	Must retrieve data, validate it, format by section — each step depends on previous
Code review assistant	Reads a PR, calls static analysis tools, queries docs, writes structured feedback
Presentation builder	Parses user intent, generates slide content, creates charts, assembles file
Test case generator	Understands spec, identifies scenarios, generates cases, checks for coverage
Customer support	Looks up account info, checks policy docs, escalates if needed

In every case, the number of steps is not fixed in advance. The agent decides what to do based on what it finds.

When NOT to Use an Agent

This is just as important.

Use a simple LLM pipeline when:

The task has a fixed, known structure (always: extract → classify → output)
Latency is critical — each agent step adds 1–5 seconds
The task is well-defined enough that a deterministic pipeline won’t miss edge cases

Use a basic LLM call when:

You just need to summarize, translate, or classify a single piece of text
There’s nothing to look up externally
You want 100% predictable, testable behavior

Agents introduce non-determinism — the same input can take different paths. They are also slower and more expensive per task. Don’t reach for an agent when a two-step pipeline does the job.

⚠️ A common mistake: wrapping every feature in an agent “for flexibility.” The result is a slow, expensive, hard-to-test system. Start with the simplest thing that works.

The Agent Loop in Plain Python

Before introducing any framework, here’s what the agent loop looks like as pure Python pseudocode. This will help you understand exactly what LangGraph is doing for you later.

# agent_loop.py — Plain Python pseudo-code (no framework)
import os
import json
from openai import OpenAI

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))

# --- Tool registry ---
def search_web(query: str) -> str:
 """Simulate a web search (replace with real implementation)."""
 return f"[Search results for '{query}': ...]"

def calculate(expression: str) -> str:
 """Evaluate a math expression safely."""
 try:
 result = eval(expression, {"__builtins__": {}})
 return str(result)
 except Exception as e:
 return f"Error: {e}"

TOOLS = {
 "search_web": search_web,
 "calculate": calculate,
}

# OpenAI-format tool schemas
TOOL_SCHEMAS = [
 {
 "type": "function",
 "function": {
 "name": "search_web",
 "description": "Search the web for current information.",
 "parameters": {
 "type": "object",
 "properties": {"query": {"type": "string"}},
 "required": ["query"],
 },
 },
 },
 {
 "type": "function",
 "function": {
 "name": "calculate",
 "description": "Evaluate a mathematical expression.",
 "parameters": {
 "type": "object",
 "properties": {"expression": {"type": "string"}},
 "required": ["expression"],
 },
 },
 },
]

# --- The agent loop ---
def run_agent(user_input: str, max_steps: int = 10) -> str:
 messages = [
 {"role": "system", "content": "You are a helpful assistant. Use tools when needed."},
 {"role": "user", "content": user_input},
 ]

 for step in range(max_steps):
 print(f"\n[Step {step + 1}] Calling LLM...")
 response = client.chat.completions.create(
 model="gpt-4o-mini",
 messages=messages,
 tools=TOOL_SCHEMAS,
 tool_choice="auto",
 )

 message = response.choices[0].message

 # No tool call → agent is done, return final answer
 if not message.tool_calls:
 print("[Step] Final answer reached.")
 return message.content

 # Execute each requested tool call
 messages.append(message) # add assistant message with tool_calls
 for tool_call in message.tool_calls:
 fn_name = tool_call.function.name
 fn_args = json.loads(tool_call.function.arguments)

 print(f"[Step] Calling tool: {fn_name}({fn_args})")
 result = TOOLS[fn_name](**fn_args)
 print(f"[Step] Tool result: {result[:100]}...")

 # Add the observation back into the conversation
 messages.append({
 "role": "tool",
 "tool_call_id": tool_call.id,
 "content": result,
 })

 return "Max steps reached without a final answer."


if __name__ == "__main__":
 answer = run_agent("What is the square root of the number of days in a leap year?")
 print(f"\nFinal Answer: {answer}")

What this shows:

The conversation history (messages) is the agent’s working memory
The loop continues as long as the model keeps making tool calls
Each tool result is appended as a "role": "tool" message before the next LLM call
max_steps is a safety guard against infinite loops

This is essentially what LangGraph automates — the graph manages the loop, the state replaces the raw messages list, and nodes replace the explicit tool execution code.

.env.example

# .env.example
OPENAI_API_KEY=your-api-key-here

💡 Using Ollama instead? Replace the OpenAI client with:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
# Use model="llama3.2" or any model you have pulled
Make sure Ollama is running (ollama serve) and the model supports tool use.

Summary

Concept	Key Takeaway
AI Agent	An LLM in a control loop that decides actions dynamically
ReAct	Alternate between Reasoning (LLM) and Acting (tools)
When to use	Dynamic tasks requiring external info or multi-step decisions
When NOT to use	Fixed pipelines, latency-sensitive tasks, simple transformations
The loop	LLM call → tool call → observation → repeat until final answer

In the next chapter, we’ll break down the four core components of an agent system and introduce one of the most important (and most overlooked) concepts in production AI: Context Engineering.

[Ch 2] AI Agent Components & Context Engineering

Tue, 07 Apr 2026 00:00:00 +0000

Knowing that an agent has a “loop” is not enough to build a good one. You need to understand what the system is made of — and exactly what the LLM sees at every step. This chapter covers both.

The Four Core Components

Every AI agent system, regardless of framework, is built from four components:

graph LR A[🧠 LLM Backbone] --- B[🔧 Tools] A --- C[🗃️ Memory] A --- D[🗺️ Planning] style A fill:#7C3AED,color:#fff,stroke:none style B fill:#2563EB,color:#fff,stroke:none style C fill:#059669,color:#fff,stroke:none style D fill:#D97706,color:#fff,stroke:none

Fig 1: The four pillars of an AI agent

1. LLM Backbone

The language model is the brain. It decides what to do next: whether to call a tool, what arguments to pass, and when to stop. The quality, speed, and cost of your agent depend heavily on which model you choose.

Key decisions:

Model choice: GPT-4o for complex reasoning; GPT-4o-mini for simpler, high-throughput tasks
Temperature: Keep it low (0–0.3) for tool-calling agents — you want deterministic decisions, not creative variance
Context window size: Determines how much history and retrieved content the model can “see”

2. Tools

Tools are functions the LLM can call to interact with the outside world. Common categories:

Category	Examples
Information retrieval	Web search, vector store lookup, database query
Action execution	File write, API call, form submission
Computation	Code execution, math evaluation, data parsing
Communication	Send email, post to Slack, create a ticket

A tool is just a function with a clear name, description, and typed parameters — the model reads the description to know when and how to use it.

3. Memory

Memory is how the agent maintains context across steps and sessions. There are four distinct memory types (covered in depth in Ch 6):

In-context: The current conversation history in the message list
Episodic: Persisted conversation state (checkpointer / DB)
Semantic: Long-term knowledge retrieved via RAG (vector store)
Procedural: The system prompt — “how to behave” instructions baked into every request

4. Planning

Planning is the agent’s ability to decompose a complex goal into steps. This ranges from implicit (the LLM decides step-by-step via ReAct) to explicit (a dedicated “planner” LLM call generates a structured plan that an “executor” then follows).

For most enterprise agents, the implicit ReAct planning is sufficient — and simpler to debug. Explicit multi-step planners are powerful but add complexity; reach for them only when the task requires long-horizon coordination across many sub-tasks.

Prompt Engineering vs. Context Engineering

Here’s a distinction that will save you weeks of frustration in production.

Prompt Engineering is about what words you use — how to phrase instructions to get the model to behave the way you want. It’s the craft of writing good system prompts, user messages, and few-shot examples.

Context Engineering is about what the model actually sees — managing everything that occupies the context window at each step of the agent loop.

Context Engineering is a superset of Prompt Engineering.

In a single LLM API call, you have to write a good prompt. In an agent system, you have to do that and manage every byte in the context window across potentially dozens of steps.

Anatomy of a Context Window

Every time the LLM is called, it receives a context window. In an agent loop, that window contains multiple layers:

block-beta columns 1 block:context["Context Window (e.g. 128,000 tokens)"] A["📋 System Prompt\n(instructions, persona, output format)"] B["🔧 Tool Schemas\n(name, description, parameters for each tool)"] C["💬 Conversation History\n(all prior messages + tool results)"] D["📚 Retrieved Documents\n(RAG chunks injected for this turn)"] E["❓ Current User Message"] end style A fill:#7C3AED,color:#fff style B fill:#2563EB,color:#fff style C fill:#059669,color:#fff style D fill:#D97706,color:#fff style E fill:#DC2626,color:#fff

Fig 2: Everything competing for space in the context window

The crucial insight: tools eat tokens even when they’re not used.

A typical tool schema (name + description + parameter definitions) costs 100–300 tokens. If your agent has 15 tools, that’s 1,500–4,500 tokens consumed before any conversation history or retrieved documents. For an agent with a 32K token budget running a long conversation, this becomes a real constraint.

Token Budget: A Practical Example

Let’s measure the real cost of each context component:

# token_budget.py
import os
import tiktoken

# GPT-4o uses the o200k_base encoding
ENCODING = tiktoken.get_encoding("o200k_base")

def count_tokens(text: str) -> int:
 return len(ENCODING.encode(text))

# --- Simulate context components ---

SYSTEM_PROMPT = """You are an expert document analyst assistant. You help users
understand and extract information from complex technical documents. You are precise,
thorough, and always cite your sources. When a document is provided, read it carefully
before answering. Format your responses in clear markdown."""

TOOL_SCHEMA_EXAMPLE = """Tool: search_documents
Description: Search through the document knowledge base for relevant passages.
Parameters:
 - query (string, required): The search query to find relevant document sections
 - max_results (integer, optional): Maximum number of results to return (default: 5)
 - document_filter (string, optional): Filter results to a specific document by name"""

# Simulate 10 back-and-forth turns
CONVERSATION_HISTORY = """
User: Can you find all mentions of the payment terms in the contract?
Assistant: [calls search_documents(query="payment terms")]
Tool Result: Found 3 sections mentioning payment terms: Section 4.2, 7.1, and Appendix B.
Assistant: I found payment terms in three locations. Section 4.2 covers...
[... 9 more turns ...]
""" * 3 # multiply to simulate a longer conversation

RETRIEVED_DOCS = """
[Retrieved chunk 1 - Section 4.2]: Payment shall be made within 30 days of invoice...
[Retrieved chunk 2 - Section 7.1]: Late payment penalties apply at 1.5% per month...
[Retrieved chunk 3 - Appendix B]: Extended payment plans available upon request...
""" * 5 # 5 retrieved chunks

CURRENT_QUERY = "What is the penalty for paying 60 days late?"

# --- Count and report ---
components = {
 "System Prompt": SYSTEM_PROMPT,
 "Tool Schema (x10 tools estimate)": TOOL_SCHEMA_EXAMPLE * 10,
 "Conversation History (10 turns)": CONVERSATION_HISTORY,
 "Retrieved Documents (5 chunks)": RETRIEVED_DOCS,
 "Current User Query": CURRENT_QUERY,
}

total = 0
print(f"{'Component':<40} {'Tokens':>8} {'% of 32K':>10}")
print("-" * 62)
for name, content in components.items():
 tokens = count_tokens(content)
 total += tokens
 print(f"{name:<40} {tokens:>8,} {tokens/32768*100:>9.1f}%")

print("-" * 62)
print(f"{'TOTAL':<40} {total:>8,} {total/32768*100:>9.1f}%")
print(f"\nRemaining budget (32K model): {max(0, 32768 - total):,} tokens")
print(f"Remaining budget (128K model): {max(0, 131072 - total):,} tokens")

Running this produces output like:

Component Tokens % of 32K
--------------------------------------------------------------
System Prompt 52 0.2%
Tool Schema (x10 tools estimate) 1,430 4.4%
Conversation History (10 turns) 1,710 5.2%
Retrieved Documents (5 chunks) 975 3.0%
Current User Query 16 0.0%
--------------------------------------------------------------
TOTAL 4,183 12.8%

Remaining budget (32K model): 28,585 tokens
Remaining budget (128K model): 126,889 tokens

Ten turns into a conversation with a moderate number of tools and RAG chunks, you’ve used only 13% of a 32K window — but that grows fast. At 30+ turns with larger documents, you’ll hit limits and need to trim.

Context Engineering Strategies

Once you understand what’s in the context window, you can manage it deliberately:

1. System Prompt — Keep It Focused

The system prompt runs on every single call. Every unnecessary sentence costs tokens across your entire agent’s lifetime. Write precisely:

# ❌ Vague, padded
BAD_SYSTEM_PROMPT = """
You are a very helpful and friendly AI assistant. You are smart and knowledgeable
about many things. You always try to be nice and helpful to users. When users ask
questions, you do your best to answer them accurately and helpfully...
"""

# ✅ Precise, structured
GOOD_SYSTEM_PROMPT = """You are a document analysis assistant.

CAPABILITIES:
- Search documents using search_documents tool
- Extract and cite specific passages
- Summarize findings in structured markdown

CONSTRAINTS:
- Only answer questions about documents in the knowledge base
- Always cite the source section when quoting
- If information is not found, say so explicitly
"""

2. Tool Schemas — Prune Unused Tools

Don’t pass every tool on every call. If your agent has a “create_document” tool but the user just asked a read-only question, remove it from the current call. In LangGraph, you can swap the tool list dynamically per node.

3. Conversation History — Trim Aggressively

Keep a sliding window of the last N turns, or summarize old turns into a single “context block.” We’ll implement this in Ch 6.

4. RAG Chunks — Quality Over Quantity

Don’t inject 20 retrieved chunks “just in case.” Use a relevance threshold: only inject chunks above a similarity score of 0.75. Three highly relevant chunks beat ten mediocre ones — and cost far less context.

The Mental Model

Think of the context window as a whiteboard that the LLM reads before every response. You control everything on that whiteboard:

What instructions appear (system prompt)
What tools are available (tool schemas)
What happened before (conversation history)
What facts are relevant (retrieved documents)
What the user just asked (current message)

Context Engineering is the discipline of making every token on that whiteboard count.

.env.example

# .env.example
OPENAI_API_KEY=your-api-key-here

💡 Ollama note: The token counting above uses tiktoken (OpenAI’s tokenizer). Ollama models use different tokenizers (Llama uses sentencepiece). Token counts will vary, but the principles — keep system prompts tight, prune unused tools, trim history — apply universally.

Summary

Concept	Key Takeaway
LLM Backbone	The decision-maker; choose model + temperature carefully
Tools	Functions the LLM can call; each schema costs tokens
Memory	Four types: in-context, episodic, semantic, procedural
Planning	Implicit (ReAct) vs. explicit; start with implicit
Prompt Engineering	Craft of writing good instructions and examples
Context Engineering	Managing everything in the context window across all steps
Token Budget	System prompt + tool schemas + history + RAG + query = limited budget

In the next chapter, we get hands-on with LangChain and LangGraph — the frameworks that automate the agent loop we’ve been building mentally.

[Ch 3] Getting Started with LangChain & LangGraph

Tue, 07 Apr 2026 00:00:00 +0000

In Ch 1 we built the agent loop in raw Python. That worked, but it required us to manually manage the message list, handle tool routing, and build the loop ourselves. LangChain and LangGraph exist to automate exactly that — while adding persistence, streaming, debugging tooling, and a clean abstraction for complex workflows.

This chapter gives you a solid conceptual and practical foundation before we build a full agent in Ch 4.

Installation

pip install langchain-core langchain-openai langgraph

# .env.example
OPENAI_API_KEY=your-api-key-here

💡 Ollama users: pip install langchain-ollama — we’ll show the swap at the end of this chapter.

Part 1: LangChain Core Concepts

LangChain provides the essential building blocks. You don’t need all of LangChain to use LangGraph — but these four concepts appear everywhere.

1.1 Messages

LangChain wraps every role in the conversation as a typed message object. These correspond exactly to the OpenAI Chat Completions API roles:

# messages_demo.py
from langchain_core.messages import (
 SystemMessage,
 HumanMessage,
 AIMessage,
 ToolMessage,
)

# System prompt
system = SystemMessage(content="You are a helpful assistant.")

# User turn
user = HumanMessage(content="What is the capital of France?")

# Assistant reply (no tool call)
assistant = AIMessage(content="The capital of France is Paris.")

# When a tool is called, the AIMessage contains tool_calls
assistant_with_tool = AIMessage(
 content="",
 tool_calls=[
 {
 "id": "call_abc123",
 "name": "search_web",
 "args": {"query": "capital of France"},
 }
 ],
)

# The tool result is always a ToolMessage
tool_result = ToolMessage(
 tool_call_id="call_abc123",
 content="Paris is the capital and most populous city of France.",
)

# Build a conversation list (this is what gets sent to the LLM)
conversation = [system, user, assistant_with_tool, tool_result]
print(f"Conversation has {len(conversation)} messages")

Why typed messages matter: LangGraph’s state uses these types directly. When you add a message to the state, the framework knows whether it’s an AI message with tool calls or a tool result — and routes accordingly.

1.2 ChatOpenAI

The ChatOpenAI class wraps the OpenAI API with LangChain’s interface. The most important parameters:

# llm_setup.py
import os
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
 model="gpt-4o-mini", # cheaper, fast — good for most agent tasks
 temperature=0.1, # low randomness for consistent tool-calling
 api_key=os.environ.get("OPENAI_API_KEY"),
)

# Simple invoke
response = llm.invoke([HumanMessage(content="Hello!")])
print(response.content) # "Hello! How can I help you today?"
print(type(response)) # <class 'langchain_core.messages.ai.AIMessage'>

💡 Ollama swap: Replace the entire ChatOpenAI block with:
from langchain_ollama import ChatOllama
llm = ChatOllama(model="llama3.2", temperature=0.1)
Everything else in this chapter stays identical.

1.3 The `@tool` Decorator

@tool converts a Python function into a LangChain tool. The framework automatically generates the tool schema (name, description, parameters) from the function’s name, docstring, and type annotations.

# tools.py
from langchain_core.tools import tool
from pydantic import BaseModel, Field

# Simple tool — docstring becomes the description
@tool
def get_current_time() -> str:
 """Get the current UTC time in ISO 8601 format."""
 from datetime import datetime, timezone
 return datetime.now(timezone.utc).isoformat()

# Tool with Pydantic input schema — best practice for complex inputs
class SearchInput(BaseModel):
 query: str = Field(description="The search query string")
 max_results: int = Field(default=5, description="Maximum number of results (1-20)")

@tool("search_documents", args_schema=SearchInput)
def search_documents(query: str, max_results: int = 5) -> str:
 """Search the document knowledge base for passages relevant to the query."""
 # In a real system, this calls your vector store
 return f"[Simulated] Found {max_results} results for: '{query}'"

# Inspect what the framework generated
print(get_current_time.name) # "get_current_time"
print(get_current_time.description) # "Get the current UTC time in ISO 8601 format."
print(search_documents.args_schema.schema()) # Full JSON schema

# Bind tools to the LLM — this sends schemas in every API call
tools = [get_current_time, search_documents]
llm_with_tools = llm.bind_tools(tools)

The key advantage of @tool over raw function schemas: the LLM SDK automatically serializes/deserializes arguments and validates them against your Pydantic schema.

1.4 RunnableConfig

RunnableConfig is how you pass runtime configuration through any LangChain/LangGraph call without polluting the message stream. You’ll use it extensively in Ch 5 for injecting user context into tools.

# runnable_config_demo.py
from langchain_core.runnables import RunnableConfig

# Pass config through invoke
config: RunnableConfig = {
 "configurable": {
 "user_id": "user-42",
 "session_id": "sess-abc",
 "thread_id": "thread-001", # LangGraph uses this for checkpointing
 },
 "callbacks": [], # attach tracing callbacks here
}

response = llm_with_tools.invoke(
 [HumanMessage(content="What time is it?")],
 config=config,
)

Inside a tool, you can access this config via the special config parameter:

@tool
def my_tool(query: str, config: RunnableConfig) -> str: # config is injected automatically
 """A tool that knows who is calling it."""
 user_id = config.get("configurable", {}).get("user_id", "unknown")
 return f"Hello user {user_id}, you searched for: {query}"

Part 2: LangGraph Core Concepts

LangGraph builds stateful agent workflows as directed graphs. The key insight: state flows through nodes, and edges determine routing.

2.1 The State

State is a TypedDict that every node reads from and writes to. The Annotated type with add_messages is the standard pattern for message lists — it appends rather than overwrites:

# state.py
from typing import Annotated, TypedDict
from langchain_core.messages import AnyMessage
from langgraph.graph.message import add_messages

class AgentState(TypedDict):
 # add_messages reducer: new messages are appended, not replaced
 messages: Annotated[list[AnyMessage], add_messages]
 # You can add other fields for custom state
 step_count: int
 error: str | None

2.2 Nodes

A node is any Python function (sync or async) that takes the current state and returns a dict of updates:

# nodes.py
from langchain_core.messages import SystemMessage

SYSTEM_PROMPT = "You are a helpful assistant. Use tools when needed."

def agent_node(state: AgentState) -> dict:
 """The brain: calls the LLM with the current message history."""
 messages = [SystemMessage(content=SYSTEM_PROMPT)] + state["messages"]
 response = llm_with_tools.invoke(messages)
 return {"messages": [response]} # add_messages will append this

def increment_step(state: AgentState) -> dict:
 """A simple state-tracking node."""
 return {"step_count": state.get("step_count", 0) + 1}

2.3 Edges and Conditional Edges

Edges connect nodes. A conditional edge uses a function to decide where to go next:

# routing.py
from langgraph.prebuilt import ToolNode

def should_continue(state: AgentState) -> str:
 """
 Return "tools" if the last AI message has tool calls,
 otherwise return "end" to finish.
 """
 last_message = state["messages"][-1]
 if hasattr(last_message, "tool_calls") and last_message.tool_calls:
 return "tools"
 return "end"

2.4 Building the Graph

# graph.py
from langgraph.graph import StateGraph, END
from langgraph.prebuilt import ToolNode

def build_graph(tools: list) -> StateGraph:
 tool_node = ToolNode(tools) # handles tool execution + ToolMessage creation

 graph = StateGraph(AgentState)

 # Add nodes
 graph.add_node("agent", agent_node)
 graph.add_node("tools", tool_node)

 # Entry point
 graph.set_entry_point("agent")

 # Conditional edge: after agent runs, decide what happens next
 graph.add_conditional_edges(
 "agent",
 should_continue,
 {
 "tools": "tools", # if tool call → run tools
 "end": END, # if no tool call → stop
 },
 )

 # After tools run, always go back to agent
 graph.add_edge("tools", "agent")

 return graph.compile()

Here’s the graph structure visualized:

graph LR START([START]) --> agent agent -->|"tool calls?"| cond{{"should_continue"}} cond -->|"yes → tools"| tools["ToolNode"] cond -->|"no → end"| END([END]) tools -->|"always"| agent style START fill:#4CAF50,color:#fff,stroke:none style END fill:#F44336,color:#fff,stroke:none style agent fill:#9C27B0,color:#fff,stroke:none style tools fill:#FF9800,color:#fff,stroke:none style cond fill:#2196F3,color:#fff,stroke:none

Fig 1: The minimal ReAct agent graph in LangGraph

Part 3: Hello World Agent

Let’s put it all together into a runnable Hello World agent:

# hello_agent.py
import os
from typing import Annotated, TypedDict

from langchain_core.messages import AnyMessage, HumanMessage, SystemMessage
from langchain_core.tools import tool
from langchain_openai import ChatOpenAI
from langgraph.graph import END, StateGraph
from langgraph.graph.message import add_messages
from langgraph.prebuilt import ToolNode

# ── State ──────────────────────────────────────────────────────────────────
class AgentState(TypedDict):
 messages: Annotated[list[AnyMessage], add_messages]

# ── LLM ────────────────────────────────────────────────────────────────────
llm = ChatOpenAI(
 model="gpt-4o-mini",
 temperature=0.1,
 api_key=os.environ.get("OPENAI_API_KEY"),
)

# ── Tools ───────────────────────────────────────────────────────────────────
@tool
def add_numbers(a: float, b: float) -> float:
 """Add two numbers together and return the result."""
 return a + b

@tool
def get_word_count(text: str) -> int:
 """Count the number of words in the given text."""
 return len(text.split())

TOOLS = [add_numbers, get_word_count]
llm_with_tools = llm.bind_tools(TOOLS)

# ── Nodes ────────────────────────────────────────────────────────────────────
SYSTEM_PROMPT = "You are a helpful assistant. Use tools when you need to compute things."

def agent_node(state: AgentState) -> dict:
 messages = [SystemMessage(content=SYSTEM_PROMPT)] + state["messages"]
 response = llm_with_tools.invoke(messages)
 return {"messages": [response]}

def should_continue(state: AgentState) -> str:
 last = state["messages"][-1]
 if hasattr(last, "tool_calls") and last.tool_calls:
 return "tools"
 return "end"

# ── Graph ─────────────────────────────────────────────────────────────────────
graph = StateGraph(AgentState)
graph.add_node("agent", agent_node)
graph.add_node("tools", ToolNode(TOOLS))
graph.set_entry_point("agent")
graph.add_conditional_edges("agent", should_continue, {"tools": "tools", "end": END})
graph.add_edge("tools", "agent")
app = graph.compile()

# ── Run ───────────────────────────────────────────────────────────────────────
if __name__ == "__main__":
 initial_state = {
 "messages": [HumanMessage(content="What is 1337 + 42? Also, how many words are in: 'the quick brown fox'?")]
 }

 # Invoke (blocking, returns final state)
 final_state = app.invoke(initial_state)
 print("\n=== Final Answer ===")
 print(final_state["messages"][-1].content)

 # Optional: print all messages to see the full trajectory
 print("\n=== Full Trajectory ===")
 for msg in final_state["messages"]:
 role = msg.__class__.__name__
 content = msg.content if msg.content else f"[tool calls: {msg.tool_calls}]"
 print(f"[{role}] {str(content)[:120]}")

Expected output:

=== Final Answer ===
I calculated both for you:
- 1337 + 42 = **1379**
- The sentence "the quick brown fox" contains **4 words**

=== Full Trajectory ===
[HumanMessage] What is 1337 + 42? Also, how many words are in: 'the quick brown fox'?
[AIMessage] [tool calls: [{'name': 'add_numbers', 'args': {'a': 1337, 'b': 42}...}]]
[ToolMessage] 1379.0
[ToolMessage] 4
[AIMessage] I calculated both for you: ...

Notice the trajectory: the LLM called both tools in one step (parallel tool calling), received both results, then composed the final answer.

LangChain vs. LangGraph: When to Use Which

	LangChain (LCEL chains)	LangGraph
Best for	Simple, linear pipelines	Stateful, multi-step agents
State	No built-in state	Full TypedDict state with reducers
Persistence	Manual	Built-in checkpointers
Loops	Not natively supported	First-class citizen
Human-in-the-loop	Awkward	Native `interrupt()`
Streaming	Token-level	Token-level + step-level events
Complexity	Low	Medium

Rule of thumb: If you need a fixed, linear chain (e.g., extract → classify → format), use LCEL. If you need a loop, branching, or persistence across sessions, use LangGraph.

In this series, we use LangGraph for everything from Ch 4 onwards.

.env.example

# .env.example
OPENAI_API_KEY=your-api-key-here

💡 Ollama swap: In hello_agent.py, replace the LLM block with:
from langchain_ollama import ChatOllama
llm = ChatOllama(model="llama3.2", temperature=0.1)
Note: smaller local models may not reliably trigger parallel tool calls. If results are inconsistent, simplify the query to one tool at a time.

Summary

Concept	Description
`HumanMessage / AIMessage / ToolMessage`	Typed wrappers for each conversation role
`ChatOpenAI`	LangChain’s OpenAI client; use `.bind_tools()` to attach tool schemas
`@tool`	Decorator that generates tool schemas from Python function signatures
`RunnableConfig`	Runtime config dict for thread IDs, user context, callbacks
`AgentState`	TypedDict with `Annotated[list, add_messages]` for message accumulation
`StateGraph`	LangGraph’s graph builder; nodes + edges + conditional routing
`ToolNode`	Prebuilt node that executes tool calls and produces `ToolMessage` results
`should_continue`	Routing function: “tools” if the last AI message has tool_calls, else “end”

In the next chapter, we build a complete, production-grade agent: persistent memory with a SQLite checkpointer, proper error handling, streaming output, and multi-turn conversation support.

[Ch 4] Build Your First Production-Ready Agent

Tue, 07 Apr 2026 00:00:00 +0000

In Ch 3 we built a Hello World agent that worked for a single query. In this chapter we build the real thing: a multi-turn agent that remembers conversation history across sessions, handles tool errors gracefully, streams its output token by token, and can run multiple independent user threads simultaneously.

This is the foundation every subsequent chapter builds on.

Project Structure

agent/
├── .env.example
├── state.py # AgentState definition
├── tools.py # Tool definitions
├── nodes.py # Agent and tool nodes
├── graph.py # Graph assembly + compile
└── main.py # Entry point: run + stream

Installation

pip install langchain-core langchain-openai langgraph

# .env.example
OPENAI_API_KEY=your-api-key-here

💡 Ollama: Replace ChatOpenAI in nodes.py with:

from langchain_ollama import ChatOllama
llm = ChatOllama(model="llama3.2", temperature=0.1)

Step 1 — Define the State

The state is the single source of truth for everything the agent knows at any point in the loop.

# agent/state.py
from typing import Annotated, TypedDict
from langchain_core.messages import AnyMessage
from langgraph.graph.message import add_messages


class AgentState(TypedDict):
 # add_messages reducer: new messages are *appended*, not replaced
 messages: Annotated[list[AnyMessage], add_messages]

Why add_messages matters: Without it, every node return would overwrite the entire message list. With it, returning {"messages": [new_msg]} appends new_msg to the existing history. This is what makes multi-turn conversations work.

Step 2 — Define the Tools

We’ll build a small task management assistant with three tools. Each uses a Pydantic schema for validated inputs.

# agent/tools.py
from datetime import datetime, timezone
from pydantic import BaseModel, Field
from langchain_core.tools import tool

# ── In-memory task store (replace with a real DB in production) ──────────────
_tasks: dict[str, dict] = {}

# ── Input schemas ─────────────────────────────────────────────────────────────

class CreateTaskInput(BaseModel):
 title: str = Field(description="Short title for the task")
 description: str = Field(description="Detailed description of what needs to be done")
 priority: str = Field(
 default="medium",
 description="Task priority: 'low', 'medium', or 'high'",
 )

class UpdateTaskInput(BaseModel):
 task_id: str = Field(description="The ID of the task to update")
 status: str = Field(description="New status: 'todo', 'in_progress', or 'done'")

class ListTasksInput(BaseModel):
 filter_status: str = Field(
 default="all",
 description="Filter by status: 'all', 'todo', 'in_progress', or 'done'",
 )

# ── Tool definitions ──────────────────────────────────────────────────────────

@tool("create_task", args_schema=CreateTaskInput)
def create_task(title: str, description: str, priority: str = "medium") -> str:
 """Create a new task and return its ID."""
 task_id = f"task-{len(_tasks) + 1:03d}"
 _tasks[task_id] = {
 "id": task_id,
 "title": title,
 "description": description,
 "priority": priority,
 "status": "todo",
 "created_at": datetime.now(timezone.utc).isoformat(),
 }
 return f"Created task {task_id}: '{title}' (priority: {priority})"


@tool("update_task_status", args_schema=UpdateTaskInput)
def update_task_status(task_id: str, status: str) -> str:
 """Update the status of an existing task."""
 if task_id not in _tasks:
 return f"Error: Task '{task_id}' not found."
 valid_statuses = {"todo", "in_progress", "done"}
 if status not in valid_statuses:
 return f"Error: Invalid status '{status}'. Use one of: {valid_statuses}"
 _tasks[task_id]["status"] = status
 return f"Updated {task_id} status to '{status}'."


@tool("list_tasks", args_schema=ListTasksInput)
def list_tasks(filter_status: str = "all") -> str:
 """List all tasks, optionally filtered by status."""
 if not _tasks:
 return "No tasks found."
 tasks = list(_tasks.values())
 if filter_status != "all":
 tasks = [t for t in tasks if t["status"] == filter_status]
 if not tasks:
 return f"No tasks with status '{filter_status}'."
 lines = [f"Found {len(tasks)} task(s):\n"]
 for t in tasks:
 lines.append(
 f" [{t['id']}] {t['title']} | {t['priority'].upper()} | {t['status']}"
 )
 return "\n".join(lines)


TOOLS = [create_task, update_task_status, list_tasks]

Key practices shown here:

Every tool has a Pydantic schema (args_schema=) — the LLM gets validated, typed inputs
Error messages are returned as strings (not exceptions) — tools should never crash the agent loop
The in-memory _tasks dict is a stand-in; swap with any DB in production

Step 3 — Define the Nodes

# agent/nodes.py
import os
from langchain_core.messages import SystemMessage
from langchain_openai import ChatOpenAI
from .state import AgentState
from .tools import TOOLS

SYSTEM_PROMPT = """You are a task management assistant. Help users create, update, and track their tasks.

When a user asks you to do something with tasks, use your tools. After tool calls, summarize
the result clearly. Always be concise and action-oriented."""

llm = ChatOpenAI(
 model="gpt-4o-mini",
 temperature=0.1,
 api_key=os.environ.get("OPENAI_API_KEY"),
)

llm_with_tools = llm.bind_tools(TOOLS)


def agent_node(state: AgentState) -> dict:
 """Call the LLM with the full conversation history."""
 messages = [SystemMessage(content=SYSTEM_PROMPT)] + state["messages"]
 response = llm_with_tools.invoke(messages)
 return {"messages": [response]}


def should_continue(state: AgentState) -> str:
 """Route: call tools if the LLM made a tool call, otherwise finish."""
 last_message = state["messages"][-1]
 if hasattr(last_message, "tool_calls") and last_message.tool_calls:
 return "tools"
 return "end"

Step 4 — Assemble the Graph with a Checkpointer

The checkpointer is what makes the agent remember conversations across multiple calls. It persists the full AgentState after every step — keyed by a thread_id.

# agent/graph.py
from langgraph.graph import END, StateGraph
from langgraph.checkpoint.sqlite import SqliteSaver
from langgraph.prebuilt import ToolNode
from .state import AgentState
from .nodes import agent_node, should_continue
from .tools import TOOLS


def build_graph(db_path: str = "agent_memory.db"):
 """Build and compile the agent graph with SQLite persistence."""

 # SQLite checkpointer — persists state after every node execution
 checkpointer = SqliteSaver.from_conn_string(db_path)

 graph = StateGraph(AgentState)

 # Nodes
 graph.add_node("agent", agent_node)
 graph.add_node("tools", ToolNode(TOOLS))

 # Edges
 graph.set_entry_point("agent")
 graph.add_conditional_edges(
 "agent",
 should_continue,
 {"tools": "tools", "end": END},
 )
 graph.add_edge("tools", "agent")

 return graph.compile(checkpointer=checkpointer)

The graph looks like this:

graph LR START([START]) --> agent["agent_node\n(LLM call)"] agent --> cond{{"should_continue"}} cond -->|"tool_calls present"| tools["ToolNode\n(execute tools)"] cond -->|"no tool_calls"| END([END]) tools --> agent style START fill:#4CAF50,color:#fff,stroke:none style END fill:#F44336,color:#fff,stroke:none style agent fill:#9C27B0,color:#fff,stroke:none style tools fill:#FF9800,color:#fff,stroke:none style cond fill:#2196F3,color:#fff,stroke:none

Fig 1: Agent graph with SQLite-persisted state

Step 5 — Run with Streaming

# agent/main.py
import os
from dotenv import load_dotenv
from langchain_core.messages import HumanMessage
from .graph import build_graph

load_dotenv()


def chat(app, thread_id: str, user_input: str) -> None:
 """Send a message and stream the agent's response token by token."""
 config = {"configurable": {"thread_id": thread_id}}
 inputs = {"messages": [HumanMessage(content=user_input)]}

 print(f"\nUser: {user_input}")
 print("Assistant: ", end="", flush=True)

 # stream_mode="messages" yields (message_chunk, metadata) tuples
 for chunk, metadata in app.stream(inputs, config=config, stream_mode="messages"):
 # Only print text from the AI (not tool results or empty chunks)
 if (
 chunk.content # has content
 and metadata.get("langgraph_node") == "agent" # from the agent node
 and not getattr(chunk, "tool_calls", None) # not a tool-call message
 ):
 print(chunk.content, end="", flush=True)

 print() # newline after streaming completes


def main():
 app = build_graph()

 # Each thread_id is a separate conversation with its own persisted history
 thread_a = "user-alice-session-1"
 thread_b = "user-bob-session-1"

 # --- Thread A: Alice's conversation ---
 chat(app, thread_a, "Create a high-priority task: Set up CI/CD pipeline")
 chat(app, thread_a, "Also create a medium task: Write unit tests for auth module")
 chat(app, thread_a, "Show me all my tasks")

 # --- Thread B: Bob's separate conversation (no access to Alice's tasks) ---
 chat(app, thread_b, "Create a task: Review Q1 budget report")
 chat(app, thread_b, "Show me all tasks")

 # --- Thread A resumes (state is restored from SQLite) ---
 chat(app, thread_a, "Mark the CI/CD task as in_progress")
 chat(app, thread_a, "Show me only in_progress tasks")


if __name__ == "__main__":
 main()

Expected output (abbreviated):

User: Create a high-priority task: Set up CI/CD pipeline
Assistant: I've created the task for you! Created task task-001: 'Set up CI/CD
pipeline' (priority: high). What else would you like to do?

User: Show me all my tasks
Assistant: Here are your current tasks:
- [task-001] Set up CI/CD pipeline | HIGH | todo
- [task-002] Write unit tests for auth module | MEDIUM | todo

User: Mark the CI/CD task as in_progress
Assistant: Done! Updated task-001 status to 'in_progress'.

Understanding Multi-Thread Isolation

The thread_id in the config is the key to how LangGraph separates conversations:

# Thread isolation explained
config_alice = {"configurable": {"thread_id": "user-alice-session-1"}}
config_bob = {"configurable": {"thread_id": "user-bob-session-1"}}

# Each thread has completely separate state in SQLite:
# - Different message histories
# - The tools share the global `_tasks` dict here (because it's in-memory),
# but in production each user would have their own DB-scoped tasks

In a real system, you’d scope tool data to the user via thread_id or a user_id passed through config["configurable"] (covered in Ch 5).

Inspecting Persisted State

You can inspect the saved state for any thread at any time — useful for debugging:

# debug_state.py
from agent.graph import build_graph

app = build_graph()

# Get the latest state for a thread
config = {"configurable": {"thread_id": "user-alice-session-1"}}
state = app.get_state(config)

print(f"Message count: {len(state.values['messages'])}")
for msg in state.values["messages"]:
 role = msg.__class__.__name__
 preview = str(msg.content)[:80] if msg.content else "[tool_calls]"
 print(f" [{role}] {preview}")

Summary

Feature	Implementation
State	`TypedDict` with `Annotated[list, add_messages]`
Tools	`@tool` with Pydantic `args_schema`, errors returned as strings
Graph	`StateGraph` → agent → conditional → tools → agent
Persistence	`SqliteSaver` checkpointer keyed by `thread_id`
Streaming	`app.stream(..., stream_mode="messages")` with node filter
Multi-user	Separate `thread_id` per user/session gives isolated state

In the next chapter we tackle the hard parts: injecting user context safely into tools, adding content moderation with NeMo Guardrails, and implementing human-in-the-loop interrupts.

[Ch 5] Tools Integration, Guardrails & Safety Patterns

Tue, 07 Apr 2026 00:00:00 +0000

In Ch 4 we built a working multi-turn agent. But it has a critical gap: the tools have no idea who is calling them. In production, you need tools to know the current user, apply safety filters to inputs and outputs, and pause for human approval before irreversible actions. This chapter implements all three.

The Three Safety Layers

graph TD U([User Input]) --> G1["🛡️ Input Guardrail\n(NeMo)"] G1 -->|"safe"| A["🧠 Agent Loop"] G1 -->|"blocked"| R1([Safe Refusal]) A --> T["🔧 Tool Execution\n(with AgentContext)"] T --> H{{"⚠️ High-stakes\naction?"}} H -->|"yes"| I["⏸️ Human-in-the-Loop\ninterrupt()"] I -->|"approved"| T2["✅ Execute Action"] I -->|"rejected"| R2([Cancel Action]) H -->|"no"| T2 T2 --> G2["🛡️ Output Guardrail\n(NeMo)"] G2 --> F([Response to User]) style U fill:#4CAF50,color:#fff,stroke:none style F fill:#2196F3,color:#fff,stroke:none style G1 fill:#E53935,color:#fff,stroke:none style G2 fill:#E53935,color:#fff,stroke:none style I fill:#FF9800,color:#fff,stroke:none style A fill:#9C27B0,color:#fff,stroke:none style T fill:#0288D1,color:#fff,stroke:none

Fig 1: Three-layer safety architecture for production agents

Part 1: AgentContext — Safe User Context Injection

The Problem

You need your tools to know user_id, session_id, and any per-request metadata. The naive approach is to embed them in the system prompt or the user message:

# ❌ Bad: pollutes the conversation history
user_message = f"[USER_ID: {user_id}] {actual_query}"

This is bad for three reasons:

It wastes precious context window tokens on every turn
The LLM may paraphrase or drop the metadata in its reasoning
It leaks internal identifiers into the conversation transcript

The Solution: `config["configurable"]`

LangGraph passes a RunnableConfig through every node and tool call. Use it to carry context that’s external to the conversation:

# agent/context.py
from pydantic import BaseModel

class AgentContext(BaseModel):
 """Typed context passed through every agent call without touching messages."""
 user_id: str
 session_id: str
 permissions: list[str] = [] # e.g. ["read", "write", "delete"]
 org_id: str = ""

# agent/tools.py — accessing context inside a tool
from langchain_core.runnables import RunnableConfig
from langchain_core.tools import tool
from .context import AgentContext

@tool
def delete_task(task_id: str, config: RunnableConfig) -> str:
 """Permanently delete a task. Requires 'delete' permission."""
 # LangGraph automatically injects `config` — just add it as a parameter
 ctx_data = config.get("configurable", {}).get("agent_context")
 if ctx_data is None:
 return "Error: No agent context provided."

 ctx = AgentContext(**ctx_data) if isinstance(ctx_data, dict) else ctx_data

 # Permission check using context — no need to pass user_id as a tool arg
 if "delete" not in ctx.permissions:
 return f"Error: User {ctx.user_id} does not have delete permission."

 if task_id not in _tasks:
 return f"Error: Task '{task_id}' not found."

 task_title = _tasks.pop(task_id)["title"]
 return f"Deleted task '{task_title}' (ID: {task_id})."

# agent/main.py — how to pass context at call time
from .context import AgentContext

ctx = AgentContext(
 user_id="user-42",
 session_id="sess-abc",
 permissions=["read", "write"], # no "delete" for this user
 org_id="acme-corp",
)

config = {
 "configurable": {
 "thread_id": "user-42-session-1",
 "agent_context": ctx.model_dump(), # serialize for LangGraph
 }
}

app.invoke({"messages": [HumanMessage(content="Delete task task-001")]}, config=config)
# → Tool returns: "Error: User user-42 does not have delete permission."

The key insight: config["configurable"] is available in every tool function — just add a config: RunnableConfig parameter, and LangGraph injects it automatically. No message pollution, no token waste.

Part 2: NeMo Guardrails

NeMo Guardrails is NVIDIA’s open-source library for adding programmable safety rails to LLM applications. It works as a layer around your agent: inputs go through rails before reaching the LLM, and outputs go through rails before being returned.

Installation

pip install nemoguardrails

Guardrail Configuration

NeMo uses Colang (a domain-specific language) for defining rails. Create a config directory:

guardrails/
├── config.yml
└── main.co # Colang flow definitions

# guardrails/config.yml
models:
 - type: main
 engine: openai
 model: gpt-4o-mini

rails:
 input:
 flows:
 - check harmful input
 output:
 flows:
 - check harmful output

# guardrails/main.co

# ── Input rail: block harmful requests ───────────────────────────────────────
define flow check harmful input
 $is_harmful = execute check_input_for_harm

 if $is_harmful
 bot refuse harmful input
 stop

define bot refuse harmful input
 "I'm sorry, I can't help with that request."

# ── Output rail: block harmful responses ─────────────────────────────────────
define flow check harmful output
 $is_harmful = execute check_output_for_harm

 if $is_harmful
 bot provide safe response
 stop

define bot provide safe response
 "I'm not able to provide that information."

Integrating Guardrails with Your Agent

The cleanest pattern is to wrap your agent call — not embed NeMo inside the LangGraph graph itself. This keeps the graph simple and makes the rails easy to swap:

# agent/guardrail.py
import os
from nemoguardrails import RailsConfig, LLMRails

def load_rails(config_path: str = "guardrails/") -> LLMRails:
 """Load NeMo guardrails from config directory."""
 config = RailsConfig.from_path(config_path)
 return LLMRails(config)

rails = load_rails()

# agent/main.py — wrapped agent call with guardrails
async def safe_chat(app, thread_id: str, user_input: str) -> str:
 """Run user input through guardrails before and after the agent."""

 # ── 1. Input guardrail ────────────────────────────────────────────────────
 input_check = await rails.generate_async(
 messages=[{"role": "user", "content": user_input}]
 )
 # NeMo returns the guardrail response; if it's a refusal, return immediately
 if _is_refusal(input_check):
 return input_check["content"]

 # ── 2. Run the agent ──────────────────────────────────────────────────────
 config = {"configurable": {"thread_id": thread_id}}
 final_state = app.invoke(
 {"messages": [HumanMessage(content=user_input)]},
 config=config,
 )
 agent_response = final_state["messages"][-1].content

 # ── 3. Output guardrail ───────────────────────────────────────────────────
 output_check = await rails.generate_async(
 messages=[
 {"role": "user", "content": user_input},
 {"role": "assistant", "content": agent_response},
 ]
 )
 if _is_refusal(output_check):
 return "I can't provide that response."

 return agent_response


def _is_refusal(response: dict) -> bool:
 """Detect if NeMo triggered a refusal rail."""
 content = response.get("content", "")
 refusal_phrases = ["I'm sorry, I can't", "I'm not able to"]
 return any(phrase in content for phrase in refusal_phrases)

⚠️ Guardrails add latency. Each rail check is an additional LLM call (~500ms–1s). Profile your p95 latency before adding rails to every endpoint. For low-risk internal tools, input rails alone may be sufficient.

Part 3: Human-in-the-Loop (HITL)

Some actions are irreversible: deleting records, sending emails, making payments. For these, you want the agent to pause and ask the human before proceeding. LangGraph has first-class support for this via interrupt().

How `interrupt()` Works

sequenceDiagram participant U as User participant A as Agent Loop participant T as Tool Node U->>A: "Delete all completed tasks" A->>T: tool_call: delete_completed_tasks T->>T: interrupt("Are you sure?") Note over T: Graph pauses here.
State is saved to checkpointer. T-->>U: "⚠️ About to delete 5 tasks. Confirm? (yes/no)" U->>A: resume with {"approved": true} A->>T: Resume execution T->>T: Execute deletion T-->>U: "Deleted 5 completed tasks."

Fig 2: HITL flow — agent pauses mid-tool for human approval

Implementation

# agent/tools.py — a tool that interrupts for confirmation
from langgraph.types import interrupt, Command

class DeleteCompletedInput(BaseModel):
 confirm_message: str = Field(
 description="Message to show the user when asking for confirmation"
 )

@tool("delete_completed_tasks", args_schema=DeleteCompletedInput)
def delete_completed_tasks(confirm_message: str) -> str:
 """Delete all tasks marked as 'done'. Requires human confirmation."""
 done_tasks = [t for t in _tasks.values() if t["status"] == "done"]

 if not done_tasks:
 return "No completed tasks to delete."

 # ── INTERRUPT: pause and ask the human ───────────────────────────────────
 approval = interrupt({
 "question": f"About to permanently delete {len(done_tasks)} completed task(s). Continue?",
 "tasks_to_delete": [t["title"] for t in done_tasks],
 "action": "delete_completed_tasks",
 })

 # Execution resumes here after the human responds
 if not approval.get("approved", False):
 return "Deletion cancelled by user."

 # Execute the deletion
 for task in done_tasks:
 del _tasks[task["id"]]

 return f"Successfully deleted {len(done_tasks)} completed task(s)."

# agent/main.py — handling the interrupt in your API/CLI layer
from langgraph.types import Command

def chat_with_hitl(app, thread_id: str, user_input: str):
 config = {"configurable": {"thread_id": thread_id}}

 # First invoke — may pause at an interrupt
 for event in app.stream(
 {"messages": [HumanMessage(content=user_input)]},
 config=config,
 stream_mode="values",
 ):
 if "__interrupt__" in event:
 interrupt_data = event["__interrupt__"][0].value
 print(f"\n⚠️ CONFIRMATION REQUIRED")
 print(f" {interrupt_data['question']}")
 print(f" Tasks: {', '.join(interrupt_data['tasks_to_delete'])}")

 answer = input(" Approve? (yes/no): ").strip().lower()
 approved = answer in ("yes", "y")

 # Resume the graph with the human's decision
 for resume_event in app.stream(
 Command(resume={"approved": approved}),
 config=config,
 stream_mode="values",
 ):
 last = resume_event.get("messages", [])
 if last:
 print(f"\nAssistant: {last[-1].content}")
 return

 # No interrupt — normal completion
 state = app.get_state(config)
 if state.values["messages"]:
 print(f"\nAssistant: {state.values['messages'][-1].content}")

What happens under the hood:

interrupt() raises a special exception that LangGraph catches
The graph state (including the pending tool call) is saved to the checkpointer
The graph returns control to your code with an __interrupt__ event
When you call app.stream(Command(resume=...), config=config), LangGraph restores the state and continues from exactly where it paused

Tool Design Checklist

Before shipping any tool to production:

Check	Why
✅ Pydantic `args_schema` with `Field(description=...)`	The LLM reads descriptions to know how to call the tool
✅ Returns `str` error messages (not exceptions)	Prevents tool errors from crashing the agent loop
✅ Validates inputs explicitly	Don’t trust the LLM to pass well-formed data
✅ Uses `config: RunnableConfig` for user context	No user IDs or secrets in the message stream
✅ `interrupt()` for irreversible actions	Delete, send, pay, deploy — always ask first
✅ Logs tool calls with input/output (see Ch 7)	Essential for debugging production issues

.env.example

# .env.example
OPENAI_API_KEY=your-api-key-here

Summary

Pattern	Problem Solved
`AgentContext` via `config["configurable"]`	Pass user context to tools without message pollution
NeMo Guardrails (input rail)	Block harmful or off-topic requests before the LLM sees them
NeMo Guardrails (output rail)	Prevent the agent from returning harmful content
`interrupt()` + `Command(resume=...)`	Pause for human approval before irreversible actions
Tool error strings (not exceptions)	Tools fail gracefully without crashing the agent loop

In the next chapter, we add proper memory management — giving the agent persistent long-term knowledge via vector stores and controlling the growth of conversation history.

[Ch 6] Memory Management for Production Agents

Tue, 07 Apr 2026 00:00:00 +0000

The SQLite checkpointer from Ch 4 persists conversation history — but that’s just one type of memory. Production agents need four distinct layers, each serving a different purpose and operating on a different time horizon.

The Four Memory Layers

graph TD subgraph "Short-term (per session)" A["🧠 In-Context Memory\nCurrent message list in context window\nLost when window fills or session ends"] B["📼 Episodic Memory\nFull conversation history in DB\nPersisted across sessions per thread_id"] end subgraph "Long-term (across sessions)" C["📚 Semantic Memory\nVector store of facts & documents\nRetrieved on-demand via RAG"] D["📋 Procedural Memory\nSystem prompt — rules & persona\nBaked into every single LLM call"] end style A fill:#7C3AED,color:#fff,stroke:none style B fill:#059669,color:#fff,stroke:none style C fill:#D97706,color:#fff,stroke:none style D fill:#2563EB,color:#fff,stroke:none

Fig 1: Four memory layers with different scope and persistence

Layer	Storage	Scope	Limits
In-Context	Message list in RAM	Current LLM call	Context window size
Episodic	Database (SQLite / MongoDB)	Per `thread_id`	Disk space
Semantic	Vector store (FAISS / Qdrant)	All users / sessions	Embedding index size
Procedural	Source code (system prompt)	Every call	Prompt token budget

Layer 1: In-Context Memory — Managing History Growth

As conversations grow, the message list eventually overflows the context window or becomes expensive. Two standard strategies:

Strategy A: Sliding Window (Trim)

Keep only the last N messages, discarding older ones:

# agent/memory/trim.py
from langchain_core.messages import AnyMessage, SystemMessage, trim_messages
from langchain_openai import ChatOpenAI


def trim_conversation(
 messages: list[AnyMessage],
 max_tokens: int = 4000,
 model: str = "gpt-4o-mini",
) -> list[AnyMessage]:
 """Trim messages to stay within a token budget, always keeping the system message."""
 return trim_messages(
 messages,
 max_tokens=max_tokens,
 strategy="last", # keep the most recent messages
 token_counter=ChatOpenAI(model=model),
 include_system=True, # never trim the system message
 allow_partial=False, # don't cut messages mid-way
 )

# agent/nodes.py — use trim inside the agent node
def agent_node(state: AgentState) -> dict:
 trimmed = trim_conversation(state["messages"], max_tokens=4000)
 messages = [SystemMessage(content=SYSTEM_PROMPT)] + trimmed
 response = llm_with_tools.invoke(messages)
 return {"messages": [response]}

Strategy B: Summarization

When trimming loses important early context (e.g., user goals set at the start), summarize instead:

# agent/memory/summarize.py
from langchain_core.messages import AnyMessage, HumanMessage, AIMessage, SystemMessage
from langchain_openai import ChatOpenAI

summarizer = ChatOpenAI(model="gpt-4o-mini", temperature=0)

SUMMARY_PROMPT = """Summarize the following conversation in 2-3 sentences.
Focus on: decisions made, key facts established, and current task status.
Be concise — this summary will replace the conversation history."""


def summarize_messages(messages: list[AnyMessage]) -> str:
 """Produce a compact summary of a list of messages."""
 conversation_text = "\n".join(
 f"{msg.__class__.__name__}: {msg.content}"
 for msg in messages
 if msg.content # skip tool-call-only messages
 )
 response = summarizer.invoke([
 SystemMessage(content=SUMMARY_PROMPT),
 HumanMessage(content=conversation_text),
 ])
 return response.content


def maybe_summarize(
 messages: list[AnyMessage],
 threshold: int = 20,
) -> list[AnyMessage]:
 """If history exceeds threshold messages, summarize the older half."""
 if len(messages) <= threshold:
 return messages

 midpoint = len(messages) // 2
 old_messages = messages[:midpoint]
 recent_messages = messages[midpoint:]

 summary = summarize_messages(old_messages)

 # Replace old messages with a single summary message
 summary_msg = AIMessage(
 content=f"[Conversation summary — earlier context]\n{summary}"
 )
 return [summary_msg] + recent_messages

Layer 2: Episodic Memory — MongoDB Checkpointer

SQLite is great for development but not for production (no concurrent writes, no horizontal scaling). For production, use the MongoDB checkpointer:

pip install langchain-mongodb motor

# agent/graph_mongo.py
import os
from motor.motor_asyncio import AsyncIOMotorClient
from langgraph.checkpoint.mongodb.aio import AsyncMongoDBSaver
from langgraph.graph import END, StateGraph
from langgraph.prebuilt import ToolNode
from .state import AgentState
from .nodes import agent_node, should_continue
from .tools import TOOLS

MONGO_URI = os.environ.get("MONGODB_URI", "mongodb://localhost:27017")
MONGO_DB = os.environ.get("MONGODB_DB", "agent_memory")


async def build_async_graph():
 """Build the agent graph with async MongoDB checkpointer."""
 client = AsyncIOMotorClient(MONGO_URI)
 checkpointer = AsyncMongoDBSaver(client, db_name=MONGO_DB)

 graph = StateGraph(AgentState)
 graph.add_node("agent", agent_node)
 graph.add_node("tools", ToolNode(TOOLS))
 graph.set_entry_point("agent")
 graph.add_conditional_edges("agent", should_continue, {"tools": "tools", "end": END})
 graph.add_edge("tools", "agent")

 return graph.compile(checkpointer=checkpointer)

# agent/main_async.py
import asyncio
from langchain_core.messages import HumanMessage
from .graph_mongo import build_async_graph


async def main():
 app = await build_async_graph()

 config = {"configurable": {"thread_id": "user-42-session-1"}}
 inputs = {"messages": [HumanMessage(content="List all my tasks")]}

 async for chunk, metadata in app.astream(inputs, config=config, stream_mode="messages"):
 if chunk.content and metadata.get("langgraph_node") == "agent":
 print(chunk.content, end="", flush=True)
 print()


if __name__ == "__main__":
 asyncio.run(main())

# .env.example (updated)
OPENAI_API_KEY=your-api-key-here
MONGODB_URI=mongodb://localhost:27017
MONGODB_DB=agent_memory

MongoDB vs SQLite at a glance:

	SQLite	MongoDB
Setup	Zero-config	Requires running server
Concurrent writes	❌ Single writer	✅ Multi-writer
Horizontal scaling	❌ File-based	✅ Replica sets / Atlas
Production-ready	Dev only	✅ Yes
LangGraph support	`SqliteSaver`	`AsyncMongoDBSaver`

Layer 3: Semantic Memory — RAG with FAISS

Episodic memory stores what happened. Semantic memory stores what the agent knows — a searchable knowledge base. The agent retrieves relevant facts at query time rather than scanning the entire history.

pip install langchain-community faiss-cpu tiktoken

# agent/memory/knowledge_base.py
import os
from langchain_community.vectorstores import FAISS
from langchain_openai import OpenAIEmbeddings
from langchain_core.documents import Document

EMBED_MODEL = OpenAIEmbeddings(
 model="text-embedding-3-small",
 api_key=os.environ.get("OPENAI_API_KEY"),
)

# ── Build the index ───────────────────────────────────────────────────────────
def build_knowledge_base(documents: list[dict]) -> FAISS:
 """
 Build a FAISS vector store from a list of {content, metadata} dicts.
 In production, load these from a database or file store.
 """
 docs = [
 Document(page_content=d["content"], metadata=d.get("metadata", {}))
 for d in documents
 ]
 return FAISS.from_documents(docs, EMBED_MODEL)


# ── Example knowledge base ────────────────────────────────────────────────────
COMPANY_DOCS = [
 {
 "content": "The annual leave policy allows 20 days per year. Requests must be submitted 2 weeks in advance.",
 "metadata": {"source": "hr-policy", "section": "leave"},
 },
 {
 "content": "Expense reports must be filed within 30 days of the expense date. Receipts are required for amounts over $25.",
 "metadata": {"source": "finance-policy", "section": "expenses"},
 },
 {
 "content": "The on-call rotation schedule is updated every Monday. Check the team wiki for the current schedule.",
 "metadata": {"source": "engineering-policy", "section": "on-call"},
 },
 {
 "content": "All production deployments require approval from the tech lead and a passing CI pipeline.",
 "metadata": {"source": "engineering-policy", "section": "deployments"},
 },
]

knowledge_base = build_knowledge_base(COMPANY_DOCS)


# ── Save and load ─────────────────────────────────────────────────────────────
def save_index(index: FAISS, path: str = "faiss_index") -> None:
 index.save_local(path)

def load_index(path: str = "faiss_index") -> FAISS:
 return FAISS.load_local(path, EMBED_MODEL, allow_dangerous_deserialization=True)

# agent/tools.py — RAG search as a tool
from pydantic import BaseModel, Field
from langchain_core.tools import tool
from .memory.knowledge_base import knowledge_base

class SearchKnowledgeInput(BaseModel):
 query: str = Field(description="The question or topic to search for in the knowledge base")
 k: int = Field(default=3, description="Number of relevant documents to retrieve (1-5)")

@tool("search_knowledge_base", args_schema=SearchKnowledgeInput)
def search_knowledge_base(query: str, k: int = 3) -> str:
 """Search the company knowledge base for policies, procedures, and guidelines."""
 results = knowledge_base.similarity_search_with_score(query, k=k)

 if not results:
 return "No relevant information found in the knowledge base."

 output_lines = [f"Found {len(results)} relevant document(s):\n"]
 for i, (doc, score) in enumerate(results, 1):
 # Only return results above a relevance threshold
 if score > 0.4: # FAISS L2 distance — lower = more similar
 continue
 source = doc.metadata.get("source", "unknown")
 output_lines.append(f"[{i}] (source: {source})\n{doc.page_content}")

 return "\n\n".join(output_lines) if len(output_lines) > 1 else "No sufficiently relevant results found."

💡 FAISS vs Qdrant: FAISS is in-memory and zero-infrastructure — perfect for tutorials and small knowledge bases (< 100K documents). For production with millions of documents, filtered search per user, or real-time updates, use or instead.

Layer 4: Procedural Memory — The System Prompt

The system prompt is the most overlooked memory layer. It defines how the agent behaves on every single call — its persona, constraints, output format, and domain-specific rules. Things that never change belong here.

# agent/nodes.py — dynamic system prompt
from datetime import datetime, timezone

def build_system_prompt(ctx: "AgentContext | None" = None) -> str:
 """Build the system prompt, optionally personalized with user context."""
 base = """You are a task management assistant for engineering teams.

CAPABILITIES:
- Create, update, list, and delete tasks
- Search the company knowledge base for policies
- Request human confirmation before destructive actions

OUTPUT FORMAT:
- Use concise, action-oriented language
- After tool calls, summarize results in 1-2 sentences
- Format task lists as bullet points with status indicators

CONSTRAINTS:
- Only manage tasks within the user's permission scope
- Never expose other users' task data
- Always cite the source when answering policy questions"""

 # Add dynamic context
 now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
 base += f"\n\nCurrent date/time: {now}"

 if ctx:
 base += f"\nCurrent user: {ctx.user_id} (org: {ctx.org_id})"
 base += f"\nPermissions: {', '.join(ctx.permissions)}"

 return base

The procedural memory principle: if something is true for every call, put it in the system prompt. If it’s true for this call only, put it in the user message or retrieved context.

Putting It All Together

graph LR U([User Query]) --> R["📚 RAG Retrieval\n(Semantic Memory)"] R --> CTX["Build Context Window:\n• System Prompt (Procedural)\n• Trimmed History (In-context)\n• Retrieved Docs (Semantic)\n• User Query"] CTX --> LLM["🧠 LLM"] LLM --> A["Agent Response"] A --> DB["💾 Save to MongoDB\n(Episodic Memory)"] style U fill:#4CAF50,color:#fff,stroke:none style A fill:#2196F3,color:#fff,stroke:none style LLM fill:#9C27B0,color:#fff,stroke:none style DB fill:#059669,color:#fff,stroke:none

Fig 2: All four memory layers working together in one agent call

# agent/nodes.py — full agent node with all memory layers integrated
from .memory.trim import trim_conversation
from .memory.knowledge_base import search_knowledge_base

def agent_node(state: AgentState, config: RunnableConfig) -> dict:
 # Layer 4: Procedural memory
 ctx_data = config.get("configurable", {}).get("agent_context")
 ctx = AgentContext(**ctx_data) if ctx_data else None
 system_msg = SystemMessage(content=build_system_prompt(ctx))

 # Layer 1: In-context memory with trimming
 trimmed_messages = trim_conversation(state["messages"], max_tokens=4000)

 messages = [system_msg] + trimmed_messages
 response = llm_with_tools.invoke(messages)
 return {"messages": [response]}

.env.example

# .env.example
OPENAI_API_KEY=your-api-key-here
MONGODB_URI=mongodb://localhost:27017
MONGODB_DB=agent_memory

💡 Ollama note: Replace OpenAIEmbeddings with:
from langchain_ollama import OllamaEmbeddings
EMBED_MODEL = OllamaEmbeddings(model="nomic-embed-text")
Pull the embedding model first: ollama pull nomic-embed-text

Summary

Memory Layer	Implementation	When to Use
In-Context	`trim_messages()` / summarization	Always — control token growth
Episodic	`SqliteSaver` (dev) / `AsyncMongoDBSaver` (prod)	Multi-turn conversations
Semantic	FAISS (dev) / Qdrant (prod) + `@tool`	Knowledge base Q&A, document retrieval
Procedural	System prompt (static + dynamic parts)	Persona, constraints, output format

In the next chapter, we add observability — making every LLM call, tool execution, and token cost visible via Langfuse.

[Ch 7] Observability with Langfuse — Tracing Every Agent Step

Tue, 07 Apr 2026 00:00:00 +0000

When something goes wrong in production — and it will — you need to answer: What did the LLM receive? Which tool did it call? What did the tool return? How long did each step take? How much did this session cost?

Without observability, debugging a multi-step agent is guesswork. This chapter adds to give you full visibility into every agent execution.

What Langfuse Provides

graph TD A["🔁 Agent Run"] --> T["Trace\n(one per user session)"] T --> S1["Span: agent_node\n(LLM generation)"] T --> S2["Span: tool — search_documents\n(function execution)"] T --> S3["Span: agent_node\n(LLM generation — 2nd call)"] S1 --> G1["Generation\n• model, tokens, cost\n• input messages\n• output message"] S3 --> G2["Generation\n• model, tokens, cost\n• input messages\n• output message"] style A fill:#4CAF50,color:#fff,stroke:none style T fill:#7C3AED,color:#fff,stroke:none style S1 fill:#2563EB,color:#fff,stroke:none style S2 fill:#FF9800,color:#fff,stroke:none style S3 fill:#2563EB,color:#fff,stroke:none style G1 fill:#059669,color:#fff,stroke:none style G2 fill:#059669,color:#fff,stroke:none

Fig 1: Langfuse trace hierarchy — one trace per agent run, spans per step, generations for LLM calls

Concept	Description
Trace	One complete agent run (from user input to final response)
Span	A single step within the trace (tool call, node execution)
Generation	An LLM API call with model, token counts, cost, and messages
Score	A metric attached post-hoc (e.g., user thumbs up, eval score)

Installation & Setup

pip install langfuse

# .env.example
OPENAI_API_KEY=your-api-key-here
LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
LANGFUSE_HOST=https://cloud.langfuse.com # or your self-hosted URL

Part 1: trace_agent_execution — Wrapping the Full Run

The cleanest integration pattern is a context manager that wraps the entire agent invocation. One context manager = one Langfuse trace.

# agent/tracing/langfuse_client.py
import os
from langfuse import Langfuse

def get_langfuse_client() -> Langfuse:
 """Create a Langfuse client from environment variables."""
 return Langfuse(
 public_key=os.environ.get("LANGFUSE_PUBLIC_KEY"),
 secret_key=os.environ.get("LANGFUSE_SECRET_KEY"),
 host=os.environ.get("LANGFUSE_HOST", "https://cloud.langfuse.com"),
 )

langfuse = get_langfuse_client()

# agent/tracing/trace_agent.py
import time
from contextlib import asynccontextmanager
from langfuse import Langfuse
from langfuse.client import StatefulSpanClient
from .langfuse_client import langfuse
from .sanitize import sanitize_payload


@asynccontextmanager
async def trace_agent_execution(
 session_id: str,
 user_id: str,
 user_input: str,
 metadata: dict | None = None,
):
 """
 Context manager that wraps a full agent run in a single Langfuse trace.

 Usage:
 async with trace_agent_execution(session_id, user_id, input) as trace:
 result = await run_agent(input)
 trace.update(output=result)
 """
 trace = langfuse.trace(
 name="agent-run",
 session_id=session_id,
 user_id=user_id,
 input=sanitize_payload({"user_input": user_input}),
 metadata=metadata or {},
 )
 start_time = time.perf_counter()
 try:
 yield trace
 except Exception as exc:
 trace.update(
 output={"error": str(exc)},
 level="ERROR",
 status_message=str(exc),
 )
 raise
 finally:
 elapsed_ms = int((time.perf_counter() - start_time) * 1000)
 trace.update(metadata={"duration_ms": elapsed_ms, **(metadata or {})})
 langfuse.flush() # ensure all events are sent before the function returns

# agent/main.py — using the context manager
from .tracing.trace_agent import trace_agent_execution

async def run_agent_with_tracing(
 app,
 thread_id: str,
 user_id: str,
 user_input: str,
) -> str:
 async with trace_agent_execution(
 session_id=thread_id,
 user_id=user_id,
 user_input=user_input,
 ) as trace:
 config = {"configurable": {"thread_id": thread_id}}
 final_state = await app.ainvoke(
 {"messages": [HumanMessage(content=user_input)]},
 config=config,
 )
 response = final_state["messages"][-1].content
 trace.update(output={"response": sanitize_payload(response)})
 return response

Part 2: @trace_tool — Per-Tool Span Tracing

Each tool call should appear as a separate span inside the trace. A decorator handles this cleanly:

# agent/tracing/trace_tool.py
import time
import functools
from typing import Callable, Any
from langfuse.client import StatefulTraceClient
from .langfuse_client import langfuse
from .sanitize import sanitize_payload


def trace_tool(
 tool_name: str | None = None,
 tags: list[str] | None = None,
) -> Callable:
 """
 Decorator that traces a tool function call as a Langfuse span.

 Usage:
 @trace_tool(tags=["retrieval"])
 def search_documents(query: str) -> str: ...
 """
 def decorator(func: Callable) -> Callable:
 name = tool_name or func.__name__

 @functools.wraps(func)
 def wrapper(*args, **kwargs) -> Any:
 # Get the current active trace from LangGraph's RunnableConfig
 # (passed through kwargs as 'config' if present)
 config = kwargs.get("config") or {}
 trace_id = config.get("configurable", {}).get("langfuse_trace_id")

 # Create a span under the current trace (or standalone if no trace)
 span = langfuse.span(
 trace_id=trace_id,
 name=f"tool.{name}",
 input=sanitize_payload({"args": args, "kwargs": {
 k: v for k, v in kwargs.items() if k != "config"
 }}),
 tags=tags or [],
 )

 start = time.perf_counter()
 try:
 result = func(*args, **kwargs)
 elapsed_ms = int((time.perf_counter() - start) * 1000)
 span.end(
 output=sanitize_payload({"result": result}),
 metadata={"duration_ms": elapsed_ms},
 )
 return result
 except Exception as exc:
 span.end(
 output={"error": str(exc)},
 level="ERROR",
 status_message=str(exc),
 )
 raise

 return wrapper
 return decorator

Apply it to your tools:

# agent/tools.py
from .tracing.trace_tool import trace_tool

@trace_tool(tags=["retrieval"])
@tool("search_knowledge_base", args_schema=SearchKnowledgeInput)
def search_knowledge_base(query: str, k: int = 3) -> str:
 """Search the company knowledge base..."""
 ...

@trace_tool(tags=["write"])
@tool("create_task", args_schema=CreateTaskInput)
def create_task(title: str, description: str, priority: str = "medium") -> str:
 """Create a new task..."""
 ...

Part 3: Payload Sanitization

Never log raw payloads blindly. Production tool calls may contain:

User PII (names, emails, phone numbers)
Internal IDs or file paths
Very large strings that exceed Langfuse’s payload limits

A recursive sanitize function handles both:

# agent/tracing/sanitize.py
from typing import Any

MAX_STRING_LENGTH = 2000 # truncate strings longer than this
MAX_LIST_ITEMS = 20 # truncate lists longer than this


def sanitize_payload(value: Any, _depth: int = 0) -> Any:
 """
 Recursively sanitize a value before sending to Langfuse:
 - Truncate long strings
 - Truncate long lists
 - Mask known sensitive key names
 - Limit recursion depth
 """
 if _depth > 5:
 return "[max depth exceeded]"

 if isinstance(value, str):
 if len(value) > MAX_STRING_LENGTH:
 return value[:MAX_STRING_LENGTH] + f"... [truncated, {len(value)} chars total]"
 return value

 if isinstance(value, dict):
 return {
 k: "[REDACTED]" if _is_sensitive_key(k) else sanitize_payload(v, _depth + 1)
 for k, v in value.items()
 }

 if isinstance(value, list):
 truncated = value[:MAX_LIST_ITEMS]
 result = [sanitize_payload(item, _depth + 1) for item in truncated]
 if len(value) > MAX_LIST_ITEMS:
 result.append(f"[{len(value) - MAX_LIST_ITEMS} more items truncated]")
 return result

 # Primitives (int, float, bool, None) pass through unchanged
 return value


_SENSITIVE_KEYS = frozenset({
 "password", "secret", "token", "api_key", "apikey",
 "authorization", "credential", "private_key", "access_key",
 "session_token", "refresh_token", "ssn", "credit_card",
})

def _is_sensitive_key(key: str) -> bool:
 key_lower = key.lower().replace("-", "_").replace(" ", "_")
 return any(s in key_lower for s in _SENSITIVE_KEYS)

Part 4: Cost Tracking

Track token usage and compute cost from the LLM callback:

# agent/tracing/cost_callback.py
import os
from langchain_core.callbacks import AsyncCallbackHandler
from langchain_core.outputs import LLMResult
from .langfuse_client import langfuse

# Pricing per 1M tokens (update from OpenAI pricing page)
PRICING = {
 "gpt-4o": {"input": 5.00, "output": 15.00},
 "gpt-4o-mini": {"input": 0.15, "output": 0.60},
 "gpt-4-turbo": {"input": 10.00, "output": 30.00},
}


class CostCallbackHandler(AsyncCallbackHandler):
 """Callback that tracks token usage and computes cost after each LLM call."""

 def __init__(self, trace_id: str | None = None):
 self.trace_id = trace_id
 self.total_input_tokens = 0
 self.total_output_tokens = 0
 self.total_cost_usd = 0.0

 async def on_llm_end(self, response: LLMResult, **kwargs) -> None:
 usage = None

 # Handle both Chat Completions and Responses API formats
 if response.llm_output:
 usage = response.llm_output.get("token_usage") or \
 response.llm_output.get("usage")

 if usage is None:
 return

 input_tokens = usage.get("prompt_tokens", 0)
 output_tokens = usage.get("completion_tokens", 0)
 model = (response.llm_output or {}).get("model_name", "gpt-4o-mini")

 # Compute cost
 pricing = PRICING.get(model, PRICING["gpt-4o-mini"])
 cost = (
 input_tokens / 1_000_000 * pricing["input"] +
 output_tokens / 1_000_000 * pricing["output"]
 )

 self.total_input_tokens += input_tokens
 self.total_output_tokens += output_tokens
 self.total_cost_usd += cost

 # Log to Langfuse
 if self.trace_id:
 langfuse.generation(
 trace_id=self.trace_id,
 name="llm-call",
 model=model,
 usage={
 "input": input_tokens,
 "output": output_tokens,
 "unit": "TOKENS",
 },
 metadata={"cost_usd": round(cost, 6)},
 )

# agent/nodes.py — attach cost callback per call
from .tracing.cost_callback import CostCallbackHandler

def agent_node(state: AgentState, config: RunnableConfig) -> dict:
 trace_id = config.get("configurable", {}).get("langfuse_trace_id")
 cost_cb = CostCallbackHandler(trace_id=trace_id)

 messages = [SystemMessage(content=SYSTEM_PROMPT)] + state["messages"]
 response = llm_with_tools.invoke(messages, config={"callbacks": [cost_cb]})
 return {"messages": [response]}

Viewing Traces in Langfuse

After running your agent, open → Traces. You’ll see:

What you see	What it tells you
Trace timeline (waterfall)	Which step took the longest — usually the LLM call or a slow tool
Input/output per span	Exact messages the LLM received and returned — invaluable for prompt debugging
Token counts per generation	Identify high-cost sessions; track context window growth over turns
Cost per trace / per user	Attribute spending to users or features for billing / optimization
Error spans (red)	Which tool failed, with the exact error message and input that caused it

.env.example

# .env.example
OPENAI_API_KEY=your-api-key-here
LANGFUSE_PUBLIC_KEY=pk-lf-your-key
LANGFUSE_SECRET_KEY=sk-lf-your-key
LANGFUSE_HOST=https://cloud.langfuse.com
MONGODB_URI=mongodb://localhost:27017
MONGODB_DB=agent_memory

💡 Self-hosted Langfuse: Set LANGFUSE_HOST to your instance URL. Langfuse is fully open-source and can run locally via Docker Compose. See the .

Summary

Component	Purpose
`trace_agent_execution()`	Context manager wrapping the full agent run as one Langfuse trace
`@trace_tool`	Decorator that creates a Langfuse span per tool call
`sanitize_payload()`	Truncate large strings, mask sensitive keys before logging
`CostCallbackHandler`	LangChain callback that computes and logs token cost per LLM call
`langfuse.flush()`	Ensure all buffered events are sent before the request ends

In the final chapter, we close the loop: how to systematically measure whether your agent is actually working correctly, using a three-stage evaluation pipeline.

[Ch 8] Building an Evaluation System for Your Agent

Tue, 07 Apr 2026 00:00:00 +0000

How do you know if your agent is actually working well? User feedback is lagging and subjective. Manual inspection doesn’t scale. Unit tests can’t capture emergent LLM behavior. This chapter builds a principled evaluation pipeline that gives you a repeatable, quantitative answer.

Why Agent Evaluation Is Hard

Traditional software testing is deterministic: given input X, assert output Y. Agent evaluation is different:

Challenge	Why It’s Hard
Non-determinism	Same input can produce different tool call sequences
Long dependency chains	A failure in step 3 is caused by a decision in step 1
Multi-dimensional quality	Correctness, completeness, tone, and tool efficiency all matter
Latent regressions	Changing the system prompt can quietly degrade performance on edge cases
No ground truth	For open-ended tasks, there’s no single “correct” answer

The answer is a layered evaluation approach — combine deterministic checks where you can, and LLM judges where you must.

The Three-Stage Pipeline

graph LR A["📝 Stage 1\nGenerate Test Data\n(run agent on golden inputs)"] --> B["✅ Stage 2\nRule-based Eval\n(pytest assertions)"] B --> C["🤖 Stage 3\nLLM-as-Judge\n(DeepEval GEval)"] C --> D["📊 Score Report\n& Regression Detection"] style A fill:#2563EB,color:#fff,stroke:none style B fill:#059669,color:#fff,stroke:none style C fill:#D97706,color:#fff,stroke:none style D fill:#7C3AED,color:#fff,stroke:none

Fig 1: Three-stage evaluation pipeline — from test data generation to scored report

Installation

pip install deepeval pytest pytest-asyncio

# .env.example
OPENAI_API_KEY=your-api-key-here
DEEPEVAL_API_KEY=your-deepeval-key # optional: for Confident AI dashboard

Stage 1: Generate Test Data

The first stage runs your agent against a set of golden test cases — inputs where you know what a good response looks like — and saves the actual outputs for evaluation.

Define Test Cases

# evaluations/test_cases.py
from dataclasses import dataclass, field

@dataclass
class AgentTestCase:
 """A single evaluation scenario."""
 id: str
 user_input: str
 expected_tool_calls: list[str] # tools that should have been called
 expected_topics: list[str] # topics that should appear in the response
 should_not_contain: list[str] = field(default_factory=list) # forbidden content

TEST_CASES = [
 AgentTestCase(
 id="tc-001-create-high-priority",
 user_input="Create a high priority task to deploy the auth service to production",
 expected_tool_calls=["create_task"],
 expected_topics=["task", "created", "high", "deploy", "auth"],
 ),
 AgentTestCase(
 id="tc-002-list-filtered",
 user_input="Show me all tasks that are currently in progress",
 expected_tool_calls=["list_tasks"],
 expected_topics=["in_progress", "task"],
 should_not_contain=["todo", "done"],
 ),
 AgentTestCase(
 id="tc-003-policy-lookup",
 user_input="What is the expense report deadline?",
 expected_tool_calls=["search_knowledge_base"],
 expected_topics=["30 days", "receipt", "expense"],
 ),
 AgentTestCase(
 id="tc-004-multi-step",
 user_input="Create a task to review the Q1 report, then tell me all my current tasks",
 expected_tool_calls=["create_task", "list_tasks"],
 expected_topics=["created", "Q1", "review"],
 ),
 AgentTestCase(
 id="tc-005-error-handling",
 user_input="Update the status of task-999 to done",
 expected_tool_calls=["update_task_status"],
 expected_topics=["not found", "error", "task-999"],
 ),
]

Run Agent and Collect Outputs

# evaluations/01_generate_test_data.py
"""
Stage 1: Run the agent against all test cases and save actual outputs.
Run this script whenever you want to refresh the evaluation dataset.
"""
import asyncio
import json
import os
from pathlib import Path
from langchain_core.messages import HumanMessage, AIMessage, ToolMessage
from dotenv import load_dotenv
from agent.graph import build_graph
from evaluations.test_cases import TEST_CASES

load_dotenv()
OUTPUT_DIR = Path("evaluations/outputs")

async def collect_agent_output(app, test_case) -> dict:
 """Run one test case and collect structured output."""
 config = {"configurable": {"thread_id": f"eval-{test_case.id}"}}
 final_state = app.invoke(
 {"messages": [HumanMessage(content=test_case.user_input)]},
 config=config,
 )
 messages = final_state["messages"]

 # Extract the tool calls that were actually made
 actual_tool_calls = []
 for msg in messages:
 if isinstance(msg, AIMessage) and msg.tool_calls:
 actual_tool_calls.extend(c["name"] for c in msg.tool_calls)

 # Extract tool results
 tool_results = [
 msg.content for msg in messages if isinstance(msg, ToolMessage)
 ]

 # Final response
 final_response = messages[-1].content if messages else ""

 return {
 "test_case_id": test_case.id,
 "user_input": test_case.user_input,
 "final_response": final_response,
 "actual_tool_calls": actual_tool_calls,
 "tool_results": tool_results,
 "message_count": len(messages),
 }


async def main():
 OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
 app = build_graph(db_path=":memory:") # use in-memory DB for eval runs

 results = []
 for tc in TEST_CASES:
 print(f"Running [{tc.id}]: {tc.user_input[:60]}...")
 output = await collect_agent_output(app, tc)
 results.append(output)

 # Save individual output for inspection
 out_file = OUTPUT_DIR / f"{tc.id}.json"
 out_file.write_text(json.dumps(output, indent=2))

 # Save combined dataset
 dataset_file = OUTPUT_DIR / "dataset.json"
 dataset_file.write_text(json.dumps(results, indent=2))
 print(f"\n✅ Saved {len(results)} outputs to {OUTPUT_DIR}/")


if __name__ == "__main__":
 asyncio.run(main())

Stage 2: Rule-Based Evaluation

Fast, deterministic checks that run in milliseconds. These catch obvious failures: wrong tools called, required topics missing, forbidden content present.

# evaluations/02_rule_based_eval.py
"""
Stage 2: Rule-based assertions using pytest.
Run: pytest evaluations/02_rule_based_eval.py -v
"""
import json
import pytest
from pathlib import Path
from evaluations.test_cases import TEST_CASES, AgentTestCase

OUTPUT_DIR = Path("evaluations/outputs")


def load_output(test_case_id: str) -> dict:
 output_file = OUTPUT_DIR / f"{test_case_id}.json"
 if not output_file.exists():
 pytest.skip(f"Output for {test_case_id} not found — run Stage 1 first.")
 return json.loads(output_file.read_text())


# Parametrize over all test cases
@pytest.mark.parametrize("tc", TEST_CASES, ids=lambda tc: tc.id)
class TestRuleBasedEval:

 def test_expected_tools_were_called(self, tc: AgentTestCase):
 """Every expected tool must appear in actual_tool_calls."""
 output = load_output(tc.id)
 actual = output["actual_tool_calls"]
 for expected_tool in tc.expected_tool_calls:
 assert expected_tool in actual, (
 f"Expected tool '{expected_tool}' was not called.\n"
 f"Actual calls: {actual}"
 )

 def test_response_not_empty(self, tc: AgentTestCase):
 """Final response must be non-empty."""
 output = load_output(tc.id)
 assert output["final_response"].strip(), "Final response is empty"

 def test_response_minimum_length(self, tc: AgentTestCase):
 """Response should be at least 20 characters (not just a status code)."""
 output = load_output(tc.id)
 assert len(output["final_response"]) >= 20, (
 f"Response too short: '{output['final_response']}'"
 )

 def test_expected_topics_present(self, tc: AgentTestCase):
 """All expected topics should appear in the final response (case-insensitive)."""
 output = load_output(tc.id)
 response_lower = output["final_response"].lower()
 missing = [t for t in tc.expected_topics if t.lower() not in response_lower]
 assert not missing, (
 f"Missing expected topics: {missing}\n"
 f"Response: {output['final_response'][:200]}"
 )

 def test_forbidden_content_absent(self, tc: AgentTestCase):
 """No forbidden phrases should appear in the response."""
 if not tc.should_not_contain:
 pytest.skip("No forbidden content defined for this test case")
 output = load_output(tc.id)
 response_lower = output["final_response"].lower()
 found = [f for f in tc.should_not_contain if f.lower() in response_lower]
 assert not found, (
 f"Forbidden content found: {found}\n"
 f"Response: {output['final_response'][:200]}"
 )

 def test_no_extra_tool_calls(self, tc: AgentTestCase):
 """Agent should not call more tools than expected (efficiency check)."""
 output = load_output(tc.id)
 actual_count = len(output["actual_tool_calls"])
 expected_count = len(tc.expected_tool_calls)
 # Allow up to 1 extra call (e.g., for retries), but flag excessive calls
 assert actual_count <= expected_count + 1, (
 f"Too many tool calls: expected {expected_count}, got {actual_count}\n"
 f"Calls made: {output['actual_tool_calls']}"
 )

Stage 3: LLM-as-Judge with DeepEval GEval

Rule-based checks miss nuanced quality issues: a response can contain the right keywords but still be poorly reasoned, incomplete, or unhelpful. For this, we use an LLM to judge the response against a detailed rubric.

What is GEval?

GEval (Generation Evaluation) is DeepEval’s framework for defining custom LLM evaluation criteria with step-by-step reasoning. Each metric has:

A name — what you’re measuring
Evaluation steps — the rubric the judge LLM follows
A weight — how much this metric contributes to the overall score

Define Custom Metrics

# evaluations/metrics.py
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCaseParams

# Metric 1: Task Action Accuracy
# Does the response correctly reflect what the tools actually did?
task_accuracy = GEval(
 name="Task Action Accuracy",
 evaluation_steps=[
 "Read the user's request and the agent's final response carefully.",
 "Check if the response accurately reflects the actions taken (e.g., task created, list shown).",
 "Verify that task IDs, titles, priorities, and statuses mentioned in the response match the tool results.",
 "Penalize heavily if the response claims an action was performed that was not, or vice versa.",
 "Score 1.0 if fully accurate, 0.5 if partially accurate, 0.0 if inaccurate.",
 ],
 evaluation_params=[
 LLMTestCaseParams.INPUT,
 LLMTestCaseParams.ACTUAL_OUTPUT,
 LLMTestCaseParams.RETRIEVAL_CONTEXT, # tool results as context
 ],
 threshold=0.7,
)

# Metric 2: Response Completeness
# Does the response answer all parts of the user's request?
completeness = GEval(
 name="Response Completeness",
 evaluation_steps=[
 "List all distinct requests made by the user in their message.",
 "For each request, determine whether the response addresses it.",
 "A response is complete if it addresses all requests and provides necessary details.",
 "Partial credit if most but not all requests are addressed.",
 "Score 1.0 if fully complete, 0.0 if the main request is unanswered.",
 ],
 evaluation_params=[
 LLMTestCaseParams.INPUT,
 LLMTestCaseParams.ACTUAL_OUTPUT,
 ],
 threshold=0.7,
)

# Metric 3: Conciseness
# Is the response appropriately brief without losing key information?
conciseness = GEval(
 name="Conciseness",
 evaluation_steps=[
 "Assess whether the response contains unnecessary repetition or filler text.",
 "Good responses confirm actions clearly without restating the user's input verbatim.",
 "Penalize responses that are verbose or padded; reward tight, informative ones.",
 "A one-sentence confirmation for a simple action is ideal.",
 "Score 1.0 if concise, 0.5 if somewhat verbose, 0.0 if excessively padded.",
 ],
 evaluation_params=[
 LLMTestCaseParams.INPUT,
 LLMTestCaseParams.ACTUAL_OUTPUT,
 ],
 threshold=0.6,
)

# Weighted combination
WEIGHTED_METRICS = [
 (task_accuracy, 0.50), # most important: did it do the right thing?
 (completeness, 0.30), # did it address everything?
 (conciseness, 0.20), # was it well-phrased?
]

Run the LLM Evaluation

# evaluations/03_llm_eval.py
"""
Stage 3: LLM-as-judge evaluation using DeepEval GEval.
Run: python evaluations/03_llm_eval.py
"""
import json
from pathlib import Path
from deepeval import evaluate
from deepeval.test_case import LLMTestCase
from evaluations.test_cases import TEST_CASES
from evaluations.metrics import WEIGHTED_METRICS, task_accuracy, completeness, conciseness

OUTPUT_DIR = Path("evaluations/outputs")


def load_output(test_case_id: str) -> dict | None:
 output_file = OUTPUT_DIR / f"{test_case_id}.json"
 if not output_file.exists():
 return None
 return json.loads(output_file.read_text())


def build_llm_test_case(tc, output: dict) -> LLMTestCase:
 """Map our AgentTestCase + agent output to DeepEval's LLMTestCase."""
 return LLMTestCase(
 input=tc.user_input,
 actual_output=output["final_response"],
 retrieval_context=output["tool_results"], # tool results inform accuracy check
 expected_output=f"Should mention: {', '.join(tc.expected_topics)}",
 )


def compute_weighted_score(test_case: LLMTestCase) -> float:
 """Compute a weighted overall score across all metrics."""
 total_weight = 0.0
 total_score = 0.0

 for metric, weight in WEIGHTED_METRICS:
 metric.measure(test_case)
 total_score += metric.score * weight
 total_weight += weight

 return total_score / total_weight if total_weight > 0 else 0.0


def main():
 llm_test_cases = []
 for tc in TEST_CASES:
 output = load_output(tc.id)
 if output is None:
 print(f"⚠️ Skipping {tc.id} — run Stage 1 first.")
 continue
 llm_test_cases.append((tc, build_llm_test_case(tc, output)))

 if not llm_test_cases:
 print("No test cases to evaluate. Run Stage 1 first.")
 return

 print(f"\nEvaluating {len(llm_test_cases)} test cases with LLM judge...\n")

 results = []
 for tc, ltc in llm_test_cases:
 weighted_score = compute_weighted_score(ltc)
 passed = weighted_score >= 0.7 # overall passing threshold

 results.append({
 "id": tc.id,
 "input": tc.user_input[:60],
 "weighted_score": round(weighted_score, 3),
 "passed": passed,
 })

 status = "✅ PASS" if passed else "❌ FAIL"
 print(f"[{status}] {tc.id}: {weighted_score:.3f}")

 # ── Run DeepEval's built-in evaluate() for the full report ───────────────
 all_ltc = [ltc for _, ltc in llm_test_cases]
 all_metrics = [task_accuracy, completeness, conciseness]
 evaluate(test_cases=all_ltc, metrics=all_metrics)

 # ── Print summary ──────────────────────────────────────────────────────────
 passed_count = sum(1 for r in results if r["passed"])
 avg_score = sum(r["weighted_score"] for r in results) / len(results)
 print(f"\n{'='*50}")
 print(f"Results: {passed_count}/{len(results)} passed")
 print(f"Average weighted score: {avg_score:.3f}")
 print(f"{'='*50}")

 # Save results for trend tracking
 results_file = OUTPUT_DIR / "llm_eval_results.json"
 results_file.write_text(json.dumps(results, indent=2))
 print(f"\nDetailed results saved to {results_file}")


if __name__ == "__main__":
 main()

Running the Full Pipeline

# Step 1: Generate test data (run the agent on all golden inputs)
python evaluations/01_generate_test_data.py

# Step 2: Rule-based checks (fast, < 1 second)
pytest evaluations/02_rule_based_eval.py -v

# Step 3: LLM-as-judge (slower, ~30-60 seconds depending on case count)
python evaluations/03_llm_eval.py

Sample output from Stage 2:

PASSED evaluations/02_rule_based_eval.py::TestRuleBasedEval::test_expected_tools_were_called[tc-001]
PASSED evaluations/02_rule_based_eval.py::TestRuleBasedEval::test_expected_topics_present[tc-001]
FAILED evaluations/02_rule_based_eval.py::TestRuleBasedEval::test_expected_topics_present[tc-005]
AssertionError: Missing expected topics: ['not found']

Sample output from Stage 3:

[✅ PASS] tc-001-create-high-priority: 0.891
[✅ PASS] tc-002-list-filtered: 0.843
[✅ PASS] tc-003-policy-lookup: 0.776
[✅ PASS] tc-004-multi-step: 0.801
[❌ FAIL] tc-005-error-handling: 0.623

==================================================
Results: 4/5 passed
Average weighted score: 0.787
==================================================

When to Run Evaluations

Trigger	Stage to Run	Why
After any system prompt change	Stage 2 + 3	Prompt changes silently affect behavior
After adding/modifying a tool	Stage 2	Check tool call patterns
After upgrading the LLM version	All 3	Model updates change behavior
Before every production release	All 3	Full regression check
Weekly (scheduled)	All 3	Catch model drift from provider updates

.env.example

# .env.example
OPENAI_API_KEY=your-api-key-here
DEEPEVAL_API_KEY=your-deepeval-key # optional: enables Confident AI dashboard

💡 Ollama note: DeepEval’s GEval by default uses OpenAI as the judge LLM. To use a local model as the judge, configure it with:
from deepeval.models import DeepEvalBaseLLM
# Implement a custom judge class wrapping OllamaLLM
# See: https://docs.confident-ai.com/docs/metrics-custom-judge
Note: smaller models (< 13B parameters) often produce inconsistent rubric scores. For reliable LLM judging, use at least GPT-4o-mini or an equivalent capable model.

Summary

Stage	Method	Speed	What It Catches
1	Agent execution on golden inputs	~5–30s	Nothing — generates data
2	`pytest` rule-based assertions	< 1s	Wrong tools, missing topics, forbidden content
3	DeepEval GEval LLM-as-judge	~30–60s	Nuanced quality, accuracy, completeness

Series Complete

You’ve now built a production-grade AI agent from scratch:

Chapter	What You Built
Ch 1	Mental model: agent loop, ReAct, when to use agents
Ch 2	Context window anatomy, token budget management
Ch 3	LangChain + LangGraph fundamentals
Ch 4	Full multi-turn agent with SQLite persistence and streaming
Ch 5	AgentContext injection, NeMo Guardrails, HITL interrupts
Ch 6	Four-layer memory: trim, MongoDB, FAISS RAG, system prompt
Ch 7	Full Langfuse observability: traces, spans, cost tracking
Ch 8	Three-stage evaluation pipeline with DeepEval GEval

The full source code for this series is organized as a single Python package. Combine these pieces and you have a system that is observable, safe, memory-aware, and measurably correct.

Build Enterprise AI Agent from Scratch: Series Overview

Tue, 07 Apr 2026 00:00:00 +0000

There is no shortage of AI agent tutorials on the internet. Most of them show you how to build an agent in 50 lines of code — it calls a search tool, formats an answer, and you’re done. That’s fine for a weekend demo.

But production is different.

In production, your agent needs to handle adversarial inputs without leaking sensitive data. It needs to recover gracefully when a tool fails mid-task. It needs to remember context across sessions, not just within a single conversation window. You need to know exactly what your LLM did when a user reports a bug — and calculate how much it cost. And before you ship any of this, you need a principled way to measure whether it actually works.

I wrote this series because I spent years building enterprise AI agent systems and found that most resources stop right before the hard parts begin.

Who This Series Is For

This series is for software engineers and ML practitioners who:

Have basic Python skills and understand what a Large Language Model (LLM) does
Have heard of LangChain/LangGraph but haven’t gone deep into production usage
Want to build agents that are deployable, observable, and maintainable — not just impressive in a notebook

You don’t need to have built an agent before. But you should be comfortable reading Python code and understand concepts like API calls and environment variables.

What You’ll Build

Across eight chapters, we build a fully-featured AI agent incrementally. Each chapter adds a new capability. By the end, you’ll have:

A working ReAct agent powered by LangGraph with proper state management
Multiple integrated tools with safe context injection (no secrets in the message stream)
NeMo Guardrails for input/output content safety
Human-in-the-loop interrupts for high-stakes decisions
Four-layer memory architecture: in-context, episodic, semantic (RAG), and procedural
Full Langfuse observability: traces, spans, cost tracking, and payload sanitization
A 3-stage evaluation pipeline: rule-based checks, LLM-as-judge, and custom rubric scoring

Real-World Background

The patterns in this series are not theoretical. They come from working on two different enterprise AI systems:

An automated document generation agent that uses a multi-step LangGraph workflow to produce structured outputs from unstructured inputs
A multimodal chatbot that creates and edits presentation files through natural language, with sandboxed code execution for chart generation

Both systems run in production with real users. The lessons in this series — especially around memory management, observability, and evaluation — come directly from debugging and improving those systems over time.

All code in this series is original and generalized. No proprietary details, internal tooling, or credentials from those systems are used.

Prerequisites

Required:

Python 3.10+
An OpenAI API key (sign up at )
Basic familiarity with how LLMs / ChatGPT work conceptually

Helpful but not required:

Experience with FastAPI or any Python web framework
Basic understanding of async Python (async def, await)

💡 Using a local model instead? Every chapter that contains OpenAI code includes a note showing how to swap ChatOpenAI for ChatOllama (free, runs locally). You’ll need installed with a model like llama3.2.

Series Outline

Chapter	Title	Key Topics
Ch 0	Series Overview (this post)	Motivation, prerequisites, roadmap
	Introduction to AI Agents	Agent loop, chatbot vs. agent, when NOT to use agents
	Components & Context Engineering	4 core components, Prompt Eng vs. Context Eng, token budget
	LangChain & LangGraph Intro	Messages, @tool, StateGraph, Hello World agent
Ch 4	Build Your First Agent	AgentState, tool_node, streaming, SQLite checkpointer
Ch 5	Tools, Guardrails & Safety	Tool design, AgentContext, NeMo Guardrails, HITL
Ch 6	Memory Management	4-layer memory, MongoDB checkpointer, FAISS RAG
Ch 7	Tracing with Langfuse	trace_agent_execution, @trace_tool, cost callbacks
Ch 8	Evaluation System	3-stage pipeline, DeepEval GEval, custom rubrics

How to Follow Along

Each chapter is self-contained. You can read sequentially or jump to a specific topic. However, Chapter 4 is the foundation — if you’re skipping ahead to Ch 5–8, make sure you’ve at least read Ch 4 first.

Code conventions used throughout:

File path is shown as a comment at the top of every snippet: # agent/graph.py
Secrets always use environment variables: os.environ.get("OPENAI_API_KEY")
Every chapter includes a .env.example showing which variables are needed
Ollama switch instructions are in a callout box in every chapter with LLM code

Let’s build.

Build Enterprise AI Agent from Scratch |

[Ch 1] Introduction to AI Agents — Why Do We Need Them?

What Is an AI Agent?

The Spectrum: Chatbot → Pipeline → Agent

The Agent Loop (ReAct)

Real-World Use Cases

When NOT to Use an Agent

The Agent Loop in Plain Python

.env.example

Summary

[Ch 2] AI Agent Components & Context Engineering

The Four Core Components

1. LLM Backbone

2. Tools

3. Memory

4. Planning

Prompt Engineering vs. Context Engineering

Anatomy of a Context Window

Token Budget: A Practical Example

Context Engineering Strategies

1. System Prompt — Keep It Focused

2. Tool Schemas — Prune Unused Tools

3. Conversation History — Trim Aggressively

4. RAG Chunks — Quality Over Quantity

The Mental Model

.env.example

Summary

[Ch 3] Getting Started with LangChain & LangGraph

Installation

Part 1: LangChain Core Concepts

1.1 Messages

1.2 ChatOpenAI

1.3 The @tool Decorator

1.4 RunnableConfig

Part 2: LangGraph Core Concepts

2.1 The State

2.2 Nodes

2.3 Edges and Conditional Edges

2.4 Building the Graph

Part 3: Hello World Agent

LangChain vs. LangGraph: When to Use Which

.env.example

Summary

[Ch 4] Build Your First Production-Ready Agent

Project Structure

Installation

Step 1 — Define the State

Step 2 — Define the Tools

Step 3 — Define the Nodes

Step 4 — Assemble the Graph with a Checkpointer

Step 5 — Run with Streaming

Understanding Multi-Thread Isolation

Inspecting Persisted State

Summary

[Ch 5] Tools Integration, Guardrails & Safety Patterns

The Three Safety Layers

Part 1: AgentContext — Safe User Context Injection

The Problem

The Solution: config["configurable"]

Part 2: NeMo Guardrails

Installation

Guardrail Configuration

Integrating Guardrails with Your Agent

Part 3: Human-in-the-Loop (HITL)

How interrupt() Works

Implementation

Tool Design Checklist

.env.example

Summary

[Ch 6] Memory Management for Production Agents

The Four Memory Layers

Layer 1: In-Context Memory — Managing History Growth

Strategy A: Sliding Window (Trim)

Strategy B: Summarization

Layer 2: Episodic Memory — MongoDB Checkpointer

Layer 3: Semantic Memory — RAG with FAISS

Layer 4: Procedural Memory — The System Prompt

Putting It All Together

.env.example

Summary

1.3 The `@tool` Decorator

The Solution: `config["configurable"]`

How `interrupt()` Works