Memory |

[Ch 2] AI Agent Components & Context Engineering

Tue, 07 Apr 2026 00:00:00 +0000

Knowing that an agent has a “loop” is not enough to build a good one. You need to understand what the system is made of — and exactly what the LLM sees at every step. This chapter covers both.

The Four Core Components

Every AI agent system, regardless of framework, is built from four components:

graph LR A[🧠 LLM Backbone] --- B[🔧 Tools] A --- C[🗃️ Memory] A --- D[🗺️ Planning] style A fill:#7C3AED,color:#fff,stroke:none style B fill:#2563EB,color:#fff,stroke:none style C fill:#059669,color:#fff,stroke:none style D fill:#D97706,color:#fff,stroke:none

Fig 1: The four pillars of an AI agent

1. LLM Backbone

The language model is the brain. It decides what to do next: whether to call a tool, what arguments to pass, and when to stop. The quality, speed, and cost of your agent depend heavily on which model you choose.

Key decisions:

Model choice: GPT-4o for complex reasoning; GPT-4o-mini for simpler, high-throughput tasks
Temperature: Keep it low (0–0.3) for tool-calling agents — you want deterministic decisions, not creative variance
Context window size: Determines how much history and retrieved content the model can “see”

2. Tools

Tools are functions the LLM can call to interact with the outside world. Common categories:

Category	Examples
Information retrieval	Web search, vector store lookup, database query
Action execution	File write, API call, form submission
Computation	Code execution, math evaluation, data parsing
Communication	Send email, post to Slack, create a ticket

A tool is just a function with a clear name, description, and typed parameters — the model reads the description to know when and how to use it.

3. Memory

Memory is how the agent maintains context across steps and sessions. There are four distinct memory types (covered in depth in Ch 6):

In-context: The current conversation history in the message list
Episodic: Persisted conversation state (checkpointer / DB)
Semantic: Long-term knowledge retrieved via RAG (vector store)
Procedural: The system prompt — “how to behave” instructions baked into every request

4. Planning

Planning is the agent’s ability to decompose a complex goal into steps. This ranges from implicit (the LLM decides step-by-step via ReAct) to explicit (a dedicated “planner” LLM call generates a structured plan that an “executor” then follows).

For most enterprise agents, the implicit ReAct planning is sufficient — and simpler to debug. Explicit multi-step planners are powerful but add complexity; reach for them only when the task requires long-horizon coordination across many sub-tasks.

Prompt Engineering vs. Context Engineering

Here’s a distinction that will save you weeks of frustration in production.

Prompt Engineering is about what words you use — how to phrase instructions to get the model to behave the way you want. It’s the craft of writing good system prompts, user messages, and few-shot examples.

Context Engineering is about what the model actually sees — managing everything that occupies the context window at each step of the agent loop.

Context Engineering is a superset of Prompt Engineering.

In a single LLM API call, you have to write a good prompt. In an agent system, you have to do that and manage every byte in the context window across potentially dozens of steps.

Anatomy of a Context Window

Every time the LLM is called, it receives a context window. In an agent loop, that window contains multiple layers:

block-beta columns 1 block:context["Context Window (e.g. 128,000 tokens)"] A["📋 System Prompt\n(instructions, persona, output format)"] B["🔧 Tool Schemas\n(name, description, parameters for each tool)"] C["💬 Conversation History\n(all prior messages + tool results)"] D["📚 Retrieved Documents\n(RAG chunks injected for this turn)"] E["❓ Current User Message"] end style A fill:#7C3AED,color:#fff style B fill:#2563EB,color:#fff style C fill:#059669,color:#fff style D fill:#D97706,color:#fff style E fill:#DC2626,color:#fff

Fig 2: Everything competing for space in the context window

The crucial insight: tools eat tokens even when they’re not used.

A typical tool schema (name + description + parameter definitions) costs 100–300 tokens. If your agent has 15 tools, that’s 1,500–4,500 tokens consumed before any conversation history or retrieved documents. For an agent with a 32K token budget running a long conversation, this becomes a real constraint.

Token Budget: A Practical Example

Let’s measure the real cost of each context component:

# token_budget.py
import os
import tiktoken

# GPT-4o uses the o200k_base encoding
ENCODING = tiktoken.get_encoding("o200k_base")

def count_tokens(text: str) -> int:
 return len(ENCODING.encode(text))

# --- Simulate context components ---

SYSTEM_PROMPT = """You are an expert document analyst assistant. You help users
understand and extract information from complex technical documents. You are precise,
thorough, and always cite your sources. When a document is provided, read it carefully
before answering. Format your responses in clear markdown."""

TOOL_SCHEMA_EXAMPLE = """Tool: search_documents
Description: Search through the document knowledge base for relevant passages.
Parameters:
 - query (string, required): The search query to find relevant document sections
 - max_results (integer, optional): Maximum number of results to return (default: 5)
 - document_filter (string, optional): Filter results to a specific document by name"""

# Simulate 10 back-and-forth turns
CONVERSATION_HISTORY = """
User: Can you find all mentions of the payment terms in the contract?
Assistant: [calls search_documents(query="payment terms")]
Tool Result: Found 3 sections mentioning payment terms: Section 4.2, 7.1, and Appendix B.
Assistant: I found payment terms in three locations. Section 4.2 covers...
[... 9 more turns ...]
""" * 3 # multiply to simulate a longer conversation

RETRIEVED_DOCS = """
[Retrieved chunk 1 - Section 4.2]: Payment shall be made within 30 days of invoice...
[Retrieved chunk 2 - Section 7.1]: Late payment penalties apply at 1.5% per month...
[Retrieved chunk 3 - Appendix B]: Extended payment plans available upon request...
""" * 5 # 5 retrieved chunks

CURRENT_QUERY = "What is the penalty for paying 60 days late?"

# --- Count and report ---
components = {
 "System Prompt": SYSTEM_PROMPT,
 "Tool Schema (x10 tools estimate)": TOOL_SCHEMA_EXAMPLE * 10,
 "Conversation History (10 turns)": CONVERSATION_HISTORY,
 "Retrieved Documents (5 chunks)": RETRIEVED_DOCS,
 "Current User Query": CURRENT_QUERY,
}

total = 0
print(f"{'Component':<40} {'Tokens':>8} {'% of 32K':>10}")
print("-" * 62)
for name, content in components.items():
 tokens = count_tokens(content)
 total += tokens
 print(f"{name:<40} {tokens:>8,} {tokens/32768*100:>9.1f}%")

print("-" * 62)
print(f"{'TOTAL':<40} {total:>8,} {total/32768*100:>9.1f}%")
print(f"\nRemaining budget (32K model): {max(0, 32768 - total):,} tokens")
print(f"Remaining budget (128K model): {max(0, 131072 - total):,} tokens")

Running this produces output like:

Component Tokens % of 32K
--------------------------------------------------------------
System Prompt 52 0.2%
Tool Schema (x10 tools estimate) 1,430 4.4%
Conversation History (10 turns) 1,710 5.2%
Retrieved Documents (5 chunks) 975 3.0%
Current User Query 16 0.0%
--------------------------------------------------------------
TOTAL 4,183 12.8%

Remaining budget (32K model): 28,585 tokens
Remaining budget (128K model): 126,889 tokens

Ten turns into a conversation with a moderate number of tools and RAG chunks, you’ve used only 13% of a 32K window — but that grows fast. At 30+ turns with larger documents, you’ll hit limits and need to trim.

Context Engineering Strategies

Once you understand what’s in the context window, you can manage it deliberately:

1. System Prompt — Keep It Focused

The system prompt runs on every single call. Every unnecessary sentence costs tokens across your entire agent’s lifetime. Write precisely:

# ❌ Vague, padded
BAD_SYSTEM_PROMPT = """
You are a very helpful and friendly AI assistant. You are smart and knowledgeable
about many things. You always try to be nice and helpful to users. When users ask
questions, you do your best to answer them accurately and helpfully...
"""

# ✅ Precise, structured
GOOD_SYSTEM_PROMPT = """You are a document analysis assistant.

CAPABILITIES:
- Search documents using search_documents tool
- Extract and cite specific passages
- Summarize findings in structured markdown

CONSTRAINTS:
- Only answer questions about documents in the knowledge base
- Always cite the source section when quoting
- If information is not found, say so explicitly
"""

2. Tool Schemas — Prune Unused Tools

Don’t pass every tool on every call. If your agent has a “create_document” tool but the user just asked a read-only question, remove it from the current call. In LangGraph, you can swap the tool list dynamically per node.

3. Conversation History — Trim Aggressively

Keep a sliding window of the last N turns, or summarize old turns into a single “context block.” We’ll implement this in Ch 6.

4. RAG Chunks — Quality Over Quantity

Don’t inject 20 retrieved chunks “just in case.” Use a relevance threshold: only inject chunks above a similarity score of 0.75. Three highly relevant chunks beat ten mediocre ones — and cost far less context.

The Mental Model

Think of the context window as a whiteboard that the LLM reads before every response. You control everything on that whiteboard:

What instructions appear (system prompt)
What tools are available (tool schemas)
What happened before (conversation history)
What facts are relevant (retrieved documents)
What the user just asked (current message)

Context Engineering is the discipline of making every token on that whiteboard count.

.env.example

# .env.example
OPENAI_API_KEY=your-api-key-here

💡 Ollama note: The token counting above uses tiktoken (OpenAI’s tokenizer). Ollama models use different tokenizers (Llama uses sentencepiece). Token counts will vary, but the principles — keep system prompts tight, prune unused tools, trim history — apply universally.

Summary

Concept	Key Takeaway
LLM Backbone	The decision-maker; choose model + temperature carefully
Tools	Functions the LLM can call; each schema costs tokens
Memory	Four types: in-context, episodic, semantic, procedural
Planning	Implicit (ReAct) vs. explicit; start with implicit
Prompt Engineering	Craft of writing good instructions and examples
Context Engineering	Managing everything in the context window across all steps
Token Budget	System prompt + tool schemas + history + RAG + query = limited budget

In the next chapter, we get hands-on with LangChain and LangGraph — the frameworks that automate the agent loop we’ve been building mentally.

[Ch 6] Memory Management for Production Agents

Tue, 07 Apr 2026 00:00:00 +0000

The SQLite checkpointer from Ch 4 persists conversation history — but that’s just one type of memory. Production agents need four distinct layers, each serving a different purpose and operating on a different time horizon.

The Four Memory Layers

graph TD subgraph "Short-term (per session)" A["🧠 In-Context Memory\nCurrent message list in context window\nLost when window fills or session ends"] B["📼 Episodic Memory\nFull conversation history in DB\nPersisted across sessions per thread_id"] end subgraph "Long-term (across sessions)" C["📚 Semantic Memory\nVector store of facts & documents\nRetrieved on-demand via RAG"] D["📋 Procedural Memory\nSystem prompt — rules & persona\nBaked into every single LLM call"] end style A fill:#7C3AED,color:#fff,stroke:none style B fill:#059669,color:#fff,stroke:none style C fill:#D97706,color:#fff,stroke:none style D fill:#2563EB,color:#fff,stroke:none

Fig 1: Four memory layers with different scope and persistence

Layer	Storage	Scope	Limits
In-Context	Message list in RAM	Current LLM call	Context window size
Episodic	Database (SQLite / MongoDB)	Per `thread_id`	Disk space
Semantic	Vector store (FAISS / Qdrant)	All users / sessions	Embedding index size
Procedural	Source code (system prompt)	Every call	Prompt token budget

Layer 1: In-Context Memory — Managing History Growth

As conversations grow, the message list eventually overflows the context window or becomes expensive. Two standard strategies:

Strategy A: Sliding Window (Trim)

Keep only the last N messages, discarding older ones:

# agent/memory/trim.py
from langchain_core.messages import AnyMessage, SystemMessage, trim_messages
from langchain_openai import ChatOpenAI


def trim_conversation(
 messages: list[AnyMessage],
 max_tokens: int = 4000,
 model: str = "gpt-4o-mini",
) -> list[AnyMessage]:
 """Trim messages to stay within a token budget, always keeping the system message."""
 return trim_messages(
 messages,
 max_tokens=max_tokens,
 strategy="last", # keep the most recent messages
 token_counter=ChatOpenAI(model=model),
 include_system=True, # never trim the system message
 allow_partial=False, # don't cut messages mid-way
 )

# agent/nodes.py — use trim inside the agent node
def agent_node(state: AgentState) -> dict:
 trimmed = trim_conversation(state["messages"], max_tokens=4000)
 messages = [SystemMessage(content=SYSTEM_PROMPT)] + trimmed
 response = llm_with_tools.invoke(messages)
 return {"messages": [response]}

Strategy B: Summarization

When trimming loses important early context (e.g., user goals set at the start), summarize instead:

# agent/memory/summarize.py
from langchain_core.messages import AnyMessage, HumanMessage, AIMessage, SystemMessage
from langchain_openai import ChatOpenAI

summarizer = ChatOpenAI(model="gpt-4o-mini", temperature=0)

SUMMARY_PROMPT = """Summarize the following conversation in 2-3 sentences.
Focus on: decisions made, key facts established, and current task status.
Be concise — this summary will replace the conversation history."""


def summarize_messages(messages: list[AnyMessage]) -> str:
 """Produce a compact summary of a list of messages."""
 conversation_text = "\n".join(
 f"{msg.__class__.__name__}: {msg.content}"
 for msg in messages
 if msg.content # skip tool-call-only messages
 )
 response = summarizer.invoke([
 SystemMessage(content=SUMMARY_PROMPT),
 HumanMessage(content=conversation_text),
 ])
 return response.content


def maybe_summarize(
 messages: list[AnyMessage],
 threshold: int = 20,
) -> list[AnyMessage]:
 """If history exceeds threshold messages, summarize the older half."""
 if len(messages) <= threshold:
 return messages

 midpoint = len(messages) // 2
 old_messages = messages[:midpoint]
 recent_messages = messages[midpoint:]

 summary = summarize_messages(old_messages)

 # Replace old messages with a single summary message
 summary_msg = AIMessage(
 content=f"[Conversation summary — earlier context]\n{summary}"
 )
 return [summary_msg] + recent_messages

Layer 2: Episodic Memory — MongoDB Checkpointer

SQLite is great for development but not for production (no concurrent writes, no horizontal scaling). For production, use the MongoDB checkpointer:

pip install langchain-mongodb motor

# agent/graph_mongo.py
import os
from motor.motor_asyncio import AsyncIOMotorClient
from langgraph.checkpoint.mongodb.aio import AsyncMongoDBSaver
from langgraph.graph import END, StateGraph
from langgraph.prebuilt import ToolNode
from .state import AgentState
from .nodes import agent_node, should_continue
from .tools import TOOLS

MONGO_URI = os.environ.get("MONGODB_URI", "mongodb://localhost:27017")
MONGO_DB = os.environ.get("MONGODB_DB", "agent_memory")


async def build_async_graph():
 """Build the agent graph with async MongoDB checkpointer."""
 client = AsyncIOMotorClient(MONGO_URI)
 checkpointer = AsyncMongoDBSaver(client, db_name=MONGO_DB)

 graph = StateGraph(AgentState)
 graph.add_node("agent", agent_node)
 graph.add_node("tools", ToolNode(TOOLS))
 graph.set_entry_point("agent")
 graph.add_conditional_edges("agent", should_continue, {"tools": "tools", "end": END})
 graph.add_edge("tools", "agent")

 return graph.compile(checkpointer=checkpointer)

# agent/main_async.py
import asyncio
from langchain_core.messages import HumanMessage
from .graph_mongo import build_async_graph


async def main():
 app = await build_async_graph()

 config = {"configurable": {"thread_id": "user-42-session-1"}}
 inputs = {"messages": [HumanMessage(content="List all my tasks")]}

 async for chunk, metadata in app.astream(inputs, config=config, stream_mode="messages"):
 if chunk.content and metadata.get("langgraph_node") == "agent":
 print(chunk.content, end="", flush=True)
 print()


if __name__ == "__main__":
 asyncio.run(main())

# .env.example (updated)
OPENAI_API_KEY=your-api-key-here
MONGODB_URI=mongodb://localhost:27017
MONGODB_DB=agent_memory

MongoDB vs SQLite at a glance:

	SQLite	MongoDB
Setup	Zero-config	Requires running server
Concurrent writes	❌ Single writer	✅ Multi-writer
Horizontal scaling	❌ File-based	✅ Replica sets / Atlas
Production-ready	Dev only	✅ Yes
LangGraph support	`SqliteSaver`	`AsyncMongoDBSaver`

Layer 3: Semantic Memory — RAG with FAISS

Episodic memory stores what happened. Semantic memory stores what the agent knows — a searchable knowledge base. The agent retrieves relevant facts at query time rather than scanning the entire history.

pip install langchain-community faiss-cpu tiktoken

# agent/memory/knowledge_base.py
import os
from langchain_community.vectorstores import FAISS
from langchain_openai import OpenAIEmbeddings
from langchain_core.documents import Document

EMBED_MODEL = OpenAIEmbeddings(
 model="text-embedding-3-small",
 api_key=os.environ.get("OPENAI_API_KEY"),
)

# ── Build the index ───────────────────────────────────────────────────────────
def build_knowledge_base(documents: list[dict]) -> FAISS:
 """
 Build a FAISS vector store from a list of {content, metadata} dicts.
 In production, load these from a database or file store.
 """
 docs = [
 Document(page_content=d["content"], metadata=d.get("metadata", {}))
 for d in documents
 ]
 return FAISS.from_documents(docs, EMBED_MODEL)


# ── Example knowledge base ────────────────────────────────────────────────────
COMPANY_DOCS = [
 {
 "content": "The annual leave policy allows 20 days per year. Requests must be submitted 2 weeks in advance.",
 "metadata": {"source": "hr-policy", "section": "leave"},
 },
 {
 "content": "Expense reports must be filed within 30 days of the expense date. Receipts are required for amounts over $25.",
 "metadata": {"source": "finance-policy", "section": "expenses"},
 },
 {
 "content": "The on-call rotation schedule is updated every Monday. Check the team wiki for the current schedule.",
 "metadata": {"source": "engineering-policy", "section": "on-call"},
 },
 {
 "content": "All production deployments require approval from the tech lead and a passing CI pipeline.",
 "metadata": {"source": "engineering-policy", "section": "deployments"},
 },
]

knowledge_base = build_knowledge_base(COMPANY_DOCS)


# ── Save and load ─────────────────────────────────────────────────────────────
def save_index(index: FAISS, path: str = "faiss_index") -> None:
 index.save_local(path)

def load_index(path: str = "faiss_index") -> FAISS:
 return FAISS.load_local(path, EMBED_MODEL, allow_dangerous_deserialization=True)

# agent/tools.py — RAG search as a tool
from pydantic import BaseModel, Field
from langchain_core.tools import tool
from .memory.knowledge_base import knowledge_base

class SearchKnowledgeInput(BaseModel):
 query: str = Field(description="The question or topic to search for in the knowledge base")
 k: int = Field(default=3, description="Number of relevant documents to retrieve (1-5)")

@tool("search_knowledge_base", args_schema=SearchKnowledgeInput)
def search_knowledge_base(query: str, k: int = 3) -> str:
 """Search the company knowledge base for policies, procedures, and guidelines."""
 results = knowledge_base.similarity_search_with_score(query, k=k)

 if not results:
 return "No relevant information found in the knowledge base."

 output_lines = [f"Found {len(results)} relevant document(s):\n"]
 for i, (doc, score) in enumerate(results, 1):
 # Only return results above a relevance threshold
 if score > 0.4: # FAISS L2 distance — lower = more similar
 continue
 source = doc.metadata.get("source", "unknown")
 output_lines.append(f"[{i}] (source: {source})\n{doc.page_content}")

 return "\n\n".join(output_lines) if len(output_lines) > 1 else "No sufficiently relevant results found."

💡 FAISS vs Qdrant: FAISS is in-memory and zero-infrastructure — perfect for tutorials and small knowledge bases (< 100K documents). For production with millions of documents, filtered search per user, or real-time updates, use or instead.

Layer 4: Procedural Memory — The System Prompt

The system prompt is the most overlooked memory layer. It defines how the agent behaves on every single call — its persona, constraints, output format, and domain-specific rules. Things that never change belong here.

# agent/nodes.py — dynamic system prompt
from datetime import datetime, timezone

def build_system_prompt(ctx: "AgentContext | None" = None) -> str:
 """Build the system prompt, optionally personalized with user context."""
 base = """You are a task management assistant for engineering teams.

CAPABILITIES:
- Create, update, list, and delete tasks
- Search the company knowledge base for policies
- Request human confirmation before destructive actions

OUTPUT FORMAT:
- Use concise, action-oriented language
- After tool calls, summarize results in 1-2 sentences
- Format task lists as bullet points with status indicators

CONSTRAINTS:
- Only manage tasks within the user's permission scope
- Never expose other users' task data
- Always cite the source when answering policy questions"""

 # Add dynamic context
 now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
 base += f"\n\nCurrent date/time: {now}"

 if ctx:
 base += f"\nCurrent user: {ctx.user_id} (org: {ctx.org_id})"
 base += f"\nPermissions: {', '.join(ctx.permissions)}"

 return base

The procedural memory principle: if something is true for every call, put it in the system prompt. If it’s true for this call only, put it in the user message or retrieved context.

Putting It All Together

graph LR U([User Query]) --> R["📚 RAG Retrieval\n(Semantic Memory)"] R --> CTX["Build Context Window:\n• System Prompt (Procedural)\n• Trimmed History (In-context)\n• Retrieved Docs (Semantic)\n• User Query"] CTX --> LLM["🧠 LLM"] LLM --> A["Agent Response"] A --> DB["💾 Save to MongoDB\n(Episodic Memory)"] style U fill:#4CAF50,color:#fff,stroke:none style A fill:#2196F3,color:#fff,stroke:none style LLM fill:#9C27B0,color:#fff,stroke:none style DB fill:#059669,color:#fff,stroke:none

Fig 2: All four memory layers working together in one agent call

# agent/nodes.py — full agent node with all memory layers integrated
from .memory.trim import trim_conversation
from .memory.knowledge_base import search_knowledge_base

def agent_node(state: AgentState, config: RunnableConfig) -> dict:
 # Layer 4: Procedural memory
 ctx_data = config.get("configurable", {}).get("agent_context")
 ctx = AgentContext(**ctx_data) if ctx_data else None
 system_msg = SystemMessage(content=build_system_prompt(ctx))

 # Layer 1: In-context memory with trimming
 trimmed_messages = trim_conversation(state["messages"], max_tokens=4000)

 messages = [system_msg] + trimmed_messages
 response = llm_with_tools.invoke(messages)
 return {"messages": [response]}

.env.example

# .env.example
OPENAI_API_KEY=your-api-key-here
MONGODB_URI=mongodb://localhost:27017
MONGODB_DB=agent_memory

💡 Ollama note: Replace OpenAIEmbeddings with:
from langchain_ollama import OllamaEmbeddings
EMBED_MODEL = OllamaEmbeddings(model="nomic-embed-text")
Pull the embedding model first: ollama pull nomic-embed-text

Summary

Memory Layer	Implementation	When to Use
In-Context	`trim_messages()` / summarization	Always — control token growth
Episodic	`SqliteSaver` (dev) / `AsyncMongoDBSaver` (prod)	Multi-turn conversations
Semantic	FAISS (dev) / Qdrant (prod) + `@tool`	Knowledge base Q&A, document retrieval
Procedural	System prompt (static + dynamic parts)	Persona, constraints, output format

In the next chapter, we add observability — making every LLM call, tool execution, and token cost visible via Langfuse.