AI Agent Runtimes: From LLM Calls to Full Agent Harnesses

AI Agent Runtimes: From LLM Calls to Full Agent Harnesses

Modern AI agents are evolving beyond simple LLM inference pipelines. A production-grade agent increasingly resembles a lightweight operating system: it manages processes, tools, sub-agents, persistent memory, context windows, permissions, sandboxes, and long-running execution.

The architectural shift can be summarized as:

[Stateless LLM]
       ↓
[Q&A Bot]
       ↓
[ReAct Loop]
       ↓
[Structured Tool Calling]
       ↓
[Layered Memory]
       ↓
[Agent Harness]

Each stage addresses a limitation of the previous one. What begins as a simple function such as answer = LLM(question) eventually becomes a persistent execution environment capable of planning, acting, observing, recovering, and delegating work.

This evolution has produced several distinct design philosophies. Pi emphasizes minimalism and token efficiency, OpenCode focuses on event-driven state and multi-client observability, Codex prioritizes security and controlled execution, while Hermes focuses on persistent learning and self-improvement.

🧬 The Six-Stage Evolution of the Agent Stack

#

The modern agent runtime did not emerge as a single architectural leap. It developed incrementally as AI applications encountered increasingly complex execution requirements.

1. Raw LLM: Stateless Token Mapping

#

The original architecture is essentially a pure function:

answer = LLM(question)

The model receives an input sequence and produces an output sequence. It has no inherent knowledge of previous interactions, external system state, or execution results unless that information is explicitly included in the input context.

The architecture can therefore be viewed as:

Input Tokens → Model → Output Tokens

This is efficient but fundamentally stateless.

2. Q&A Bot: Context Assembly

#

The next stage introduces a context-management layer:

system_prompt
      +
conversation_history
      +
user_prompt
      ↓
    LLM

The application reconstructs a temporary working state for every model invocation.

This creates the concept of Working Memory: the subset of information currently placed inside the model’s context window.

However, the model still cannot independently interact with external systems or execute actions.

3. ReAct Loop: Model-Driven Execution

#

The ReAct pattern turns a model call into an iterative state machine:

Model
  ↓
Action
  ↓
Environment
  ↓
Observation
  ↓
Model
  ↺

Conceptually, the runtime becomes:

while not finished:
    action = model(state)
    observation = environment.execute(action)
    state = update(state, observation)

This is an important architectural transition. The agent is no longer simply generating an answer; it is participating in an execution loop.

4. Structured Tool Calling

#

Early agents often relied on string parsing or regular expressions to interpret model-generated commands.

Structured tool calling replaces this fragile mechanism with explicit tool schemas and deterministic routing:

LLM
 ↓
Tool Call
 ↓
Tool Router
 ↓
Tool Execution
 ↓
Structured Tool Result
 ↓
LLM

The runtime can now enforce:

• Tool schemas
• Argument validation
• Permission checks
• Execution policies
• Structured result messages
• Tool-specific error handling

This makes agent behavior significantly easier to control and observe.

5. Layered Memory Architecture

#

As agents operate across longer sessions, a single context window becomes insufficient.

Modern runtimes increasingly separate memory into multiple layers:

┌──────────────────────────┐
│ Working Memory           │
│ Current context window   │
└────────────┬─────────────┘
             │
┌────────────▼─────────────┐
│ Long-Term Memory         │
│                          │
│ • Procedural Skills      │
│ • Semantic Facts         │
│ • Episodic Events        │
└──────────────────────────┘

This distinction allows the agent to retain useful information without continuously injecting an entire historical record into every model call.

Procedural knowledge may be stored in reusable files such as SKILL.md, while semantic facts and episodic events can be persisted independently from the active working context.

6. Agent Harness: The Full Execution Environment

#

The final stage adds the infrastructure required for reliable, long-running agent execution.

An Agent Harness typically manages:

• Session persistence
• Process execution
• Tool routing
• Permission boundaries
• Sandboxing
• Context compaction
• Memory persistence
• Error recovery
• Sub-agent delegation
• Long-running task state

At this point, the LLM is only one component of the overall system.

The architecture becomes:

                  ┌───────────────┐
                  │      LLM      │
                  └───────┬───────┘
                          │
              ┌───────────▼───────────┐
              │     Agent Harness      │
              ├────────────────────────┤
              │ Context Management      │
              │ Tool Routing            │
              │ Memory                  │
              │ Process Execution       │
              │ Permissions             │
              │ Sandboxing              │
              │ Sub-Agent Management    │
              │ Session Persistence     │
              └────────────────────────┘

The harness effectively becomes the operating environment surrounding the model.

🏗️ Four Architectural Philosophies

#

Different agent runtimes optimize for different problems rather than converging on a single universal architecture.

The differences are architectural rather than merely implementation-level. Each runtime makes a different trade-off between simplicity, observability, security, and adaptation.

⚡ Pi: The Minimalist Agent Harness

#

Pi takes the position that an agent runtime should introduce as little overhead as possible.

The Harness Tax

#

Benchmark results cited for Pi suggest that the same underlying LLM can achieve strong task-completion rates while consuming substantially fewer tokens and reducing execution cost.

Reported results include a 66.7% success rate with a median cost of approximately $0.012 in the referenced Composio and Databricks evaluations.

The architectural principle is straightforward:

Give the model only the infrastructure it actually needs.

Working-Set Optimization

#

Pi keeps its default tool surface deliberately small:

read
write
edit
bash

A Resource Loader and Session Manager help control how much information enters the model’s context.

Reducing irrelevant context has two effects:

1. Fewer tokens are processed on each turn.
2. The model spends less reasoning capacity navigating unnecessary information.

The result is a smaller and more focused execution loop.

Trade-Off: Limited Isolation

#

The minimalist approach also creates a security trade-off.

Pi does not provide the same level of built-in sandboxing and permission boundaries as security-oriented runtimes. In environments where tool execution can affect sensitive files or systems, stronger isolation may need to come from containers or external execution boundaries.

🔄 OpenCode: Event-Sourced Agent State

#

OpenCode takes almost the opposite approach. Instead of minimizing runtime state, it makes execution state highly explicit and observable.

Granular Session Events

#

Assistant activity is decomposed into discrete event types such as:

Reasoning
Text
Tool
Step Start
Step Finish
Patch
Compaction

These events can be projected into persistent storage such as SQLite.

Rather than treating a model response as one opaque block, the runtime exposes the individual state transitions that produced it.

Agent Profiles

#

OpenCode can apply specialized profiles to the same underlying execution loop.

Examples include:

• Build
• Plan
• Explore
• Background compaction
• Summary agents

This allows the runtime to separate different task behaviors without necessarily creating completely independent architectures.

Multi-Client State

#

One of OpenCode’s important architectural properties is its shared state model.

Different interfaces can consume the same underlying execution stream:

             ┌── TUI
             │
Agent State ─┼── Web
             │
             ├── Desktop
             │
             └── SDK

This means the user interface becomes a client of the runtime state rather than the owner of that state.

If a client disconnects, the underlying task can continue without losing its execution history.

🔐 Codex: Security, Threads, and Long-Running Execution

#

Codex places much greater emphasis on execution boundaries and task lifecycle management.

Thread, Turn, and Item

#

Long-running work is represented through a hierarchical task model:

Thread
  └── Turn
       └── Item

This structure allows the runtime to reason about individual execution units rather than treating an entire conversation as one undifferentiated stream.

Operations can include:

Start
Resume
Fork
Interrupt
Steer

This becomes especially important when an agent is working on a task that spans many model calls and tool executions.

Dual Security Boundaries

#

Codex separates two security concepts that are often conflated:

User Approval
      +
Sandbox Policy

Approval Policy determines whether the user must explicitly authorize an action.

Sandbox Policy determines what the process is technically allowed to access, regardless of whether the user approves the action.

For example, approval may allow an operation to proceed while the sandbox still restricts:

• File-system paths
• Network access
• OS privileges
• Process capabilities

This separation provides a stronger security model than relying on user confirmation alone.

Sub-Agent Forking

#

Codex can also create child agents as separate execution threads.

Conceptually:

Parent Thread
      │
      ├── Child Thread A
      │
      ├── Child Thread B
      │
      └── Child Thread C

Child threads can inherit or snapshot relevant context while maintaining independent execution state.

This helps prevent concurrent tasks from contaminating one another’s working context.

🧠 Hermes: The Self-Evolving Runtime

#

Hermes focuses on a different limitation: an agent may complete thousands of tasks without necessarily becoming better at future tasks.

Its architecture therefore extends memory beyond simple retrieval.

Beyond Memory Retrieval

#

A persistent learning system needs to distinguish between different types of historical information:

Session History
      ↓
Knowledge Extraction
      ↓
┌─────────────────────────┐
│ Stable Facts            │
│ Reusable Procedures     │
│ Successful Workflows    │
│ Obsolete Patterns       │
└─────────────────────────┘
      ↓
Future Agent Sessions

The objective is not to replay old conversations. It is to extract information that remains useful after the original session has ended.

Procedural Learning

#

Hermes can transform historical experience into reusable Skills and procedural files.

This creates a feedback loop:

Execute Task
    ↓
Record Session
    ↓
Review Outcome
    ↓
Extract Knowledge
    ↓
Create / Update Skill
    ↓
Apply Skill to Future Task

The runtime therefore becomes an adaptive system rather than a static orchestration layer.

PAST-Bench Results

#

The referenced PAST-Bench evaluation positions Hermes as a strong performer in memory retention, procedural reuse, and information updating.

Reported results include a +0.13 overall score gain and a 0.64 mechanism score, highlighting the potential value of persistent procedural learning.

The broader architectural idea is more important than the benchmark itself: long-term agent improvement requires mechanisms for deciding what experience should become reusable knowledge.

🧭 Agent vs. Harness: Where Responsibility Belongs

#

One of the most important distinctions in modern agent architecture is the boundary between the model and the runtime.

The agent model primarily determines:

What should I do next?

The harness determines:

Under what conditions may I do it?
What context should I see?
What tools are available?
What permissions do I have?
How is state persisted?
What happens if execution fails?
How do I recover?

This separation is becoming increasingly important as models gain stronger planning and tool-use capabilities.

A more capable model does not eliminate the need for infrastructure. It increases the importance of infrastructure because more capable agents can perform more consequential actions.

📈 Why Agent Harnesses Matter More as Models Improve

#

It may initially seem that better frontier models should make agent frameworks less important.

In practice, the opposite can happen.

As model capability increases:

Better Model
     ↓
More Complex Tasks
     ↓
More Tool Calls
     ↓
Longer Execution Horizons
     ↓
Greater State + Security Requirements

A simple model may need only a prompt and a response.

An autonomous engineering agent may need:

• Persistent sessions
• Hundreds of tool interactions
• Multiple processes
• Large codebases
• Context compaction
• Recovery after failures
• Parallel sub-agents
• Fine-grained permissions
• Sandboxed execution
• Long-term procedural memory

The model becomes more capable, but the surrounding runtime must become more sophisticated to make that capability usable and controllable.

🔭 The Emerging Agent Runtime Stack

#

The evolution can therefore be summarized as a progression from inference to infrastructure:

LLM
 │
 ├── Context Assembly
 │
 ├── ReAct / Execution Loop
 │
 ├── Structured Tools
 │
 ├── Memory
 │
 ├── Session Persistence
 │
 ├── Process Management
 │
 ├── Security / Sandbox
 │
 ├── Multi-Agent Coordination
 │
 └── Self-Improvement
        ↓
   Agent Harness

Pi, OpenCode, Codex, and Hermes represent different points in this design space.

• Pi demonstrates how far an agent can go with a deliberately small runtime.
• OpenCode treats execution state as an observable event stream.
• Codex emphasizes secure, recoverable, long-running execution.
• Hermes extends the runtime toward persistent procedural learning.

The common trend is clear: AI agents are becoming runtime systems rather than isolated model calls.

As frontier models continue to improve, the competitive advantage may increasingly shift from the model invocation itself to the infrastructure surrounding it—how efficiently context is managed, how safely tools are executed, how reliably state is recovered, and how effectively experience is converted into reusable knowledge.

The future of agent engineering is therefore not simply about building a better LLM call. It is about building the harness that allows an increasingly capable model to operate reliably in the real world.