A living book about agentic workflows, agent orchestration, and agentic scaffolding
This chapter identifies the scaffolding layers that make agentic workflows reliable, covering tool access, context management, execution environments, and communication protocols. It explains how to balance flexibility with safety controls, ensuring agents can accomplish their tasks without causing unintended harm. Finally, it maps scaffolding decisions to operational risks, helping you understand which architectural choices matter most for your use case.
Agentic scaffolding is the infrastructure, frameworks, and patterns that enable agents to operate effectively. Just as scaffolding supports construction workers, agentic scaffolding provides the foundation for agent-driven development.
Agents need controlled access to tools and APIs.
class ToolRegistry:
"""Registry of tools available to agents"""
def __init__(self):
self._tools = {}
def register_tool(self, name, tool, permissions=None):
"""Register a tool with optional permission constraints"""
self._tools[name] = {
'tool': tool,
'permissions': permissions or []
}
def get_tool(self, name, agent_id):
"""Get tool if agent has permission"""
tool_config = self._tools.get(name)
if not tool_config:
raise ValueError(f"Tool {name} not found")
if self._check_permissions(agent_id, tool_config['permissions']):
return tool_config['tool']
raise PermissionError(f"Agent {agent_id} lacks permission for {name}")
Maintain and share context between agent invocations.
class AgentContext:
"""Manages context for agent execution"""
def __init__(self):
self.memory = {}
self.history = []
def store(self, key, value):
"""Store information in context"""
self.memory[key] = value
self.history.append({
'action': 'store',
'key': key,
'timestamp': datetime.now()
})
def retrieve(self, key):
"""Retrieve information from context"""
return self.memory.get(key)
def get_history(self):
"""Get execution history"""
return self.history
Provide safe, isolated environments for agent execution.
# Docker-based agent environment
FROM python:3.11-slim
# Install dependencies
RUN pip install langchain openai requests
# Set up workspace
WORKDIR /agent_workspace
# Security: Run as non-root user
RUN useradd -m agent
USER agent
# Entry point for agent execution
ENTRYPOINT ["python", "agent_runner.py"]
Production agentic workflows require safe execution of potentially untrusted agent-generated code without exposing credentials, allowing unrestricted network access, or risking the host system. The isolation strategy you choose determines the security boundaries and operational characteristics of your agent infrastructure.
Different isolation technologies offer varying levels of security, performance, and complexity. Understanding these trade-offs helps you choose the right approach for your use case.
Process-level isolation is the simplest approach, running agents as separate operating system processes with restricted permissions. This provides basic separation but shares the kernel and much of the system state with other processes. A vulnerability in the kernel or a privilege escalation exploit can compromise the entire system. Use this for trusted code in low-risk environments where simplicity matters more than strong isolation.
Container isolation uses Linux kernel features like namespaces and cgroups to create isolated execution contexts. Docker and Podman implement this approach, providing filesystem isolation, network isolation, and resource limits. Containers share the host kernel, which means kernel vulnerabilities affect all containers. They boot quickly (seconds) and have low overhead, making them suitable for many agentic workflows. Use containers when you need better isolation than processes but can accept shared kernel risks.
Full virtualization runs a complete operating system inside a hypervisor, providing the strongest isolation at the cost of higher overhead. QEMU, VirtualBox, and VMware implement full virtualization. Each VM has its own kernel, eliminating shared kernel vulnerabilities. Boot times are slower (tens of seconds to minutes) and resource overhead is higher. Use full VMs when security requirements justify the performance cost, such as when executing code from untrusted sources or handling sensitive data.
MicroVMs combine the strong isolation of VMs with the performance characteristics of containers. Firecracker (used by AWS Lambda) and Cloud Hypervisor implement this approach, booting minimal Linux kernels in under a second while maintaining kernel-level isolation. MicroVMs use hardware virtualization but minimize guest OS overhead, providing a practical balance for production agent workloads. Use microVMs when you need strong isolation without sacrificing the rapid iteration cycles that agentic workflows require.
| Technology | Boot Time | Isolation | Overhead | Use Case |
|---|---|---|---|---|
| Processes | Instant | Weak | Minimal | Trusted code, low risk |
| Containers | 1-5 seconds | Moderate | Low | Most agent workflows |
| MicroVMs | <1 second | Strong | Low-Medium | High-security agents |
| Full VMs | 30-60 seconds | Strongest | High | Maximum isolation |
Agents frequently need credentials to call external APIs—language model providers, code repositories, cloud services. Exposing these secrets to the agent execution environment creates risk. If the agent is compromised or generates malicious code, credentials can be exfiltrated. Different secret management patterns offer varying levels of security.
Environment variables are the simplest approach, injecting secrets as environment variables that agent code reads at runtime. This is easy to implement and widely supported but offers minimal protection. Any code running in the environment can access these variables, and they may appear in process listings, logs, or error messages. Use this only for non-sensitive credentials in trusted environments.
# Simple but insecure: secrets visible in environment
import os
api_key = os.environ.get('API_KEY')
# If agent code is compromised, api_key is directly accessible
Sealed secrets use encryption and access control systems like Kubernetes secrets or HashiCorp Vault to protect credentials at rest and in transit. Secrets are decrypted only when needed and only by authorized agents. This prevents static credential exposure but still requires the decrypted secret to exist in the agent’s memory space. Use this when you need better protection than environment variables but can accept in-memory credential presence.
# Better: fetch secrets from secure store
from vault_client import VaultClient
vault = VaultClient(token=os.environ.get('VAULT_TOKEN'))
api_key = vault.get_secret('api_keys/openai')
# Secret is encrypted until fetched, but still in memory
Network-layer injection provides the strongest protection by keeping credentials entirely outside the agent execution environment. A transparent proxy intercepts outbound API calls from the agent and injects real credentials at the network layer. The agent sees and uses a placeholder token, but actual API calls work seamlessly because the proxy rewrites requests in flight. If the agent is compromised, the attacker only obtains the placeholder, which is useless outside the sandboxed environment.
# Most secure: agent never sees real credentials
api_key = "placeholder_token_12345" # Not the real secret
client = OpenAI(api_key=api_key)
# Proxy intercepts this request:
# - Sees placeholder token in Authorization header
# - Replaces it with real credential
# - Forwards to actual API endpoint
# - Returns response to agent
response = client.chat.completions.create(...)
This pattern requires infrastructure support—a MITM proxy with vsock or similar communication channel—but provides defense in depth. Even if agent code is fully compromised, credentials remain protected. Use this for high-security environments or when executing untrusted agent code.
Secret rotation is essential regardless of injection method. Credentials should expire and rotate regularly, limiting the window of exposure if a secret is compromised. Automated rotation with short-lived tokens (hours to days rather than months) reduces risk without requiring manual intervention.
Unrestricted network access allows agents to exfiltrate data, call arbitrary APIs, or participate in distributed attacks. Default-deny networking with explicit allowlisting provides control without breaking legitimate functionality.
Default-deny networking blocks all outbound connections unless explicitly permitted. This prevents agents from reaching unexpected endpoints. Implement this at the firewall, container network policy, or virtual network level. Agents can only call services you have explicitly authorized.
Explicit allowlisting per host defines which external services each agent workflow can access. A code review agent might need GitHub API and language model APIs, but not database access. A documentation agent might need only static site generators and file storage. Granular policies limit blast radius when agents behave unexpectedly.
# Example network policy for agent workspace
agent_policies:
code_review_agent:
allowed_hosts:
- api.github.com
- api.openai.com
- api.anthropic.com
blocked_hosts:
- internal-database.corp
- admin-panel.corp
documentation_agent:
allowed_hosts:
- api.openai.com
- storage.googleapis.com
blocked_hosts:
- "*" # default deny
Egress filtering logs and monitors agent network activity, providing visibility into what agents actually do. Even with allowlisting, tracking connection attempts helps detect anomalies. If an agent suddenly attempts connections to unexpected hosts, this indicates potential compromise or unintended behavior.
Monitoring and logging complement filtering by recording which APIs agents call, how often, and with what results. This telemetry helps debug agent behavior and detect security issues early. Pattern-based alerting can flag unusual activity for human review.
To make these patterns concrete, consider Matchlock (https://github.com/jingkaihe/matchlock), an open-source CLI tool that combines microVMs with transparent secret injection. Matchlock has matured since its February 2026 debut, shipping Go and Python SDKs for embedding sandboxes directly in applications, and gaining community recognition as a developer-first, cross-platform option for agent sandboxing.
Problem addressed: Agent workflows need to execute potentially untrusted code while calling authenticated APIs. Traditional approaches either expose credentials to the agent (security risk) or require complex credential management (operational burden).
Solution architecture: Matchlock runs agents in ephemeral Firecracker microVMs (Linux) or Virtualization.framework (macOS) with:
The agent code looks normal—it imports libraries, reads a placeholder API key, and makes API calls. But the execution environment ensures credentials never enter the VM and network access is tightly controlled.
# Agent code running inside Matchlock microVM
import openai
# This is a placeholder token, not the real credential
client = openai.OpenAI(api_key="matchlock_placeholder")
# Proxy intercepts this call and injects the real API key
response = client.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": "Hello"}]
)
# Agent receives response as if it had used real credentials
print(response.choices[0].message.content)
Comparison with alternatives:
When to use each approach:
Use containers (Docker, Podman) for trusted agent code in controlled environments where convenience and ecosystem maturity matter. Use microVMs (Firecracker, Matchlock) when you need strong isolation without sacrificing rapid iteration, especially for executing user-provided or LLM-generated code. Use full VMs when security requirements are paramount and boot time is less critical, such as for isolated compliance workloads. Use process-level isolation only for prototyping or fully trusted code where security is not a concern.
The architecture of transparent proxying plus ephemeral environments provides a reference pattern for high-security agent scaffolding, applicable beyond any specific tool implementation.
Standardize how agents communicate.
interface AgentMessage {
id: string;
sender: string;
recipient: string;
type: 'task' | 'result' | 'error' | 'query';
payload: any;
timestamp: Date;
metadata?: Record<string, any>;
}
class MessageBus {
async send(message: AgentMessage): Promise<void> {
// Route message to recipient
}
async subscribe(agentId: string, handler: MessageHandler): Promise<void> {
// Subscribe agent to messages
}
}
Enable agents to combine tools effectively.
class ComposableTool:
"""Base class for composable tools"""
def __init__(self, name, func, inputs, outputs):
self.name = name
self.func = func
self.inputs = inputs
self.outputs = outputs
def compose_with(self, other_tool):
"""Compose this tool with another"""
if self.outputs & other_tool.inputs:
return CompositeTool([self, other_tool])
raise ValueError("Tools cannot be composed - incompatible inputs/outputs")
def execute(self, **kwargs):
return self.func(**kwargs)
# Usage
read_file = ComposableTool('read_file', read_func, set(), {'content'})
analyze_code = ComposableTool('analyze', analyze_func, {'content'}, {'issues'})
pipeline = read_file.compose_with(analyze_code)
Organize reusable agent capabilities.
# skills/code_review.py
class CodeReviewSkill:
"""Skill for reviewing code changes"""
def __init__(self, llm):
self.llm = llm
self.tools = ['git_diff', 'static_analysis', 'test_runner']
async def review_pull_request(self, pr_number):
"""Review a pull request"""
diff = await self.get_diff(pr_number)
issues = await self.analyze(diff)
tests = await self.run_tests()
return self.create_review(issues, tests)
# ... implementation details
# skills/__init__.py
from .code_review import CodeReviewSkill
from .documentation import DocumentationSkill
from .testing import TestingSkill
__all__ = ['CodeReviewSkill', 'DocumentationSkill', 'TestingSkill']
Manage computational resources efficiently.
class ResourceManager:
"""Manages resources for agent execution"""
def __init__(self, max_concurrent=5, timeout=300):
self.max_concurrent = max_concurrent
self.timeout = timeout
self.active_agents = {}
self.semaphore = asyncio.Semaphore(max_concurrent)
async def execute_agent(self, agent_id, task):
"""Execute agent with resource limits"""
async with self.semaphore:
try:
async with timeout(self.timeout):
result = await self._run_agent(agent_id, task)
return result
except asyncio.TimeoutError:
self._cleanup_agent(agent_id)
raise AgentTimeoutError(f"Agent {agent_id} timed out")
Monitor and debug agent behavior.
class AgentObserver:
"""Observes and logs agent behavior"""
def __init__(self):
self.logger = logging.getLogger('agent_observer')
self.metrics = {}
def log_execution(self, agent_id, task, result, duration):
"""Log agent execution"""
self.logger.info(f"Agent {agent_id} executed {task} in {duration}s")
self._update_metrics(agent_id, duration, result.success)
def get_metrics(self, agent_id):
"""Get performance metrics"""
return self.metrics.get(agent_id, {})
def export_trace(self, agent_id):
"""Export execution trace for debugging"""
return self._build_trace(agent_id)
# agent_config.yaml
agents:
content_writer:
type: specialized
tools: [markdown_editor, research_tool]
max_execution_time: 600
code_reviewer:
type: specialized
tools: [git, static_analyzer, test_runner]
max_execution_time: 300
orchestrator:
type: coordinator
tools: [task_queue, notification_service]
manages: [content_writer, code_reviewer]
Centralize tool access and management.
Provide starting points for common agent types.
# templates/base_agent.py
class BaseAgent(ABC):
"""Base template for all agents"""
def __init__(self, agent_id, config):
self.agent_id = agent_id
self.config = config
self.tools = self._load_tools()
self.context = AgentContext()
@abstractmethod
async def execute(self, task):
"""Execute the agent's main task"""
pass
def _load_tools(self):
"""Load tools from registry"""
return [get_tool(name) for name in self.config['tools']]
Build resilience into your scaffolding.
class ResilientAgent:
"""Agent with built-in error recovery"""
async def execute_with_recovery(self, task, max_retries=3):
"""Execute with automatic retry on failure"""
for attempt in range(max_retries):
try:
result = await self.execute(task)
return result
except RecoverableError as e:
if attempt < max_retries - 1:
await self._recover(e)
continue
raise
except Exception as e:
self._log_error(e)
raise
Warning: Sandboxing and permission boundaries are not optional. Treat every tool invocation as a least-privilege request and validate all side effects in a separate review step.
This book’s scaffolding includes several interconnected components.
GitHub Actions provides workflow orchestration, triggering agents in response to issues, pull requests, and schedules. Issue Templates provide structured input for suggestions, ensuring agents receive information in a consistent format they can parse reliably. Agent Scripts are Python scripts for content management that handle tasks like generating tables of contents and updating cross-references. Tool Access includes Git for version control, markdown processors for content transformation, and PDF generators for final output. State Management uses the Git repository itself as persistent state, with commits recording the history of changes. Communication flows through the GitHub API, which provides the coordination layer for all agent interactions.
In this repository, the scaffolding is implemented in concrete files:
scripts/check-links.pyscripts/build-combined-md.sh.github/workflows/issue-*.md.github/workflows/issue-*.lock.yml.github/workflows/pages.yml and .github/workflows/build-pdf.yml.github/workflows/check-links.yml, .github/workflows/check-external-links.yml, and .github/workflows/compile-workflows.yml.github/workflows/copilot-setup-steps.ymlWORKFLOW_PLAYBOOK.mdFor workflow semantics, see GitHub Agentic Workflows (GH-AW). For failure handling and validation strategy, see Common Failure Modes, Testing, and Fixes.
Start Simple. Build minimal scaffolding first and expand only as needed. Over-engineering early creates maintenance burden without corresponding benefit; let actual requirements drive complexity.
Security First. Implement permissions and isolation from the start, not as an afterthought. Retrofitting security into an existing architecture is far more difficult than designing it in from the beginning.
Observability. Log everything—you will need it for debugging. When agents behave unexpectedly, logs are often the only way to reconstruct what happened and why.
Version Control. Version your scaffolding alongside your agents. The two must evolve together, and tracking their relationship helps diagnose regressions.
Documentation. Document tools, APIs, and patterns clearly. Agents rely on this documentation to use scaffolding correctly, and humans need it to maintain the system.
Testing. Test your scaffolding independently of agents. This allows you to verify that infrastructure works correctly before introducing the additional variability of agent behaviour.
Over-engineering. Building scaffolding for hypothetical needs wastes time and creates complexity that obscures the actual architecture. Wait until a requirement is real before addressing it.
Tight Coupling. When agents depend heavily on specific scaffolding details, changes become risky and testing becomes difficult. Keep agents loosely coupled to scaffolding through well-defined interfaces.
Poor Error Handling. Agents encounter failures—network timeouts, API errors, unexpected input. Scaffolding that does not plan for these scenarios will leave agents stuck or produce corrupt output.
No Monitoring. You cannot improve what you cannot measure. Without visibility into how agents use scaffolding, you cannot identify bottlenecks or verify that changes help.
Ignoring Security. Security must be built in, not bolted on. Scaffolding that allows unrestricted tool access or does not validate inputs creates vulnerabilities that grow harder to fix over time.
Scaffolding provides the foundation for effective agent operation, enabling capabilities that agents could not achieve in isolation. Core components include tools for interacting with the environment, context for maintaining state across invocations, execution environments for safe isolated operation, and communication protocols for agent coordination. Patterns like tool composition and resource management improve scalability by letting you combine simple pieces into complex capabilities. Build incrementally, focusing on security and observability as primary concerns rather than afterthoughts. Good scaffolding makes agents more capable and easier to manage by providing reliable infrastructure they can depend on.