[EAAPL-SEC004] Secure Tool Invocation

Category: Security / Agentic AI Controls Sub-category: Tool & Function Call Security Version: 1.1 Maturity: Proven Tags: agent-security tool-calling function-calling human-in-the-loop permission-scoping audit-logging rate-limiting Regulatory Relevance: EU AI Act Art. 9 & 14, APRA CPS234, NIST AI RMF GOVERN 6.2, ISO 42001 §8.5

1. Executive Summary

Secure Tool Invocation defines the security architecture for AI agents that can call external tools, APIs, databases, or execute code. As organisations deploy agentic AI systems — coding assistants that run terminal commands, customer service bots that can query and update CRM records, AI workflows that send emails or make financial transactions — the attack surface moves from "what the AI says" to "what the AI does."

A single successful prompt injection in an agent with uncontrolled tool access can result in data exfiltration, financial transactions, communications sent to external parties, or destructive actions on production systems. This is not theoretical: documented incidents have demonstrated that agents with broad tool access can be manipulated to perform actions their operators never intended.

This pattern establishes five core controls: tool allow-listing (only approved tools are callable), permission scoping (each tool call uses the minimum permissions required), input/output validation for tool calls (tools receive only validated inputs and return validated outputs), human approval gates for high-risk operations, and comprehensive audit logging of every tool invocation. Together these controls make agent tool use auditable, reversible, and bounded — transforming agentic AI from an uncontrolled actor to a governed one.

2. Problem Statement

Business Problem

Organisations deploying AI agents with tool access are creating a new category of automated actor that can take real-world actions at machine speed. Without appropriate controls:

• An agent manipulated by prompt injection can exfiltrate customer data by calling a data export tool.
• An agent instructed to "help with finance" can make unintended financial transfers if it has access to payment APIs.
• An agent that can send emails can be manipulated to send phishing messages to customer lists.
• An agent with database write access can modify or delete production records.

Unlike a human employee who can recognise a suspicious instruction, an AI agent will attempt to execute whatever instruction appears most consistent with its current context — including adversarially crafted instructions.

Technical Problem

Modern AI frameworks (LangChain, LlamaIndex, OpenAI Assistants, Anthropic Tool Use) make it trivially easy to give an agent access to many tools. The default security posture is permissive: tools are made available to the agent, and the agent decides which to call and with what arguments. There is typically:

• No allow-listing of which tools can be called in which contexts.
• No input validation on tool arguments generated by the LLM.
• No output validation on data returned by tools (which may itself contain injection content).
• No human approval for high-risk operations.
• Minimal audit logging of tool calls.
• Credentials for tool APIs embedded in agent configuration with excessive permissions.

Symptoms

• Agent making unexpected API calls discovered in application logs.
• Credentials used by agents have broad permissions (admin roles, full database access).
• No centralised record of what actions agents have taken.
• Agents taking irreversible actions (sending emails, deleting records) without human review.
• Tool call arguments not validated — LLM-generated arguments passed directly to tools.

Cost of Inaction

3. Context

When to Apply

• Any AI agent that can call external APIs, execute code, query/modify databases, send communications, or interact with file systems.
• Multi-agent orchestration systems where one agent calls another.
• AI workflows that can take actions in external systems (CRM, ERP, cloud infrastructure).
• Coding assistants with terminal/IDE access.
• AI assistants with email, calendar, or communication platform access.

When NOT to Apply

• Pure read/retrieval agents (RAG-only) with no ability to take actions — apply simplified read-only version of tool validation.
• Agents calling only internal, non-sensitive information retrieval endpoints with no mutation capability.

Prerequisites

Industry Applicability

4. Architecture Overview

The Secure Tool Invocation pattern wraps every tool call in a security envelope: before the tool is called, the invocation is validated against policy; after the tool returns, the output is validated before being returned to the agent. Between these validation points, the tool executes with the minimum permissions necessary.

Tool Registry and Allow-Listing

The foundation of the pattern is a centralised tool registry that serves as the authoritative source of which tools exist, what they do, and who can call them. The registry stores: tool identifier and schema, security classification (READ / WRITE / DESTRUCTIVE / FINANCIAL / COMMUNICATION), permission requirements (what credentials are needed), allowed callers (which agents/applications may use this tool), rate limits per caller, and whether human approval is required.

When an agent requests a tool call, the invocation control layer checks the registry before forwarding the call. Tools not in the registry are rejected. Tools for which the calling agent is not authorised are rejected. This allow-list approach — as opposed to a deny-list — ensures that new tools cannot be called until explicitly registered and approved.

Permission Scoping

Each tool call uses the minimum credentials required for that specific operation. Rather than giving an agent a single omnibus credential (e.g., a database connection string with full admin access), credentials are scoped per operation:

• A read_customer_record tool receives a read-only database token scoped to the customers table.
• A create_support_ticket tool receives a write-scoped API key valid only for ticket creation, not ticket deletion.
• Credentials are short-lived (15–60 minutes) and generated at invocation time from a vault, not pre-configured in the agent.

This scoping ensures that even if the agent is manipulated into calling a tool with adversarial arguments, the tool's credential limits the damage to what that specific credential permits.

Input Validation for Tool Calls

LLM-generated tool arguments are untrusted. The LLM may produce arguments that are structurally correct (matching the tool's JSON schema) but semantically dangerous (e.g., a SQL query that accesses tables beyond the intended scope, a file path that traverses outside the permitted directory). Input validation applies:

• Schema validation: Tool arguments must conform to the JSON schema registered for the tool. Unexpected fields are stripped; required fields are enforced.
• Content validation: String arguments are inspected for injection patterns (SQL injection, path traversal, command injection) using the same techniques as EAAPL-SEC005.
• Scope validation: For tools that reference specific resources (user IDs, file paths, API endpoints), validation ensures the referenced resource is within the agent's authorised scope.

Human Approval Gates

Tool calls classified as DESTRUCTIVE, FINANCIAL above a threshold, or COMMUNICATION to external parties require synchronous human approval before execution. The approval workflow:

1. The invocation control layer intercepts the tool call.
2. An approval request is created in the ITSM/approval system with full context: which agent, what tool, what arguments, what triggered this call.
3. The agent is paused (tool call result is pending).
4. An authorised approver reviews and approves or rejects.
5. If approved, the tool call proceeds; if rejected, the agent receives an "operation not approved" result.
6. Approval timeouts (e.g., 30 minutes) result in automatic rejection and alert.

Output Validation

Tool outputs returned to the agent are also validated. This is critical for indirect injection: a malicious document retrieved by a search tool may contain instruction text designed to manipulate the agent's subsequent actions. Output validation applies:

• Schema validation: Output conforms to expected structure.
• Content inspection: Output inspected for instruction-like text (prompt injection fragments) before being returned to the agent context.
• Data classification: Output classified and labelled; data above the agent's clearance level is redacted or blocked.

5. Architecture Diagram

6. Components

7. Data Flow

Primary Flow

Error Flow

8. Security Considerations

Authentication & Authorisation

• Calling agent must authenticate to the Invocation Control Plane (mTLS or JWT from AI Gateway context).
• Each tool call uses a credential scoped to the minimum permission required for that specific operation — not the agent's overall credential.
• Human approvers authenticate through the organisation's IdP before approving tool calls.

Secrets Management

• No long-lived credentials embedded in agent configuration or code.
• All credentials retrieved at invocation time from Vault.
• Credentials expire automatically (15–60 min TTL).
• All credential access logged by Vault.

Data Classification

• Tool outputs are classified before being returned to the agent. If a tool output contains data classified above the agent's authorised level (e.g., CONFIDENTIAL data returned to a PUBLIC-level agent), the output is redacted and the incident is logged.
• Agents should only be granted access to tools that return data at or below their classification level.

Encryption

• All tool call traffic encrypted in transit (TLS 1.3).
• Audit log records encrypted at rest.
• Tool arguments in audit log stored as hashed representations if they may contain sensitive data.

OWASP LLM Top 10 Coverage

9. Governance Considerations

Responsible AI

• Every tool call that takes a real-world action is logged in the audit trail, enabling post-hoc accountability for all AI-initiated actions.
• Human approval gates ensure that the most consequential AI actions have a human decision point.

Model Risk Management

• Tool registry changes (adding new tools, expanding permissions) are subject to model risk review: what is the worst-case action the agent can now take? Is that acceptable?
• Regular review of audit logs to identify agents using tools in unexpected patterns.

Human Approval

• The human approval gate is a mandatory control for DESTRUCTIVE, FINANCIAL (above threshold), and COMMUNICATION tool classes.
• Approval decisions are logged with approver identity, timestamp, and stated rationale.
• Approval bypass (emergency override) is available only with break-glass credentials and requires dual approval.

Governance Artefacts

10. Operational Considerations

Monitoring

• Tool call rate per agent and per tool; approval queue depth and latency; tool error rates; blocked invocation rate (schema / injection / ACL); credential expiry events.

SLOs

Incident Management

• Agent calling unregistered tool → P2 security incident (possible prompt injection escalation attempt).
• Injection detected in tool output → P1 security incident (confirmed indirect injection attack).
• High-risk tool called without approval (bypass) → P1 security incident.
• Vault credential unavailable → P1 (all tool calls blocked until resolved).

DR

11. Cost Considerations

Cost Drivers

Indicative Cost Range

12. Trade-Off Analysis

Option Comparison

Architectural Tensions

13. Failure Modes

14. Regulatory Considerations

15. Reference Implementations

AWS

Azure

On-Premises

16. Related Patterns

17. Maturity Assessment

Overall Maturity: Proven

18. Revision History