This series is a complete 2027 revision of the "Enterprise Chatbot Strategy" originally written in 2021. Part 3 covers RAG (Retrieval-Augmented Generation) and AI Agents — the two technologies that turn a general-purpose LLM into a practical enterprise AI.
RAG and AI Agents — Activating Intelligent Service (this article)
Why RAG? — Why LLMs alone are not enough for enterprise use, and what RAG solves
RAG Architecture — Chunking, embedding, vector DB, retrieval, generation — step by step
From Chatbot to AI Agent — Function Calling, MCP, and the autonomy spectrum
Enterprise Design Choices — Human-in-the-loop, guardrails, and what to watch out for
Prologue
In 2021, the technical challenge was natural language understanding: how do you reliably parse user intent from text? NLU engines, intent classification, entity extraction — this was the frontier. The 2021 series covered these in detail in Parts 3 and 4.
In 2027, LLMs have solved the understanding problem. GPT-4o, Claude 3.5, Gemini 2.0 understand intent with near-human accuracy without special training. The new frontier is different: how do you connect an LLM to enterprise knowledge and enterprise systems?
The two technologies answering that question are RAG and AI Agents. This article explains both — non-technically enough for decision-makers, accurately enough for engineers.
1. The Three Gaps LLMs Can't Fill Alone
LLMs like GPT-4o have broad general knowledge — but three structural gaps make them unreliable for enterprise use without augmentation.
LLMs are trained on data up to a certain date and have no knowledge of events, regulations, or prices after that cutoff. For industries where documents change constantly — shipping regulations, safety standards, port procedures — this is a critical flaw.
An LLM trained on the public internet knows nothing about your company's internal policies, product specifications, contract terms, or organizational procedures. It cannot answer "what is our SLA for this customer?" or "what does our deck maintenance standard require?"
When an LLM lacks knowledge, it sometimes generates plausible-sounding but factually wrong answers — "hallucinations." In high-stakes domains like maritime safety or financial compliance, a hallucinated regulation or procedure number can be dangerous.
Retrieval-Augmented Generation (RAG) connects an LLM to an external, up-to-date knowledge base. Instead of relying solely on training data, the model retrieves relevant documents at query time and grounds its answer in them — eliminating the cutoff, adding enterprise specificity, and dramatically reducing hallucination.
2. RAG Architecture — Step by Step
RAG consists of two phases: Indexing (done once, then updated) and Retrieval + Generation (done at each user query). Here's how each step works.
Gather all relevant enterprise documents: internal policies, product manuals, regulations, FAQs, contracts. PDFs, Word files, web pages — all supported.
Documents are split into small, semantically coherent chunks (typically 200–500 words). Chunk size is a key design variable — too large and retrieval loses precision; too small and context is lost.
Each chunk is converted to a numerical vector (embedding) by an embedding model (OpenAI text-embedding-3, Cohere Embed, etc.). Semantically similar text produces similar vectors — enabling semantic search beyond keyword matching.
Vectors are stored in a specialized database optimized for fast similarity search (Pinecone, Weaviate, Qdrant, pgvector). The index is updated whenever new documents are added.
The user's question is converted to a vector using the same embedding model used during indexing.
The vector DB finds the top-k most semantically similar chunks (typically k=3–10). These are the candidate source passages.
Retrieved chunks are injected into the prompt sent to the LLM: "Based only on the following documents, answer the user's question." The LLM generates a grounded answer using the retrieved context.
The answer includes source references — which document, which section. Users can trace every answer back to its origin, enabling trust and auditability.
3. From Chatbot to AI Agent — The Critical Distinction
A chatbot — even an LLM-powered one — answers questions. An AI agent answers questions and takes actions. This is the leap that changes enterprise AI from a search tool into a workflow automation platform.
| Dimension | Chatbot / RAG | AI Agent |
|---|---|---|
| Output type | Text answer | Text answer + executed actions |
| Tool access | None / read-only | APIs, databases, code execution, file systems |
| Task length | Single-turn exchange | Multi-step autonomous workflows |
| Memory | Session context only | Long-term memory + task history |
| Risk | Incorrect information | Incorrect information + incorrect actions |
4. How AI Agents Take Actions — Function Calling and MCP
AI agents gain the ability to act through two mechanisms.
Introduced by OpenAI in 2023 and now standard across all major LLM APIs. You define a set of available functions (tools) with their parameters. The LLM decides when and how to call them — to search a database, calculate something, send an email, or update a record — and returns the result for the next step. Think of it as giving the LLM a set of buttons it can press.
Anthropic's open standard released in late 2024. MCP defines a standardized protocol for connecting LLMs to external systems — file systems, databases, APIs, web browsers, code execution environments. The key advance: instead of writing custom integrations per LLM and per system, MCP provides one universal interface. By 2027, thousands of pre-built MCP connectors exist for major enterprise tools (Salesforce, SAP, Google Workspace, Microsoft 365, GitHub, and more).
Enterprise implication: an AI agent built with MCP can switch between LLM providers without rewiring integrations — avoiding vendor lock-in.
5. The Autonomy Spectrum — How Much to Trust the Agent
AI agent autonomy exists on a spectrum. Enterprise decisions should explicitly place each agent use case on this spectrum — not leave it undefined.
AI provides recommendation; human makes all decisions and takes all actions. Example: draft email suggested, human edits and sends.
AI executes routine steps autonomously; pauses at defined checkpoints for human approval before consequential actions. Example: auto-drafts purchase order, human approves before submission.
AI operates autonomously; humans monitor outputs and receive exception alerts. Human approval only needed when anomalies detected.
AI executes complete workflows without human checkpoints. Appropriate only for low-stakes, fully auditable, easily reversible tasks. Not recommended for enterprise use cases without extensive testing and legal review.
For most enterprise deployments in 2027, Human-in-the-Loop is the right default. The performance gap between HITL and full autonomy is narrowing rapidly, but the accountability gap remains wide. Starting at Level 2 and progressing to Level 3 as trust is established is the sustainable path.
Regulation search (RAG): SOLAS, MARPOL, ISM Code, STCW — these are vast, complex, and continuously updated. A RAG system indexing flag state regulations, classification rules, and port state control requirements could answer complex compliance questions instantly, with full citations. DNV and Bureau Veritas are actively developing such systems.
Vessel inspection agent (AI Agent, Level 2): An AI agent connected to inspection databases could pre-populate deficiency reports from sensor data, cross-reference past findings, draft corrective action plans — and pause for chief engineer approval before submission to authorities.
Offline constraint: RAG requires connectivity to a vector database. At sea, index the most critical document set on-vessel for local retrieval, syncing updates in port. Design AI agents for graceful degradation when satellite connectivity is limited.
In 2021, the technology gap was the main obstacle. Natural language understanding was imperfect, chatbot platforms were expensive, and integration was painful. In 2027, the technology barriers have largely fallen. RAG and AI agents are well-documented, widely available, and deployable by competent engineering teams.
The remaining obstacle is organizational: data quality, governance frameworks, autonomy boundaries, human training, and regulatory compliance. Part 4 covers the other half of that equation — how LLMs actually work, why they fail, and how to manage performance and ethics over time.
RAG and AI Agents — Activating Intelligent Service (this article)
⚓ Join the ShipPaulJobs Community
Join →
Comments
Post a Comment