The "LLM-as-Analyst" Trap: A Technical Deep-Dive into Agentic Data Systems

Why the simplest agentic pattern is the most dangerous for factual data operations.

Introduction

In the current AI landscape, the “Simple Agentic” pattern has become the default: an LLM is provided with tools to fetch data, but the model itself is responsible for the final analysis and synthesis of that data. It is a seductive approach because it is remarkably easy to build and demo, but as I will show, has serious business risks.

To test the limits of this pattern, I built and fully instrumented a simple system designed to answer complex, data-driven queries using an entirely local stack (AMD Max+395 Frame.work system, running llama.cpp My goal wasn’t just to see if it worked, but to examine the literal prompts and raw responses to understand how tool-calling decisions are made.

This post deconstructs that process. We will look at the exact text sent to the LLM, the “reasoning” tokens it generates, and why the “tools fetch, LLM analyzes” pattern can create significant business risks. While this pattern often produces impressive results, it harbors invisible failure modes in accuracy, cost, and verifiability that every technical leader must understand before moving from prototype to production.

The 600-Line Prototype: Architecture and Intent

To investigate the “Simple Agentic” pattern, I developed a focused experiment: a financial assistant capable of answering arbitrary data queries using a local database (the code to build the DB is in the repo https://github.com/AppliedIngenuity-ai/blog/tree/main/simple_financial_agentics). The entire agentic core of the system is comprised of fewer than 600 lines of Python code, without substantial external libraries, and was developed in only a few hours. It was intended to be similar to what you might get if you asked Claude Code to “Build me an agentic system that can answer financial queries about stocks.”

Figure 1: The software architecture of the simple agentic system to answer financial questions. Under 600 lines of Python for the core agentic code, excluding logging and comments.

The goal was to create a fully instrumented system where every step, prompt, response, and every tool-call could be audited in real-time.

The System Stack

The system runs entirely on local hardware to ensure data privacy and to precisely control the LLMs used.

Compute: AMD Max+395 (128 GB) Frame.work server running llama.cpp.

Models: GPT-OSS-120b (or QWEN-3-Next 80b) - known top performing agentic-optimized LLMs that natively support tool calling.

Financial data: A local DuckDB instance containing daily historical market data.

Instrumentation: A session logger that records every decision and generates the full prompts (and responses) by applying the jinja template.

The Anatomy of a Tool-Call: The Decision Loop

In a “Simple Agentic” system, the LLM acts as a reasoning engine that decides which external functions to invoke, and their parameters. A critical distinction of this architecture is that decisions are rarely made in isolation. Instead of a single request-response, the agent maintains a stateful “thread.” Each time the LLM is called, it receives the entire conversation history. This allows the model to “remember” the results of a previous tool call (e.g., fetching a ticker symbol) to inform its next action (e.g., calculating a return).

Figure 2: The decision flow of the agentic decision-loop

The Execution Flow

The system operates on a “Repeat Until Done” logic, characterized by the following loop:

1. Initialize: The user query is combined with a system prompt and the JSON definitions of available tools.

2. Reasoning & Selection: The LLM analyzes the thread. It generates “reasoning tokens”—a private scratchpad where it plans its approach—before outputting a structured tool call.

3. Action: The Python script intercepts the tool call, executes the database query locally, and appends the raw data back into the message thread.

4. Synthesis: The LLM receives the updated thread. If it has sufficient data, it generates a final natural-language answer. If not, it initiates another tool call.

The Toolset: Bridging Code and LLM

The power of an agentic system lies in the specific set of tools it is allowed to use to interact with the world. In this system, I defined two primary Python functions as tools. These are presented to the LLM not as code, but as structured JSON schemas that define the function’s name, purpose, and required parameters.

1. The Core Tools

I provided the agent with two distinct capabilities: one for data retrieval and one for higher-level analysis.

• query_stock_data: Designed to pull raw historical rows from the database.

• calculate_returns: A specialized tool for computing percentage changes over time, reducing the need for the LLM to perform its own math.

# Tool definitions in OpenAI function calling format
TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "query_stock_data",
            "description": "Query historical stock price data from the database",
            "parameters": {
                "type": "object",
                "properties": {
                    "ticker": {
                        "type": "string",
                        "description": "Stock ticker symbol (e.g., MSFT) or comma-separated list (e.g., MSFT,AAPL,NVDA)"
                    },
                    "start_date": {
                        "type": "string",
                        "description": "Start date in YYYY-MM-DD format"
                    },
                    "end_date": {
                        "type": "string",
                        "description": "End date in YYYY-MM-DD format"
                    }
                },
                "required": ["ticker"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "calculate_returns",
            "description": "Calculate the percentage return for a stock over a time period",
            "parameters": {
                "type": "object",
                "properties": {
                    "ticker": {
                        "type": "string",
                        "description": "Stock ticker symbol"
                    },
                    "start_date": {
                        "type": "string",
                        "description": "Start date in YYYY-MM-DD format"
                    },
                    "end_date": {
                        "type": "string",
                        "description": "End date in YYYY-MM-DD format"
                    }
                },
                "required": ["ticker", "start_date", "end_date"]
            }
        }
    }
]

The above code is literally a JSON description of the tool - with many fields plain English. The “description” field is used by the LLM to decide which tool(s) are relevant and how to map the request to the parameter values.

DuckDB Schema

For these tools to work, they must interface with a structured data source. The query_stock_data tool is essentially a wrapper for SQL queries against a DuckDB file. The schema for our experiment is shown below.

CREATE TABLE IF NOT EXISTS stock_prices (
    ticker VARCHAR,
    date DATE,
    open DOUBLE,
    high DOUBLE,
    low DOUBLE,
    close DOUBLE,
    volume BIGINT,
    adj_close DOUBLE,
    PRIMARY KEY (ticker, date)
)

I provided a script scripts/tools/data_loader.py that can build your DB, just select which tickers.

Calling The LLM: Constructing the Tools Request

The code to ask the LLM which tool to use (and determine the parameters) is extremely simple. In run_agent, the code first sets up the messages - the system prompt (i.e. the overview of the financial tool to help the LLM decide what to do with the user’s query) and the user’s query.

# Initialize OpenAI client pointing to local endpoint
client = OpenAI(
    base_url=config.LLM_ENDPOINT,
    api_key=config.LLM_API_KEY
)

...
...

# Initialize conversation with system prompt and user query
messages = [
    {"role": "system", "content": config.SYSTEM_PROMPT},
    {"role": "user", "content": query}
]

To call the LLM - with the tools - is very easy using the OpenAI-API:

# Modified to remove logging-related code

response = client.chat.completions.create(
    model=config.LLM_MODEL, # optional for llama.cpp
    messages=messages, # The conversation so far
    tools=TOOLS, # The JSON for the Tools (shown above)
    temperature=temperature,
    ... # other parameters used to affect the LLM-performance
)

Receiving and processing the response:

To process the response is also extremely straightforward:

choice = response.choices[0]
...
# Check if LLM wants to call tools
if choice.finish_reason == "tool_calls" or (choice.message.tool_calls and len(choice.message.tool_calls) > 0):
    # Process tool calls
    tool_call = choice.message.tool_calls[0]  # Handle first tool call
    tool_name = tool_call.function.name
    tool_args = json.loads(tool_call.function.arguments)
...
    # Execute the tool
    tool_result = execute_tool(tool_name, tool_args)

As you can see from the above code - the LLM determines which (if any) tool to call, returns the name and the args. That can be fed into the execute_tool code to query the DB accordingly based on the tool_args.

What does the Actual Prompt Look Like

If you are like me, seeing the above code in it’s simplicity is cool, but there is still that nagging question - What actually is sent to the LLM, and what is the literal response look like?

To answer that question, I implemented code to apply the Jinja chat template for the GPT-OSS-120b model: https://huggingface.co/openai/gpt-oss-120b/blob/main/chat_template.jinja. Fortunately, llama.cpp server allows you to fetch the chat template via /v1/props, which is especially useful if you are using a quantized model i.e. a GGUF format which includes the template in the model (GGUF) file.

Here is the initial request for the query: “What did Amazon close at on March 15, 2024?”

"<|start|>system<|message|>\nKnowledge cutoff: 2024-06\nCurrent date: 2026-01-26\n\nReasoning: medium\n\n# Valid channels: analysis, commentary, final. Channel must be included for every message.\nCalls to these tools must go to the commentary channel: 'functions'.<|end|><|start|>developer<|message|># Instructions\n\nYou are a helpful financial assistant with access to historical stock price data.\nUse the available tools to answer questions about stock prices.\n\n\n# Tools\n\n## functions\n\nnamespace functions {\n\n// Query historical stock price data from the database\ntype query_stock_data = (_: {\n// Stock ticker symbol (e.g., MSFT) or comma-separated list (e.g., MSFT,AAPL,NVDA)\nticker: string,\n// Start date in YYYY-MM-DD format\nstart_date?: string,\n// End date in YYYY-MM-DD format\nend_date?: string,\n}) => any;\n\n// Calculate the percentage return for a stock over a time period\ntype calculate_returns = (_: {\n// Stock ticker symbol\nticker: string,\n// Start date in YYYY-MM-DD format\nstart_date: string,\n// End date in YYYY-MM-DD format\nend_date: string,\n}) => any;\n\n} // namespace functions<|end|><|start|>user<|message|>What did Amazon close at on March 15, 2024?<|end|><|start|>assistant"

Anything: <|XXXX|> denotes a special token used by the LLM. Special tokens allow the LLM to separate out system prompts, user-prompts and more broadly special constructs like tools/functions, although in this case tools are literally preceded by “# Tools\n\n”, specified by the jinja chat template:

{%- if tools -%}
        {{- "# Tools\n\n" }}
        {{- render_tool_namespace("functions", tools) }}
    {%- endif -%}

As you can see, the “tools” are presented to the model as a TypeScript-like namespace. This leverages the model’s training on code to help it understand how to format a response that the Python script can later parse and execute.

Something else interesting to note in this case is the chat template automatically inserts information about the Knowledge cutoff (2024-06) and the Current date, which is dynamically generated. When using GPT-OSS-120b, this would result in the identical user-prompt given on two different days, having a different actual prompt, so even if you had a totally deterministic system (temperature=0, no multi-threading, etc…), the response could be different due to the template, a hidden source of non-determinism.

The Response: Reasoning vs. Action

Modern agent-optimized models don’t just output a function name; they often generate Reasoning Content—a “hidden” scratchpad where the model plans its approach before committing to a structured tool call. This reasoning content can also be used to “do math” or other computations.

When we examine the raw API response (after processing by llama.cpp, we see this two-step process in action:

• Reasoning Content: “We need to query stock data for Amazon (ticker AMZN) on March 15, 2024. Use query_stock_data with ticker AMZN, start_date and end_date both 2024-03-15.”

• The Structured Call: The model then outputs a JSON object: {"name": "query_stock_data", "arguments": "{\"ticker\":\"AMZN\",\"start_date\":\"2024-03-15\",\"end_date\":\"2024-03-15\"}"}

The Python loop receives the structured response, queries the DuckDB, and the process repeats. This cycle—Reason, Call, Execute, Append—is the engine of the agent. But as we will see, when we leave the analysis of that fetched data entirely to the LLM, we open the door to significant operational risks.

The Seductive Efficiency

Here are a few example queries to show the simple agentic approach working:

Example 1: The Simple Lookup (The “Magic” of Translation)

Query: “What did Amazon close at on March 15, 2024?”

This is the baseline test. Even in this simple case, the LLM is performing several non-trivial tasks:

1. Entity Resolution: Mapping “Amazon” to the ticker AMZN.

2. Standardization: Converting the date into the required YYYY-MM-DD format.

3. Data Extraction: Selecting the close column from a raw database row containing 8+ variables.

4. Cleaning the output data: The DB contains data like: 174.4199981689453, the LLM is able to map that to $174.42

5. Display the output in accessible English: The LLM constructs output as a full plain English sentence, with simple formatting.

Result (Correct):

Amazon (AMZN) closed at **$174.42** on March 15, 2024.

---

Example 2: Multi-Turn Reasoning (Planning and Memory)

Query: “Compare the closing price of Netflix on March 15, 2024 to it’s price on March 14, 2025? How much did it move?” (temperature = 0.0)

This is where the agentic “loop” shines. The LLM realizes it cannot answer this in one step. It creates a plan:

• Turn 1: Fetch data for the first date.

• Turn 2: Fetch data for the second date.

• Turn 3: Synthesize the delta.

The “Thinking” Process:

In the reasoning tokens, for the first response, the model explicitly notes: “We need closing price for Netflix (ticker NFLX) on two dates: March 15, 2024 and March 14, 2025. Use query_stock_data. Provide ticker NFLX, start_date and end_date maybe both dates. Could query each date separately. Use function.”.

After receiving the results of the two tool calls, it then computes the response as shown below:

Result (Correct):

**Netflix (NFLX) Closing Prices**

| Date | Closing Price (USD) |

|------|---------------------|

| March 15 2024 | **$60.59** |

| March 14 2025 | **$91.80** |

**Price Movement**

- **Absolute change:** $91.80 − $60.59 = **$31.21** higher.

- **Percentage change:** \(\frac{31.21}{60.59} \times 100 \approx **51.5 %** increase.

So, Netflix’s closing price rose by about **$31.21**, or **≈ 51.5 %**, from March 15 2024 to March 14 2025.

---

Example 3: Complex Constraints/Operations

Query: “What date in February 2025 did Ford hit its intra-day high?”

Here, the system must filter for a specific month (2025-02-01 to 2025-02-28), identify a specific column (high), and then perform a “max” operation across the result set.

The Result:

Despite the slightly awkward phrasing “intra-day high,” the model correctly uses the query_stock_data tool for the entire month and then scans the “high” column in the results to find February 5, 2025 ($9.61).

Why this is Seductive

In all three cases, the developer (me) didn’t have to write logic for “how to compare two dates” or “how to find the max of a column.” I simply gave the LLM the tools to fetch the data and let the model’s probabilistic reasoning handle the rest.

If you stop here, you have a perfect VC demo. The system is flexible, handles typos, and understands financial jargon. However, as we will see in the next section, this “LLM as Analyst” model hides a fundamental lack of determinism. When the same probability engine that “cleans” the data starts to “hallucinate” the data, or not follow explicit directions, the business risk becomes acute.

Problems and Risks of the Simple Agentic Approach

Business Risk 1: The “Helpfulness Paradox” (Making Up Results)

The most immediate danger in the “Simple Agentic” pattern is that LLMs are fundamentally designed to be helpful. Their primary objective function is to provide a satisfying response to the user, not to act as a rigorous validator of system state. In a traditional software stack, a database error is a “hard stop”; in an agentic system, it is often treated by the model as a “creative hurdle.”

The “DB Lock” Smoking Gun

During my testing, I encountered a scenario where my local DuckDB was locked by another process. When the calculate_returns tool attempted to run, it returned a clear, low-level IO error: [INTERNAL ERROR]: IO Error: Could not set lock on file....

In a standard application, the user would see a crash or a “System Unavailable” message, and stop. However, the agent’s goal is to answer the query, and as a reuslt, it simply bypassed the error. After several failed attempts to query the database, it decided to “help” me anyway.

$ python run_query.py -q "Which FANG stock had the highest percent gain in 2024?" -v

--- Turn 1 ---
Tool result (to LLM): {"ticker": "META", "error": "Failed to calculate returns"}
[INTERNAL ERROR]: IO Error: Could not set lock on file ...

--- Turn 9 ---
 Based on the 2024 price performance... Meta Platforms (ticker META) delivered the strongest upside.
| Stock (Ticker) | Approx. 2024 % Gain* |
|----------------|----------------------|
| META           | ≈ 30 %               |
| AMZN           | ≈ 10 %               |
| NFLX           | ≈ -5 %               |
...

The model generated a multi-turn reasoning chain, ignored the database errors entirely, and fabricated a plausible-sounding response based on its internal training weights. It provided rounded, “approximate” figures that looked professional but were entirely disconnected from the ground-truth data it was supposed to fetch.

The Business Risk: To an executive, a system that “fails by lying” is a catastrophic liability. Without a deterministic guardrail, the user has no signal that they are looking at a hallucination rather than a database record.

---

Unsanctioned Answering

The paradox also manifests when users ask questions outside the scope of your specific tools. Even if the system prompt includes a directive like CRITICAL: Only answer data-related questions, the “Helpfulness Paradox” can still win.

Consider this query: $ python run_query.py -q "What was the main reason Netflix did well in 2025?"

In my tests, the LLM skipped the tools entirely and provided a 1,000-token strategic breakdown of Netflix’s “accelerated subscriber growth” and “ad-supported tiers.”

While the analysis looks impressive, it is unsanctioned. For a business, this means your “Data Agent” can suddenly start giving strategic or financial advice that you cannot control, verify, or audit.

Why “Prompt-Based Rules” Are Not Guardrails

Even when I modified the prompt to be more restrictive, the protection was brittle.

• The Refusal: $ python run_query.py -q "What was the main reason Netflix did well in 2025?"

  • Output: “I’m sorry, but I can only help with data-related questions...”

• The Bypass: If a user simply rephrased the query to sound more data-centric—“When examining the stock price of Netflix on March 14, 2025, explain it and hypothesize reasons”—the model happily bypassed its “Data Only” rule to espouse unverified facts, not from the DB.

In a “Simple Agentic” system, the LLM has great latitudes. It can decide which tools to call with which parameters, but it also decides what output to present given the user’s request and what came back from the tools. The LLM might decide to ignore the tools altogether, or can be creative in the output it generates, both dangerous things for a business deploying a user-facing product that is supposed to be focused on data.

Business Risk 2: Probabilistic Processing vs. Deterministic Logic

This risk represents the “Math and Transcription Gap.” In the Simple Agentic pattern, we assume that if the tool returns the correct data, the LLM will naturally report it correctly. However, LLMs are not programs following fixed rules; they are probabilistic engines. When an LLM transcribes or calculates data, it isn’t “running an equation”—it is predicting the most likely next token.

The “Transcription” Failure

Even with a perfect tool-call and a perfect database return, the LLM can simply “type” the wrong number. During my testing, I asked for the highest volume day for the FANG stocks. The system correctly pulled the data for all four tickers from DuckDB. I verified the raw logs: the correct volume for Meta (META) was present in the prompt.

However, in the final output, the model correctly identified the date (April 4, 2025) but then falsely set the open price and volume of META to be identical to Amazon’s.

This wasn’t a hallucination in the sense of making something up from scratch; it was a synthesis error, likely due to the fact the final prompt used more than 90% of the maximum context-window size (~30K out of 32K) . The model “attended” to the Amazon data tokens while it was supposed to be writing the Meta row. For a business, this is a nightmare: the data is right, but the report is wrong.

---

The 50% Context Threshold and “Intelligence Degradation”

As an expert, I want to highlight a technical reality that executives rarely hear: LLM accuracy is not linear. As the context window (the model’s working memory) fills up, its ability to reason over that data can degrade.

Recent 2026 research into “Intelligence Degradation” has pinpointed a critical threshold—often around 40-50% of the maximum context length. Once a conversation (including all the raw DB data returned by tools) crosses this 50% mark, we see a “catastrophic collapse” in performance, sometimes dropping accuracy (F1 scores) by over 45%.

• The 40-50% Threshold: A 2026 study titled “Intelligence Degradation in Long-Context LLMs” [arXiv:2601.15300] precisely identified a critical degradation threshold for models like Qwen2.5. Once the prompt (including tool-fetched data) reaches 40–50% of the maximum context length, performance doesn’t just dip—it collapses. F1 scores were shown to drop by as much as 45.5% within a narrow 10% range.

• Context Rot: This is often referred to as “context rot” or “attention dilution.” Even when models can perfectly retrieve information (as confirmed by “Context Length Alone Hurts LLM Performance Despite Perfect Retrieval”, 2025), the sheer length of the input alone can hurt performance by 13.9% to 85%.

The Business Reality: In a “Simple Agentic” system, you are incentivized to give the LLM more data so it can be “smarter.” But in practice, the more data you fetch, the more likely the LLM is to make a “clerical error” that looks perfectly plausible to the human eye.

Business Risk 3: Lack of Native Domain Concept Understanding

Modern Open Source LLMs are giant static models. They map words to tokens, not to concepts. In the “Simple Agentic” pattern, we rely on the LLM to understand specialized abstractions, such as time, fiscal quarters, or industry groupings and map them to tool parameters. This creates a hidden layer of risk where the model uses its training data “intuition” instead of the current facts.

The Temporal Blindness Problem

LLMs have no built-in clock. They are static snapshots of data from their training cutoff. Without explicit “time-stamping” in the prompt (which some models do using their chat-templates), an agent has no concept of “today.”

• The “Last Week” Failure: When I asked the Qwen-3-Next model, “What did MSFT close at last week?”, it didn’t query the database for 2026 data. Instead, it hallucinated the desired date as September 2023. Why? Because without a “Current Date” reference in the system prompt, the model defaulted to its internal training context - an older static state.

• The “Future” Hallucination: Conversely, when asked about a date it’s model says was in the future (January 2, 2026), the model refused to query the DB entirely, stating the data was unavailable. It didn’t even attempt to see if the “future” date was actually present in the provided tool’s database.

Conceptual Mapping vs. Ground Truth

Business logic often relies on “group concepts” that are fluid. In finance, names like “The Magnificent Seven,” or “The Dow 30” may change. The knowledge in a static model can not change, and even regular updates might have gaps.

• Stale Definitions: Relying on the LLM to “know” which stocks belong in a group is a gamble. If the S&P 500 rebalances or a company changes its ticker (e.g., FB → META), the model may use its older training data to incorrectly populate a tool’s parameters.

• Domain Concepts: Consider a query for “Q1 returns.” Does “Q1” start on January 1st or the first trading day? Does the return calculate from the open of the first day , close of the first day or the close of the previous year? The LLM will pick one based on probability, not your company’s specific accounting standards. Likewise, minor prompt changes could result in different selections. Maybe adding an extra space into your query causes the model’s token probabilities to shift slightly resulting in picking open to close instead of close to close.

The Business Risk: When you leave conceptual mapping to the LLM, you lose semantic control. You aren’t just fetching data; you are delegating the definition of your business rules to a 3rd-party model’s training set.

---

Business Risk 4: Operational Cost and Latency Explosions

In the “Simple Agentic” approach, the agentic loop creates a quadratic scaling problem for both your bill and your users’ patience. Because the entire message thread, with all of the tool generated data, is sent back to the LLM for every new “turn,” the system becomes exponentially more expensive as it works.

The Token Bloat Trap

If a tool returns 5,000 tokens of raw data (e.g., a month of stock history) in Turn 1, that data is now part of the conversation.

• Turn 2: The LLM receives the original query + the 5,000 tokens of data.

• Turn 3: If the LLM needs a second tool call, it sends the query + the 5,000 tokens + the second tool’s result.

You are paying for that same 5,000-token block over and over again. For a complex, multi-turn query, a single user session can easily balloon to 50,000+ tokens. If you are using a 3rd-party provider with a 1M token window, a poorly formed query that causes a “loop” could cost $10 to $20 for a single answer.

The Latency Crawl

For local models, this isn’t just about money; it’s about responsiveness.

1. Context Pre-filling: Every time the thread grows, the LLM must “re-read” the entire prompt before it can generate a single new character.

2. The Slowdown: A system that feels “snappy” at the start of a chat (50+ tokens/sec) will noticeably crawl as the context window hits the “Intelligence Degradation” threshold we discussed earlier.

The Reality: The “LLM-as-Analyst” philosophy incentivizes “dumping” raw data into the prompt for the model to sort through. This is the most expensive and slowest way to process structured data.

Business Risk 5: Lack of Trust and Verifiability (Operational Opacity)

Figure 3: Output from a non-Internet connected LLM-call. Looks believable, but is entirely wrong since the market was closed that day.

In the same way LLM’s are trained to be convincing in their choice of words, they lare also trained to appear authentic in their formatting and use of citations. This is the “Black Box” problem. For a business, data is only as valuable as it is verifiable. In a “Simple Agentic” system, there is a fundamental lack of a deterministic audit trail between the source data and the final presentation.

The Illusion of Accuracy

When a user receives a polished response from an agent—complete with bold text, tables, and currency symbols—they have no way of knowing the “ancestry” of that data, did it come from the DB? Is it computed? Is it even accurate? The UI presents a “Final Answer” that looks identical whether it was:

1. A direct, 1:1 transcription from the database.

2. A calculation performed correctly by a specialized tool.

3. An incorrect transcription or computation

4. A “Helpful” hallucination based on the model’s internal weights.

Anyone using modern AI has likely seen results that are very convincing, even when they are totally wrong. We have all heard the stories of the lawyers who were sanctioned for submitting ChatGPT generated briefs that included fictitious case citations. This behavior is a result of how some LLMs are trained, judges, possibly non-experts under time pressure, giving thumbs up for “looks good” and not “is accurate”. Figure 3 shows an output from the llama.cpp web GUI (no Internet connectivity) for a static model. The result seems convincing, it even claims that the data was from legitimate sources. Unfortunately, this is just a very real looking hallucination, since the market was actually closed for the Juneteenth holiday. Because the model formats the answer so convincingly, users are likely to accept it as fact rather than questioning why the database didn’t throw an error, which for even the simple agentic approach does happen (the database indicates no data for that day).

The Lack of an Audit Trail

In a traditional reporting system, you can “click through” a number to see the underlying SQL query or the raw CSV or source. In the “Simple Agentic” pattern, that connection is severed. The LLM consumes the raw data, “processes” it in its hidden layers, and spits out a summary - that was built in a non-deterministic, and unpredictable way.

• Opacity for Auditors: If a financial report is generated by an agent, how do you prove to a regulator that the “90.25% gain” for Netflix wasn’t a rounding error or a synthesis failure or an outright hallucination not based on actual data?

• The “Invisible Processing” Risk: Even when the model is “correct,” it often performs invisible transformations. It might round a closing price from 174.4199 to $174.42. While this is great for readability, it is a lossy transformation, the true value is not preserved. If that rounded number is then used in a subsequent calculation by the LLM, the errors compound. Likewise, the decision to round, or how it rounds (chop vs round up) is not controllable as would be the case in deterministic code.

The Business Reality: Trust is hard to build and easy to lose. A system that cannot provide a clear, verifiable bridge between the raw data and the final word on the screen is a system that cannot be used for high-stakes decision-making.

Conclusion: Beyond the 600-Line Demo

The “Simple Agentic” pattern is a remarkable achievement in ease-of-use. It allows us to build flexible, natural-language interfaces for our data in a matter of hours with minimal code. However, as our investigation into the logs and the “on-the-wire” prompts has shown, it is a fragile foundation for production-grade software where accuracy and trust matter.

We have seen how the “Helpfulness Paradox” leads to silent hallucinations, how “Intelligence Degradation” causes synthesis errors at scale, and how the lack of deterministic control creates a black box that is expensive, slow, and unverifiable.

What’s Next?

The flaws we’ve identified aren’t a death sentence for AI agents; they are a call for better architectures. The “Simple” approach fails because it treats the LLM as the planner, the processor and the UI.

In my next post, I will present a more robust design: the “LLM-Computation Reduced, Deterministic Output Generation” pattern. I will show how to reduce the problems of concept-drift, eliminate potentially incorrect LLM-computation, and engender user trust by guaranteeing that the results they see are fully traceable and verifiable. In addition, in many cases the operational cost is also reduced.

The improved architecture:

• Guarantees accuracy by ensuring the LLM never “touches” the data.

• Reduce Prompt Length by shifting the work away from the LLM.

• Provides a clear audit trail so your users, and your auditors, know exactly what data was used, how it was processed and have confidence in the results presented.