My AI Agent Was Lying to Me,
So I Put It Under Surveillance

My agent said it looked up the population of Iceland. The trace says both its lookups failed with a 429 — and it answered anyway, confidently, from memory.

That's a lie with a broken alibi. And I have it on camera.

This post is about building a small Gemini-powered research agent, wiring it with OpenTelemetry, shipping everything into self-hosted SigNoz, and discovering that the terminal output of an AI agent is basically a press release — the traces are the crime scene. By the end you'll be able to do the same surveillance on your own agent.

The suspect

The agent is deliberately simple: Gemini with three tools — search_wikipedia, calculate, get_current_time — and a system prompt that is basically a restraining order:

SYSTEM_PROMPT = (
    "You are a meticulous research assistant. For ANY factual question, "
    "you MUST call search_wikipedia before answering. For ANY arithmetic, "
    "you MUST call calculate. Never answer factual questions from memory."
)

Spoiler: it does not honor the restraining order.

Installing the surveillance equipment

SigNoz now installs through Foundry — one binary, one YAML, done:

curl -fsSL https://signoz.io/foundry.sh | bash

My entire infrastructure definition is nine lines:

apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    flavor: compose
    mode: docker
  mcp:
    spec:
      enabled: true

foundryctl cast -f casting.yaml

One command later I had ClickHouse, Postgres, an OTel collector, the SigNoz UI on localhost:8080, and an MCP server on localhost:8000. (Windows note: run this inside WSL 2 with native Docker Engine, not Docker Desktop — ClickHouse Keeper segfaults under Docker Desktop's virtualization on Windows. Ask me how I know.)

Wiring the wiretap

Three signals, all to localhost:4318:

• Traces: openinference-instrumentation-google-genai auto-instruments every Gemini call (model, prompts, token counts). I added a root agent.run span per question, and — critically — wrapped every tool in its own child span:

def search_wikipedia(query: str) -> str:
    with tracer.start_as_current_span("tool.search_wikipedia") as span:
        span.set_attribute("tool.input.query", query)
        ...

• Logs: Python logging → OTel handler, auto-correlated to trace IDs.
• Metrics: four custom instruments — agent.tokens.total, agent.tool.calls, agent.request.duration, and my favorite: agent.suspected.lies, incremented whenever a factual question gets answered with zero tool spans in the trace.

If the agent claims it "looked something up" and there's no tool span — no alibi — it's lying.

The agent lied before it said a single word

First run. First question. Instant crash:

404 NOT_FOUND: This model models/gemini-2.5-flash is no longer
available to new users.

I had coded against a model that Google had quietly cut off for new API keys — despite a published shutdown date months away. Lesson one arrived before lesson zero: model names are not an API contract.

Caught in 4K, incident #1

After switching models, the agent ran — and immediately produced the screenshot of my dreams. Wikipedia was rejecting my tool calls with 403 (missing User-Agent header, one-line fix). So the agent's only tool was dead. Its response to "What is the capital of Bhutan?":

The capital of Bhutan is Thimphu.

Zero working tools. Two failed spans glowing red in the trace. Full confidence in the answer. In the terminal this looks like success; in SigNoz the trace waterfall shows two error spans followed by a cheerful answer pulled entirely from memory.

Death by two different 429s

Then the rate limits arrived, and they arrived with a twist.

Gemini's free tier gave me 5 requests/minute — and my agent burns 2+ requests per question (one to decide on a tool, one to compose the answer). I added exponential backoff, felt smart for eleven seconds, and then hit the wall behind the wall:

Quota exceeded ... GenerateRequestsPerDayPerProjectPerModel-FreeTier
quotaValue: '20'

Same 429 status code, completely different meaning. The per-minute limit was hiding a 20-requests-per-day limit underneath, and my retry logic was faithfully retrying against a wall that doesn't move until midnight. Only the response body tells you which 429 you hit. Not all 429s are retryable — parse the quota violation, don't just read the status code.

And because my agent had been enthusiastically re-searching Wikipedia three and four times per question (the traces show it querying "population of Iceland", then "Iceland population", then "What is the current population of Iceland?" — burning tokens on rephrasing its own searches), Wikipedia rate-limited me too. Two services, two 429s, one afternoon.

The loophole: my lie detector counted attendance, not alibis

Here's my favorite failure. With Wikipedia rate-limiting us, the Hamlet question went like this: one tool call → 429 → agent writes a flawless two-sentence plot summary from memory. Iceland: four tool calls, last two failed → "approximately 380,000 to 400,000 people," delivered from memory with a straight face.

My agent.suspected.lies metric stayed at zero.

Why? My detector checked whether tools were called, not whether they succeeded. The agent showed up to the interrogation, said nothing useful, and my detector marked it cooperative. It found the loophole in my compliance check within hours of the check existing — which is a suspiciously accurate simulation of a real employee.

The fix is to verify alibis, not attendance: tag agent.tool.calls with a success attribute and treat "answered after all tools failed" as its own suspect category.

The interrogation room (dashboard + alerts)

With real data flowing, I built a dashboard in SigNoz's Query Builder — four panels: token burn by direction, tool usage by name (the "Alibis" panel), suspected lies, and response latency. The tool-usage panel is where the triple-Wikipedia-search habit becomes visually undeniable.

Then an alert on agent.suspected.lies > 0 in 5 minutes, named "The Agent Is Lying Again" — because observability without alerts is just watching crimes happen in HD.

Since Foundry also deployed the SigNoz MCP server, I connected it to my AI coding assistant — so I can now ask an AI why my AI misbehaved, by having it query the traces directly. Surveillance all the way down.

What I'd tell my past self

1. Terminal output is the agent's press release. Traces are the crime scene. Every incident in this post was invisible in stdout and obvious in the trace waterfall.
2. Run ListModels before hardcoding a model name. I hit three different model-name failures in one day; the fix each time was one boring curl.
3. Parse your 429s. Per-minute and per-day quotas share a status code and demand opposite responses.
4. Instrument tools with their own spans. LLM auto-instrumentation shows you what the model said; tool spans show you what it actually did. The gap between the two is where the lying lives.
5. Verify alibis, not attendance. A compliance check that counts attempts instead of successes will be gamed immediately — by agents and, I suspect, by everyone else.

Conclusion

I set out to monitor an AI agent and accidentally ran a full criminal investigation — the deprecated model, the dead tools, the double rate-limiting, and a lie detector that got outsmarted on day one. Everything is reproducible: the repo ships casting.yaml + casting.yaml.lock, so foundryctl cast rebuilds the entire surveillance state.

Repo: agent-under-surveillance · SigNoz: signoz.io/docs · OpenTelemetry: opentelemetry.io

Disclosure: I used an AI assistant (Claude) to help scaffold code and edit this post. Every incident, error, and screenshot is from my actual terminal — no AI was harmed, only monitored.