Observability · Incident RCA · MCP

We gave Claude Code
access to production.

Used to take 15 minutes just to build a theory. Kibana, APM, Mongo logs, Sentry, each in a different tab, each with its own time zone. batman changed how we investigate, find root cause, and write the analysis. One conversation in Claude Code. Every data source. The write-up at the end.

Python 3.12+ 10 tools
batman · claude code
You said
There were slow response complaints around 10am. Check mongo.
→ calling mongo_logs · 09:30–10:30 IST
✓ queried MongoDB diagnostic logs via Athena
✓ found 2 slow queries in window
slowusers: full collection scan, 2.6 GB, 6s CPU. Scheduled batch job with no request context.
slowsupportRequests: aggregation on (brokerid, userid), no compound index, scanning hundreds of docs
Neither visible in APM. Background operations only show in DB logs.

The investigation takes 5 minutes.
The scaffolding takes 15.

We have Elasticsearch for app logs, Elastic APM for traces, MongoDB for database diagnostics, Prometheus for infrastructure metrics, Sentry for errors, and AWS for everything else. Each queryable. None aware of the others.

During an incident, you're not really thinking about the problem. You're managing the tooling around it. By the time you finish the Kibana query, you've half-forgotten what APM said. By the time you write the database query, you've lost the thread. Context switching is not free.

The write-up was worse. After the fire was out, you'd try to reconstruct what happened from browser history, Slack threads, and memory that was already fading. Half the detail was gone before you typed the first word.

batman plugs all of this into Claude Code as an MCP server. You describe the problem in plain English. Claude picks the right tools, runs the queries with the right parameters, and reasons over everything in one place. The investigation and the write-up happen in the same thread.

One tool
per data source.

Each tool is a typed wrapper around one part of the stack. Claude decides which ones to call based on what you ask.

app_logs
Elasticsearch app logs across all services. 7-day default. Pass any service name fragment and it finds the right index family. Text search defaults to all fields, so a session ID or trace ID finds the entry without knowing which field it's in.
mongo_logs
MongoDB diagnostic logs via Athena. Slow queries, COLLSCAN warnings, CPU burn, bytes read. This is where you find the queries actually hurting you.
mongo_audit_logs
MongoDB audit trail. Every operation by collection, user, and command type. Good for "which service is running 200 aggregations at 3am on this collection."
apm_transactions
Elastic APM. Latency percentiles (p50/p95/p99), error rates, slow spans, downstream service breakdown. Usually the first thing to check.
prometheus_query
PromQL range queries, merged across both Prometheus instances. Memory pressure, restart counts, network errors. Anything not visible in APM.
sentry_issues
List and filter Sentry issues by project, query, or time range. Default filter is is:unresolved. Good for catching things before they hit Slack.
sentry_issue_events
Individual Sentry occurrences with full stack traces and breadcrumbs. Drill in after finding an issue ID from sentry_issues.
elastic_synthetics
Heartbeat and synthetic monitor results. Useful for checking whether external monitors already detected issues before alerts fired.
aws_query
Any read-only boto3 call. EC2, RDS, EKS, Route53, IAM, 30+ services. Hard-blocked on anything that is not a describe/get/list call.
kubernetes_query
Proxies to kubernetes-mcp-server. Pod logs, events, resource descriptions. For when you need the Kubernetes layer without leaving the conversation.

What changed
in practice

Slow complaints at 10am
investigation · mongodb logs
You
There were slow response complaints around 10am today. Check mongo.
Claude via batman
Called mongo_logs for 09:30–10:30 IST. Two slow queries:
Batch job on users collection. Full collection scan. 2.6 GB read, 6s CPU. No request context, scheduled job.
Aggregation on supportRequests matching on (brokerid, userid) with no compound index. Scanning hundreds of docs to return nothing.
Neither would show up in APM. APM only traces request-scoped work. These were background operations.
Who is hitting this collection
investigation · mongodb audit logs
You
Who is hitting the usersAuth collection?
Claude via batman
Called mongo_audit_logs for the last 24 hours. The notifications service ran 134 aggregations against usersAuth in a 6-hour window. One query per user, not batched. Not a bug, but not a pattern that scales.
New Sentry issues after a deploy
investigation · sentry
You
Any new Sentry issues since this morning's deploy?
Claude via batman
Called sentry_issues with is:unresolved, last 4 hours. Two issues, both first seen within 2 minutes of the deploy:
Null reference on a path that only exists post-auth. Matches the JWT middleware change in the diff.
Duplicate key on order creation. Race condition introduced by the deploy.
Both found before the first user complaint reached support.
Alert to write-up in one shot
investigation + postmortem · apm, mongo, app_logs
You
Got complaints about slow portfolio loads around 2pm today. Figure out what happened and draft the postmortem.
Claude via batman
Called apm_transactions (13:30-14:30 IST): p99 on the portfolio endpoint went from 320ms to 4.2s at 14:05, back to baseline at 14:22.

Called mongo_logs for the same window: one COLLSCAN on the positions collection, 850ms, firing every 30 seconds during that window. Scheduled batch refresh job, no index on (userId, timestamp).

Called app_logs: 340 timeout errors on the portfolio service between 14:05 and 14:22.

Root cause: batch refresh job added last week queries positions on userId with no compound index. Full collection scan at production volume (40x staging). Each scan contends with concurrent portfolio reads for that 850ms window.

Draft postmortem

Duration: 14:05-14:22 IST (17 minutes)   Severity: P2

What happened: Portfolio endpoint p99 latency jumped to 4.2s at 14:05. 340 timeouts before auto-recovery at 14:22. No data loss.

Root cause: Batch refresh job introduced last week queries positions.userId with no compound index. At production volume the full scan takes 850ms and runs every 30 seconds, blocking concurrent reads during that window.

Why staging didn't catch it: Staging positions collection is 2% of production. Scan completes in 18ms there.

Fix: Add compound index on (userId, updatedAt). Batch job should also carry request context so it appears in APM traces.

The postmortem used to be a separate task. Now it's the last message in the investigation thread.

The write-up is already
half-written.

Claude already has the tool calls, time windows, results, and dead ends. Asking for a postmortem keeps the specific numbers and the reason staging missed it, instead of rebuilding both from browser history and memory.

This changed our workflow more than the faster queries did. The write-up is now the final step of the investigation, not a separate act of reconstruction.

Flexible enough to
not need maintenance

Elasticsearch at scale accumulates index naming debt. Teams use different conventions, log shippers evolve, naming schemes diverge and never get reconciled. A naive approach hardcodes prefixes and breaks silently when a new service ships with a different convention.

batman uses two strategies depending on what you ask for.

Service-specific queries search *{service_name}*. Wildcard on both sides. Hits any index family regardless of prefix, so it works even if the service changed naming schemes three times.

Broad queries without a service scope to date-suffixed indices when you give a single day, or use prefix wildcards with the time filter doing the work for multi-day ranges. The 7-day default kicks in when you don't specify a window.

Text search defaults to fields: ["*"] with lenient: true. A session ID or trace ID finds its log entries without the caller needing to know which field it lives in.

app_logs · index resolution logic
# specific service: wildcard both sides f"*{service_name}*" # single-day broad: scope to date suffix ",".join(f"{p}*{date}" for p in all_prefixes) # multi-day or no date: prefix wildcards + time filter ",".join(f"{p}*" for p in all_prefixes)

Streamable-HTTP,
not SSE

batman started on the older HTTP+SSE transport. In our setup, long-lived connections were getting dropped while slower boto3 and Athena calls were still running. It looked random because quick calls worked. Moving to Streamable HTTP fixed that failure mode for us by letting the client initiate each MCP message with a POST. The protocol can still use SSE in a response, so the useful lesson is narrower: remove the long-lived connection when your proxy path cannot keep it healthy.

Two other things I learned the hard way during the migration.

gotcha 1 · stateless mode

Use stateless=True in the session manager. Stateful mode expects the client to GET an endpoint first to establish a session, then POST with the session ID. Claude Code's client just POSTs. Without stateless mode, every call fails with a missing session ID error and the message gives you no hint why.

Symptom: every tool call returns a session error. Fix: stateless=True in StreamableHTTPSessionManager.
gotcha 2 · trailing slash

The trailing slash in the URL is not optional. Starlette's Mount("/mcp", ...) redirects /mcp to /mcp/ as a GET. The client follows that redirect as a GET, not a POST, which breaks the handshake. Fix is trivial but took embarrassingly long to find.

Always use /mcp/ with the trailing slash. Never /mcp.

Tool descriptions are
the real prompt

The quality of responses depends heavily on how well tool descriptions are written. Claude reads them to decide which tool to call, with what parameters, and in what order.

Vague descriptions produce vague tool calls. Concrete examples work better than abstractions. "Service name fragment, matches *{name}* across all index families" is more useful than "the service to query." The model infers the right call shape immediately instead of guessing.

Defaults matter too. The app_logs description says it defaults to the last 7 days when no time range is given. "Check logs for the orders service" just works without specifying a window. The description is what teaches the model to do that.

This is the part most MCP tutorials skip. They query clean toy data with predictable schema. Production data has years of inconsistent field types, naming conventions, and logging library choices baked in. The tool description is where you encode that institutional knowledge so the model can navigate it.

Read-only is not
a security model.

The title is intentionally uncomfortable. Giving an agent production visibility is useful, but a method prefix is not enough to make it safe.

The AWS wrapper blocks operations outside describe_*, get_*, and list_*. That is only the first layer. Some read calls can still return secrets, personal data, or enough metadata to cross a team boundary. The real boundary has to live in IAM and source-specific credentials, outside the prompt and outside the model.

What I expect around a setup like this: separate least-privilege credentials per source, explicit service and operation allowlists, secret-bearing APIs blocked, query windows and result sizes capped, tool calls audited, and sensitive fields removed before results enter the conversation.

Logs are also untrusted input. A log line can contain instructions just as easily as it can contain a stack trace. Tool output should be treated as data, never as authority. batman does not get a shell, write credentials, or a path to make production changes. A human still owns the diagnosis and the fix.

This is the part I would keep boring. If a control exists only because the prompt says so, it is not a control yet.

Built for
the team.

batman is an internal tool we built at smallcase. It's not open source, but the patterns (MCP as observability glue, tool descriptions as institutional knowledge, index routing that adapts to naming debt) work for any team with a similar stack.

If you're building something similar, the MCP SDK docs are a good starting point. The hard part isn't wiring up the tools. It's the months of understanding what your data actually looks like before the tool descriptions get good enough to be useful.

Build your own observability MCP → Pick your stack, get copy-pasteable Claude prompts to scaffold it.