Systems | Development | Analytics | API | Testing

Observe Opaque Services With OpenTelemetry eBPF + proxymock

Every SRE team operates services it cannot see into: a vendor binary, an inherited legacy deployment, a container whose owning team dissolved two reorgs ago. The routes are undocumented, the dependencies are unknown, and when a request takes 130 milliseconds nobody can say whether that time is application work or a wait across a network boundary.
Sponsored Post

Two confident fixes missed this production bug

Every new signup posts a message to our Slack. The format is dull and reliable: Overnight this week one arrived like this: That trailing nothing was the entire incident. No error logs, no alerts. A returning user had signed up, our signup service had attached them to a tenant we deprovisioned back in December, and the only symptom in the whole company was a Slack message that ran out of words. Their company's email domain still pointed at the dead tenant, so instead of a fresh workspace they landed in a corpse: domain mapping still live, cloud infrastructure long gone. I'm on call this week, so I pulled the thread.

The Pod Was Cheaper. The Service Wasn't.

A smaller Kubernetes pod can lower allocation cost while completing less work. Green status codes and matching schemas can hide it. This walkthrough combines OpenCost allocation data with proxymock behavior and performance evidence. A candidate passes only when behavior and throughput hold while unit cost falls.

Diagnose Serial N+1 API Calls With Tempo + proxymock

One API request took 302 milliseconds. Nothing failed. CPU was mostly idle. The response was correct. The trace made the problem obvious: eight inventory calls, each waiting for the previous one. But the trace could not tell me why the application made eight calls, or whether changing their execution would preserve the response. It showed the shape of the wait, not the input that created it.

Cut AI coding defects by 33% #mcpserver #aicoding #aiagents #grafana #aitools

We spend thousands of dollars "token maxing" and running endless debugging cycles just to walk our LLMs through a problem. But is the AI actually failing, or are we just withholding the right environment? Giving your AI assistant its own sandbox to test hypotheses might just be the missing link in your development workflow.