Systems | Development | Analytics | API | Testing

speedscale

Your Observability Stack Found the Fire. Congratulations.

Aug 27, 2026 By Alan Mon In Speedscale
The dashboards are red, the alerts are firing, and Slack has officially become a war room. Someone has asked, “Anyone else seeing this?” and 14 people have immediately responded with screenshots. Welcome to another day in distributed systems. The recent GitHub outage is a great reminder of how complicated modern applications have become. Services talk to services, infrastructure scales up and down, retries multiply traffic, and dependencies behave in ways nobody expected.
Read Post
speedscale

The Compounding Platform Tax

Aug 27, 2026 By Matt LeRay In Speedscale
Private cloud and on-premises infrastructure are different. I know this. I used the terms interchangeably anyway because I was trying to compress a pattern into one sentence. That was sloppy, but the underlying pattern still matters. Banks run Kubernetes differently from most other organizations. Not because every bank makes the same choices, but because similar constraints can produce a similar pattern.
Read Post
speedscale

Application Level Dependency Chaos Testing

Aug 26, 2026 By Matt LeRay In Speedscale
Somewhere in your service is a branch that has never executed. Not a rare one, a never one. It handles a dependency being unavailable: it reads from a cache, it returns a stale value, it marks the response degraded so callers know not to trust it too far. It was written carefully. It was reviewed. Whether it works is an open question, because nothing in the test suite makes that dependency fail, and the dependency does not fail on request.
Read Post
speedscale

I built an API traffic classifier for business workflows

Aug 25, 2026 By Ken Ahrens In Speedscale
An engineering leader asked me a question a few weeks ago: could we read their business workflows out of API traffic instead of asking people to document them? I said it should be possible. Then I tried it. A few engineers know how the system really works. They know which calls make up a work order and which checks happen after a write. That stuff rarely makes it into the test plan. Usually it’s in somebody’s head. Sometimes it’s in several heads, with slightly different answers.
Read Post
speedscale

From Telemetry to Traffic

Aug 19, 2026 By Matt LeRay In Speedscale
A metric says latency increased. A log says a request failed. A trace identifies the slow dependency. An APM agent points to the method. Manual instrumentation explains the business operation. Traffic capture shows the exact request and response that triggered it. Each layer answers a question the previous layer could not. Each also introduces a new cost, blind spot, and failure mode.
Read Post
speedscale

Reliability Engineering in the AI Era

Aug 19, 2026 By Matt LeRay In Speedscale
Engineering leaders have been claiming to “shift quality left” for years but production remains stubbornly stuck out of reach of software engineers. The realm of production remains mysterious with tools no one has access to and UIs that wouldn’t make sense to engineers anyway. I’ve noticed a small but growing trend of large enterprises hiring Reliability Engineers instead of Site Reliability Engineers. Dropping one word looks cosmetic but I think it points to a much bigger change.
Read Post
speedscale

Chaos Monkey Won't Find Your Bug

Aug 17, 2026 By Matt LeRay In Speedscale
We shipped a chaos feature that never caused any chaos. Our mock server has had a fault-injection effect for years with a straightforward job: withhold the response entirely and see whether the caller copes. Last week I audited it against the actual code path. It had never withheld anything. The handler returned early without writing a response. Go’s net/http then did what it is designed to do, which is synthesize a 200 OK and flush the recorded body.
Read Post
speedscale

Synthetic Monitoring Is Broken. Your Production Traffic Can Fix It.

Aug 13, 2026 By Ken Ahrens In Speedscale
Synthetic monitoring has been a critical part of application reliability for years. It gives engineering and operations teams a way to proactively test applications, APIs, and critical customer journeys before users encounter problems. But there is a fundamental limitation with the traditional approach: Someone has to create the tests. As applications become more distributed and customer journeys become more complex, organizations can end up maintaining hundreds or even thousands of synthetic scripts.
Read Post
speedscale

Observe Opaque Services With OpenTelemetry eBPF + proxymock

Aug 12, 2026 By Matt LeRay In Speedscale
Every SRE team operates services it cannot see into: a vendor binary, an inherited legacy deployment, a container whose owning team dissolved two reorgs ago. The routes are undocumented, the dependencies are unknown, and when a request takes 130 milliseconds nobody can say whether that time is application work or a wait across a network boundary.
Read Post
Sponsored Post
speedscale

Two confident fixes missed this production bug

Aug 11, 2026 By Ken Ahrens In Speedscale
Every new signup posts a message to our Slack. The format is dull and reliable: Overnight this week one arrived like this: That trailing nothing was the entire incident. No error logs, no alerts. A returning user had signed up, our signup service had attached them to a tenant we deprovisioned back in December, and the only symptom in the whole company was a Slack message that ran out of words. Their company's email domain still pointed at the dead tenant, so instead of a fresh workspace they landed in a corpse: domain mapping still live, cloud infrastructure long gone. I'm on call this week, so I pulled the thread.
Read Post
speedscale

The Pod Was Cheaper. The Service Wasn't.

Aug 11, 2026 By Matt LeRay In Speedscale
A smaller Kubernetes pod can lower allocation cost while completing less work. Green status codes and matching schemas can hide it. This walkthrough combines OpenCost allocation data with proxymock behavior and performance evidence. A candidate passes only when behavior and throughput hold while unit cost falls.
Read Post
speedscale

Diagnose Serial N+1 API Calls With Tempo + proxymock

Aug 11, 2026 By Matt LeRay In Speedscale
One API request took 302 milliseconds. Nothing failed. CPU was mostly idle. The response was correct. The trace made the problem obvious: eight inventory calls, each waiting for the previous one. But the trace could not tell me why the application made eight calls, or whether changing their execution would preserve the response. It showed the shape of the wait, not the input that created it.
Read Post

Cut AI coding defects by 33% #mcpserver #aicoding #aiagents #grafana #aitools

Aug 5, 2026 By Speedscale In Speedscale
We spend thousands of dollars "token maxing" and running endless debugging cycles just to walk our LLMs through a problem. But is the AI actually failing, or are we just withholding the right environment? Giving your AI assistant its own sandbox to test hypotheses might just be the missing link in your development workflow.
View Video

Copyright © 2026 OpsMatters™. All rights reserved.