Observability Before You Need It: Logs, Metrics, and Traces for Small Systems

Your dashboard says average latency is 100 ms. Everything is green.

Google’s SRE book does the math that dashboard hides, in its chapter on monitoring distributed systems: “If you run a web service with an average latency of 100 ms at 1,000 requests per second, 1% of requests might easily take 5 seconds.” That’s ten users every second waiting five seconds, inside a chart that says 100 ms, all good. The footnote is even better: “if you’re not measuring your distribution, the idea that most of your requests are near the mean is just hopeful thinking.”

Small systems fall into this trap from two directions. Some have no instrumentation at all, so debugging means adding a console.log and redeploying. Others copy a big company’s setup and end up with averages that lie more expensively. The useful middle is small and cheap. You don’t need a Datadog bill the size of a junior salary. You need to answer one question without shipping new code: what happened at 3am?

The Question That Defines Observability

That question is my working definition of observability. Charity Majors, co-founder of Honeycomb, has spent years sharpening the idea at charity.wtf: can you understand what’s happening inside your system from the outside, including failures you never predicted, without deploying new instrumentation first?

Monitoring answers questions you wrote in advance (“is CPU above 80%?”). Observability lets you ask the ones you didn’t see coming (“why is this one user slow on this one endpoint, only after deploys?”). Whether that’s a separate discipline or a rebrand is a real debate, and every metrics vendor now selling “observability” doesn’t help. You don’t need to pick a side. For a small system the capability comes from three unglamorous parts: structured logs, a handful of metrics, and IDs that tie events together.

Structured Logging: Most of the Value, a Fraction of the Effort

If you do one thing from this article, do this: stop logging sentences and start logging key-value pairs.

"payment failed for order 4521 after retry" is greppable if you remember the exact wording. This is a record a machine can filter, group and count:

logger.Error("payment failed",
    "order_id", 4521,
    "customer_id", "c_8812",
    "attempt", 2,
    "provider", "stripe",
    "err", err)

Since Go 1.21 this needs zero dependencies. The standard library ships log/slog: leveled, structured logging with a JSON handler built in.

logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
    Level: slog.LevelInfo,
}))

// Attach request-scoped fields once; every line logged
// through reqLogger carries them.
reqLogger := logger.With("request_id", reqID, "user_id", userID)

Structured logs turn “read the logs” into “query the logs”: every error for customer c_8812 in the last hour, payment failed counted by provider, every request slower than a second.

My projects, this blog included, run on a single VM. At that size a legitimate log platform is JSON to stdout, Docker’s log rotation turned on (max-size, max-file) and jq. I’m not apologizing for that. Below a certain scale it is the correct architecture.

One habit turns logs into forensics: the correlation ID. Generate one at the edge, or accept an incoming X-Request-ID, put it in the context.Context, log it on every line, and pass it along in headers to downstream services and inside every queued job’s payload:

type requestIDKey struct{}

var validRequestID = regexp.MustCompile(`^[A-Za-z0-9-]{1,64}$`)

func RequestID(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        id := r.Header.Get("X-Request-ID")
        if !validRequestID.MatchString(id) { // never trust arbitrary client input in your logs
            id = uuid.NewString()
        }
        w.Header().Set("X-Request-ID", id)
        ctx := context.WithValue(r.Context(), requestIDKey{}, id)
        next.ServeHTTP(w, r.WithContext(ctx))
    })
}

When something breaks across a boundary, that one field decides whether you reconstruct the story or stare at disconnected lines. It’s also distributed tracing in embryo: the same idea, industrialized later.

The Four Golden Signals, Sized for One Server

For metrics, the minimal set comes from chapter 6 of the SRE book, written by Rob Ewaschuk: “If you can only measure four metrics of your user-facing system, focus on these four.”

Latency. Time to serve a request, measured as a distribution, never as an average. Track failed requests separately, because “a slow error is even worse than a fast error!” A 500 that takes 30 seconds holds resources hostage and disappoints the user anyway.

Traffic. Demand, in units that make sense for your system: requests per second for an API, concurrent sessions for streaming. It’s the context for every other signal. An error spike at 10x traffic is a different animal from the same spike at normal load.

Errors. Failed requests, in three flavors: explicit (500s), implicit (a 200 with the wrong content) and by policy: “If you committed to one-second response times, any request over one second is an error.”

Saturation. How full your most constrained resource is: memory, connections, disk, worker pool. Two lines from the book worth taping to your monitor: “many systems degrade in performance before they achieve 100% utilization,” and “Latency increases are often a leading indicator of saturation.” A climbing p99 is saturation announcing itself early. Saturation also has a future tense (“It looks like your database will fill its hard drive in 4 hours”), and that’s the alert that saves your Saturday.

For latency, use the trick the book recommends: don’t store latencies, count requests into buckets whose boundaries grow by roughly 3x (0–10 ms, 10–30 ms, 30–100 ms, 100–300 ms and so on). A Prometheus histogram does exactly that, runs on the same VM and costs nothing:

var httpDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
    Name:    "http_request_duration_seconds",
    Help:    "Request latency by route template and status class.",
    Buckets: []float64{.01, .03, .1, .3, 1, 3, 10},
}, []string{"route", "code"})

Label with the route template (/boards/{id}), never the raw path, or your label cardinality explodes. That one histogram gives you latency, its counts give you traffic, and the status-class label gives you errors. Add a gauge for your scarcest resource, expose /metrics, and you have the whole metrics stack a small system needs.

Health Checks That Tell the Truth

The last piece answers a machine’s question, “is this instance alive?”, and most implementations answer it dishonestly:

// The lie: returns 200 while the database is down,
// the queue is unreachable and every real request fails.
mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
})

The fix is to separate two questions. Liveness: is the process running? (The snippet above is a perfectly good liveness check.) Readiness: can it actually serve? That means exercising the critical dependencies:

mux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
    ctx, cancel := context.WithTimeout(r.Context(), 2*time.Second)
    defer cancel()
    if err := db.PingContext(ctx); err != nil {
        http.Error(w, "db unreachable", http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
})

Note the timeout: a readiness check that hangs is worse than one that fails, for the reasons in the timeouts article. And keep dependency checks shallow. A readiness probe that checks your dependency’s dependencies turns one slow service into a fleet-wide cascade of “unready” instances.

Alerts: Fewer Is the Point

Small systems have an advantage here and usually waste it. The SRE book’s paging philosophy is restraint: “Every time the pager goes off, I should be able to react with a sense of urgency. I can only react with a sense of urgency a few times a day before I become fatigued.” And, right after: “Every page should be actionable.” Page on symptoms (users are affected: error rate, latency, downtime), not on causes (CPU is high, which might be fine).

The book’s cautionary tale is Google’s own Bigtable team. Their alerts fired so often that, in the team’s words, “we often missed the problems that actually affected users, because so few of them did.” Part of the fix was disabling email alerts entirely and temporarily relaxing the SLO, which bought the time to fix root causes. If Google’s answer to alert fatigue was fewer alerts, your two-service system doesn’t need forty.

For a system you run alone, my minimum is three alerts: an uptime check from outside your infrastructure (your VM can’t tell you it’s down), sustained error rate, and disk approaching full. Everything else goes on a dashboard you look at when you’re curious.

“I’ll Add It When I Need It”

The counterargument deserves a fair hearing. Instrumentation is speculative work, YAGNI applies, a small system’s failures are simple, and your time is better spent on features. Over-instrumentation is also a real disease: the SRE book warns that monitoring “can become so complex that it’s fragile,” recommends removing rules that are rarely exercised, and says Google avoids “magic” systems that try to learn thresholds. Cargo-culting Google’s stack onto one VM is the most common way to catch it.

The flaw is timing. You need observability during or after an incident, which is exactly when you can’t add it retroactively. No amount of new instrumentation shows you what happened last night. The cost is small and paid once: slog is an afternoon, golden-signal metrics another, truthful health checks an hour, correlation IDs one middleware. About two days, against the first 3am incident where you’d otherwise be guessing.

Running systems alone makes that trade more lopsided, not less, because there’s nobody to ask what happened. Without a request ID, a failure that crosses the client, the API and a background job is archaeology. With one, it’s a query.

When to skip it: prototypes you’ll throw away, internal tools with two users sitting next to you, systems where a crash costs nothing. And keep pruning: signals with no dashboard and no alert attached are, per the SRE book, “candidates for removal.” That applies at every scale.

The Two-Day Setup

Swap sentence logs for slog JSON. Leveled, structured, request-scoped fields through logger.With(...). Afternoon one.
Thread a correlation ID through the context, every log line, outbound headers and queue payloads. One middleware.
Expose the four golden signals: one latency histogram labeled by route template and status class, plus a gauge for your scarcest resource. Afternoon two.
Split /healthz from /readyz, with timeouts on dependency pings and shallow checks only.
Three alerts, one of them external. Symptoms, not causes. Everything else is a dashboard.
Put a date on the graduation review. When you catch yourself joining logs across three services by hand, adopt distributed tracing with OpenTelemetry: the industrial version of the correlation ID you already have.

That closes the fundamentals layer of this series. Next up: event-driven architecture, where all six of these fundamentals get stress-tested at once.