← HomeLogin
A couple million lines of Haskell: production engineering at Mercury
~devhaskell
blog.haskell.org 2 weeks ago

Summary

For a new engineer who learned Haskell three months ago, "purity is a boundary you try to maintain" is much more useful than "Haskell is pure." One tells them what to do when they sit down to design a module. The other mostly sits there looking profound.

This boundary-oriented view of purity sets up a more general pattern that recurs throughout production engineering in Haskell: dangerous things are tolerable when they are fenced in, carefully exposed, and hard to misuse. That is true of mutation. It is true of retries, transactions, state machines, distributed workflows, and type-level machinery. Much of what follows is really just this same idea, wearing different hats.

[...]

I think of Temporal as Frankenstein's monster, in the flattering sense: assembled from excellent parts, animated by improbable effort, and smarter than many of the people alarmed by it. It takes durable history, replay, and determinism (things some platforms get natively,) and bolts them onto runtimes that were never born knowing how to do any of this. Most of us are not going to rewrite our companies in Erlang. Temporal is a prosthetic for the rest of us. It gives ordinary languages a shot at some of the same operational virtues by slightly mad but highly effective means.

The alignment with Haskell dovetails nicely with the virtues often attributed to Haskell: A Temporal workflow is, in an important sense, a pure function over its event history. Temporal Workflows have a determinism requirement — that a replayed workflow must produce the same sequence of commands as the original — which is exactly the same constraint Haskell imposes on pure code: same inputs, same outputs. Side effects are isolated into activities, which are the workflow's equivalent of IO. The workflow orchestrates; the activities execute. If you have spent any time thinking about pure core / impure shell, this is that model with the platform enforcing the separation rather than relying on sheer discipline.

[...]

If reliability is about adaptive capacity, introspection is one of the ways you buy it. Operators cannot understand what they cannot see. Teams cannot adapt systems whose internals are opaque. Observability is not a garnish you sprinkle on at the end. It is part of the design surface of the software.

This matters a great deal in Haskell, because Haskell does not have monkey patching. You cannot, at runtime, reach into a library and replace its HTTP client with one that records timings, or swap its database calls for ones that emit OpenTelemetry spans. This is not unique to Haskell, of course. Rust has the same fundamental constraint: no monkey patching, no runtime method swizzling, no language-level "let me just interpose here real quick" escape hatch. The orphan rule even prevents you from adding trait implementations for types you do not own. The difference is that the Rust ecosystem has largely converged on the tower middleware pattern as the answer, while Haskell's ecosystem is still fragmented across several approaches. The constraint is the same; the question is whether your ecosystem gives you a conventional escape hatch or leaves every team to improvise one.

[...]

The solution I reach for most often is records of functions. Instead of exposing a module full of concrete functions, you expose a record whose fields are the functions. The caller can then wrap, instrument, mock, or replace any individual function without touching the rest. I wrote about this at length in Embracing Flexibility in Haskell Libraries, but the short version is:

[...]

Libraries that do not do this are the ones that cost us the most operational pain. At Mercury, we very rarely use web API client bindings published on Hackage. This is not because they are necessarily poorly written (some are quite good). The problem is that we cannot trust code we cannot instrument. If a third-party binding makes HTTP calls through concrete functions, we have no way to add tracing, no way to inject timeouts tuned to our SLOs, no way to simulate partner outages in testing, and no way to explain the 400ms gap in a trace except by squinting at it and developing theories. So we write our own. More work upfront, but the clients we write are observable by construction, because we built them that way from the start.

[...]

If you are writing a Haskell library, leave escape hatches. Provide records of functions, or effect types, or callbacks, or something that lets the consumer of your code inject behavior without modifying it. Haskell's type system is wonderful for enforcing constraints. But it can also, if you are not careful, seal a system so tightly that the people who have to operate it cannot see inside. The perfect abstraction, if it is operationally opaque, simply cannot be used in production.