The Soul of Erlang and Elixir: Building Systems That Keep Working

Software systems must keep serving people while individual requests fail or run slowly. How can the rest of a system continue, and how can an operator diagnose the problem while it runs? In this 2019 talk, Saša Jurić argues that BEAM's lightweight processes, scheduling and runtime introspection make those tasks part of ordinary system design. He dedicates the talk to Joe Armstrong, Erlang's principal inventor, and his legacy.

The examples use Elixir, but Jurić's subject is the shared Erlang virtual machine, BEAM. Treating languages as interfaces to the runtime is his deliberate simplification, not a claim that Erlang and Elixir are identical. He moves from concurrency mechanics to a live demonstration, then to a more speculative argument about system architecture.

A BEAM process is a sequential execution context, not an operating-system process or thread. It runs ordinary function calls, branches and loops; concurrency comes from running multiple such processes. For example, spawn starts a new process from a function while the caller continues.

Each process has its own execution flow, stack and heap, and is garbage-collected separately. Processes cooperate by sending messages to an addressed recipient, whose mailbox is roughly like a queue. Jurić compares the model to independent microservices exchanging requests, but his local example uses runtime-supported messaging rather than HTTP and networking.

One BEAM instance runs as an OS process and can host many lightweight processes. Scheduler threads execute them. Jurić's teaching model imagines a queue: a scheduler takes a process, runs it briefly, then moves on to another. The actual scheduler is more sophisticated than that picture. He describes the then-current default as one scheduler per available CPU core and gives an approximately 130-million-process runtime limit; these are his 2019 account, not a capacity promise for an application.

07:08View full image

Jurić's example system runs as one background BEAM instance. He starts 10,000 synthetic jobs, each in a process: each performs a short CPU-bound loop, reports success to a known process, sleeps for a second and repeats. Other processes collect and display the reports.

The dashboard shows roughly 10,000 successes per second and about 20% scheduler utilization. The jobs are not synchronized, and the browser's frequent chart rendering also affects the measurements. Although the machine would normally use four schedulers, he deliberately configures one to make the effect of competing work visible. These figures describe this synthetic demonstration, not a general throughput benchmark.

The user-facing example sums the positive integers from 1 through n: for n = 3, the answer is 6. Each browser connection has its own process; when a request arrives, that connection process starts a short-lived calculation process. The latter sends the answer back and terminates, and the connection forwards the result to the browser. Connection processes come and go with users, while calculation processes come and go with requests.

11:57View full image

The demo deliberately makes input 13 raise an unhandled exception. The calculation process terminates, but the connection remains and receives a termination notification, allowing it to report an error. Meanwhile the synthetic jobs continue at roughly their previous rate. Jurić's point is that process boundaries can separate failures while still making the failure observable; they do not make the failed request succeed.

The original summation also uses a brute-force CPU loop. With an input of nine nines, it occupies the single scheduler for roughly ten seconds and drives utilization to 100%. Yet the dashboard continues to show approximately 10,000 successes per second, and the same connection can handle other input while the long calculation runs.

16:15View full image

Jurić attributes this to frequent preemptive scheduling: the runtime switches between processes without requiring this calculation to voluntarily yield. He describes intervals below a millisecond in the talk. Fairer progress across activities comes at the expense of maximum efficiency for an individual activity. His two reasons for splitting work into processes are therefore failure separation and latency separation. He defines availability in terms of useful service to users, while acknowledging that some service is still lost until defects are fixed.

A negative input exposes a more dangerous defect: the calculation enters an infinite CPU-bound loop. Other jobs initially continue, but repeated bad requests could accumulate enough runaway work to leave the system busy without producing useful results. Isolation buys time; it does not remove the need to repair the bug.

Playing the operator, Jurić sees a CPU alert and finds no useful new log entry. He connects to a remote IEx shell in the running system, without restarting it. He lists processes, inspects their properties and compares the change in each process's reduction count over a sampling interval. He describes reductions as roughly proportional to work performed since the process started; differences help rank the busiest processes. His own helper displays the top CPU consumers over the previous second, and a repeat sample identifies the same leading process.

22:30View full image

The runtime can inspect even a process that is not responding to messages. Jurić obtains its current stack trace and uses another demo helper to trace function calls for at most one second or 50 calls, whichever comes first. He then sends the process a kill exit signal. Scheduler utilization falls, the connection process reports the error, and the other demonstrated work continues. He presents this as runtime-level cancellation, including for timeouts or user requests; the demonstration is evidence about this BEAM process, not a blanket claim about every kind of external work.

25:00View full image

The stack trace locates the function and source line; the trace shows repeated recursion, with one argument increasing while the target remains negative. Jurić concludes that the deeper calculation assumes positive integers. The business domain permits that assumption, but the system boundary should reject invalid input before it reaches the calculation.

27:51View full image

For the demonstration he enables the missing validation, removes the special failure for 13, and replaces the brute-force sum with n × (n + 1) / 2. His tests pass. An explicitly makeshift deployment task recompiles and loads two modules without restarting the running system. Without refreshing the page, he shows negative numbers and zero rejected, ordinary inputs and 13 accepted, and the large sum completing quickly. The dashboard continues to show the synthetic workload. This is a demonstration of live repair, not a complete production deployment procedure.

Multiple BEAM instances can connect as nodes in a cluster. Jurić describes enumerating nodes, observing joins and departures, starting processes on nodes and exchanging messages across them using much of the same process-oriented model. A node is a runtime instance; the demo can run multiple nodes on one machine, so the observed node failure is not itself a test of a physical machine failure.

In the demonstration, named services are processes. He starts two nodes and names services foo and bar; when node 2 stops, bar reappears on node 1 and responds there. This illustrates the particular demo's recovery behavior, not a proof of general distributed consistency or availability.

Jurić explicitly says BEAM distribution has serious issues at both runtime and library levels, although he believes they can be fixed in implementations without fundamentally changing the interface. He does not detail those issues in the talk. That qualification matters when evaluating the demonstration for production use.

The local service code is unaware of distribution. The demo adds Swarm, a higher-level library that starts and locates named processes across the cluster. Its own processes watch node membership and manage the association between names and processes. A call starts a named service; another resolves the name to a process identifier, which the local service interface can use for a local or remote process.

36:31View full image

Jurić characterizes the integration as three lines of code and emphasizes that the orchestration runs inside the system rather than as another separately managed service. That economy depends on the demonstrated library and its runtime behavior; it does not erase the distribution caveats he just described.

Jurić compares two real systems he worked on. System A combined several technologies and external products to meet needs such as web serving, long requests, shared state, persistence, background work and recovery. System B met its needs in Erlang within one project and runtime. His slide gives examples of the different components, not a controlled benchmark between the systems.

36:38View full image

His proposed benefit of one language, one project and one OS process per machine is less coordination across development setup, tests, CI, deployment, metrics, logs and debugging. He especially values a lower learning burden for new teammates, who can contribute across more of the system. A general-purpose language can also express system behavior with the flexibility of code rather than many separate configuration formats.

This is a tradeoff, not a rule to eliminate every specialized product. Jurić acknowledges that the embedded solutions in his example lacked features and performance available from full-fledged products, even though they met that team's needs. Uniformity is useful when the total system and team benefit; he says it is not an end in itself.

Jurić closes with an ambition he explicitly says was not fully realized: create a small or medium web-facing system from an Elixir project and a few dependencies, with distribution and replicated SQL persistence integrated into the same system. The hoped-for benefit is a quick start that still leaves room to handle much larger and more complex requirements without rewriting everything.

That is his prospective argument, not a guarantee that five minutes of setup supplies production-ready distributed persistence or that every system can grow without a rewrite. The practical lesson of his demonstration is narrower and concrete: model independent activities explicitly, observe their behavior, and design the system to continue providing useful service while faults are investigated and repaired.