Let's build! A distributed, concurrent editor: Part 5 - Actors

In this series:


Last week I explored how the server could store a document and its history on disk, and how it can calculate and communicate the correct document state when it receives an undo or redo message. This week I want to look at the overall architecture of the server, and servers in general. I want to discuss the Actors model for dealing with concurrency, and why I believe it is an excellent approach to architecting servers. There are definitely opinions ahead, which are a result of my experiences and biases.

A brief history

When I was at university, in both undergrad and PhD courses, I studied several formal models for concurrent programming. At the time, study into concurrency was a hot topic: consumer CPUs had started going parallel, with hyper-threading appearing. Everyone knew that writing programs that make good use of these new CPUs was difficult: parallel computation had been around for a long time in more expensive and rarer CPUs. But now it was starting to cause headaches for everyone.

The dominant approach, both then and now, is to use locks to regulate how multiple concurrent threads access data structures. But as a program gets larger, it becomes difficult to use locks correctly: it is easy to accidentally introduce deadlocks.

  • Locks don’t compose, so you always have to reason about the program as a whole, not small parts in isolation: small parts can be correct, but the overall program can still be faulty.
  • Pointer aliasing and other features of mainstream programming languages make it impossible for static analysis (type-checkers) to prove a program is free of deadlocks, without significant changes to the type-systems of those languages.
  • Tests normally don’t help much because whether a faulty program deadlocks when running is dependent on how its threads get scheduled, which is not something a program typically has any control over.

Lots of ideas were receiving attention, from ownership types (which I believe went on to form a major part of the design of Rust) to software transactional memory (which didn’t really go on to anything much as far as I know; although I’ve implemented it a few times), and others. At the time, there was still significant resistance from some quarters to statically-typed languages: a lot of people seemed to revel in the fact that a human is smarter than a computer (at some tasks), and so object to type-checkers that tell them their program can’t be proven safe. Thankfully, from what I can tell, attitudes seem to have changed. But I suspect that if Rust 1.0 had been released 10 years earlier, it would have been dead on arrival.

Ownership types, and similar research at the time, was focused on trying to prove at compile-time (i.e. before running it), that a program which shares data between multiple threads is free from data-races and deadlocks. Transactional memory was focused on identifying at run-time unsafe concurrent access to shared data, and making sure that when it occurs, the effects are undone and tidied up without doing any damage. By contrast, the Actor model prohibits sharing mutable data between actors. Instead, data is sent between actors, but a piece of data should only ever be accessible by a single actor at a time.

Actor implementations can rely on ownership types, or use transactional memory if they want to. Ultimately, communication between actors is achieved with a shared data structure: a mailbox or queue of some sort, which must be safe for multiple actors to concurrently send to, and a single actor to receive from. But really, Actors don’t require fancy new type-systems, or anything particularly novel from a language run-time. The Actors model has been around for some time: Carl Hewitt and others created it in the 1970s, and Gul Agha further developed it in the 1980s. Type-systems have been developed that allow the communication patterns between actors (protocols) to be specified and verified (session types). To my knowledge, they have not enjoyed mainstream adoption.

Towards the end of my time at university, I did more and more work on RabbitMQ, which is written in Erlang. Erlang is an Actor programming language. So a lot of my early career was spent building a pretty scalable and reasonably well-performing distributed messaging system, using Actors. This probably explains why I find myself biased in favour of Actors, and why I have built Actor frameworks a few times.

What are Actors?

An actor is a little server. It has a mailbox; other actors will send it messages by posting them into the mailbox. When a message appears in the mailbox, the actor will retrieve it, and will process it in some way. Then it’ll go back to sleep until the next message arrives. In the course of processing a message, an actor can send messages to other actors it knows about (including sending to itself), it can mutate its own state, it can spawn new actors if it wishes, and it can choose whether to terminate instead of waiting for the next message. When a message is sent to an actor, ownership of that message transfers to the recipient.

Unlike most other models of concurrency, actors combine the unit of concurrency (a thread, or thread-like thing) with state ownership: you cannot talk about state without talking about the actor who owns and manages that state. It is nonsensical to talk about multiple actors having access to the same state. If you want an actor to change its state then you send it a message asking it to do so. Sending a message to an actor is typically asynchronous: the sender does not block, waiting for either the message to be received, or some sort of reply to be issued. But if your message requires a reply then you can include your own mailbox as part of that message, and the actor can use that to send you its response (amongst other techniques). A single actor is always a single thread. So sometimes you’ll want to make sure you spawn enough actors of a particular type so that you have one per CPU core, to ensure you can make best use of the CPU resources available to you.

It’s this concept of unified state and concurrency that I think appeals to me: one model to keep in my head and think about instead of two or more. Uncomplicated rules about state ownership.

Actors can be short-lived or long-lived. Some programs might create a bunch of actors as soon as they start, and those actors will stay alive for a long time. Equally, it should be very cheap to spawn short-lived actors: actors that come into existence just to carry out some specific task and then terminate. Because of the expectation that spawning new actors should be very cheap, Actor languages and frameworks often work best on green-threads: virtual threads that are typically managed by the language run-time. They’re lighter-weight than OS-level threads (i.e. context switching between them is cheaper) because the language run-time is able to take advantage of extra knowledge it has, to do less work to switch between threads. It’s common for such language run-times to create one OS-level thread for each CPU core, and then the run-time chooses how to schedule its actors (or green-threads) across those OS-level threads.

Go has always supported green-threads, in the form of Go-routines. Its channels are a little like mailboxes. It makes no further provisions for Actors, but it’s not too difficult to fill in the missing parts, which I’ve done with my actors library. There are Actor frameworks in most mainstream languages; and several in Go (though many of them look abandoned to me).

Actors have some ideas in common with micro-services. But actors don’t need to use TCP or HTTP or GRPC to send messages to each other, and you don’t need docker-compose or mini-kube or any other pile of ridiculous, unnecessary, and accidental complexity to orchestrate them. An essential feature of any Actor framework is the provision for creating, managing, and terminating actors. You don’t need external tools for it: it’s all baked in. Being notified that an actor has terminated (and why), or being able to terminate a set of actors in a careful and deterministic manner, is critical for a reliable and well-behaved program. Erlang has an entire set of libraries and principles to help with this, and I’ve borrowed several design ideas from there.

Actors also differ from micro-services in that micro-services normally form a distributed system: each service may run on a different machine, and they use a network to pass messages between them. Each of those machines could fail independently (catch fire etc), and messages could be lost on the network. That friendly dog could come bounding in and chew up a network cable. These sorts of failure scenarios are part and parcel of a distributed system. For actors all running within a single program, a single OS-process, these sorts of failures can’t happen. Nevertheless, some Actor frameworks also work in a distributed setting, allowing messages to be sent between actors on different machines, without the code having to show any knowledge of the location of the actors. Erlang can do this, for example. My actor framework cannot; in Go, it seems very difficult to send objects between machines and maintain pointer equality properties (if one actor sends the same pointer to another actor, twice, then the recipient should be able to see that both pointers have the same value (point to the same object). In light of garbage collection and the fact Go does not support weak references, I currently believe it’s not possible to implement this correctly without changes to the language run-time).

So Actors allow you to architect your program around a set of servers, which have a simple combined model of state ownership and concurrency. They send messages to each other to coordinate and communicate. The Actor framework provides mechanisms to spawn new actors, to monitor actors for termination, and to manage the life-cycle of actors. I believe that building programs using actors helps you practise thinking about things like:

  • the different orders in which messages might be sent and received;
  • the different orders in which your actors might be scheduled and preempted;
  • what bits of state should belong together because they need to be updated at the same time;
  • how to run multiple actors of the same type to scale your program and make good use of a parallel CPU.

Thinking about these sorts of things regularly helps when it comes to designing and building distributed systems: it’s all the same stuff, apart from that distributed systems can fail in even more exciting ways.

Actor Life-cycle

An actor can be divided into two parts: the client-side API, and the server-side. If you’re familiar with Go, you may have heard of a general design principle which says “don’t make channels part of your API”. Go’s channels can have complex and subtle semantics, and it’s generally advisable to wrap them in friendlier API. The same is true of an actor and its mailbox: if you expose just the mailbox as an actor’s API then it’s not clear what messages the actor responds to, where there’s any ordering requirements of particular messages, and so on. So instead, it’s advised that you wrap the mailbox with a client-side API. This client-side code presents to the world the methods that your actor supports, but hides the details of posting messages into the actor’s mailbox, maybe waiting for a response, and any other logic, from the user of your actor.

The server-side of the actor gets created and completes its Init method before the call to Spawn returns.

Figure 1: Spawning an actor.

Figure 1 depicts spawning a new actor. The server-side of the new actor completes its call to the Init method before the call to Spawn returns. If the Init method returns an error then that error will be the returned from Spawn. It also means that whilst Init is running, no other actor can know of the existence of the new actor. This can be useful: for example it means that the new actor, in its Init method, can send itself messages. Those messages are then guaranteed to be the first items in the actor’s mailbox which means it can safely do some long, complex initialisation asynchronously, and not keep its spawner blocked. Figure 2 shows the flow of messages and control in an actor.

Figure 2: The flow of control and messages in an actor and its mailbox.

If actor X posts two messages in some order (m1, then m2) to actor Z’s mailbox, then it is guaranteed that if both messages are received by actor Z, then m1 will be received before m2. If actor’s X and Y concurrently post messages to actor Z’s mailbox, then without any other mechanism to impose a specific order, the messages could be received by Z in either order.

The server-side of the actor is the code that runs in the actor’s own go-routine. There are 3 call-backs that the server-side code can provide:

  1. Init(arguments) error. This is always the first thing that gets called by the actor’s new go-routine as soon as it gets created. It should do any setup work necessary, any state initialisation that’s required. If it returns a non-nil error then the actor will terminate.
  2. HandleMsg(msg) error. This is called for each message received from the actor’s mailbox. As part of processing the received message, the actor can: send messages to other actors; it can spawn new actors; depending on the type of message it send a reply to the message; and it can choose to terminate: if the method returns a non-nil error then the actor will terminate.
  3. Terminate(reason). The actor terminates when either Init or HandleMsg return a non-nil error or panic. When that occurs, the actor’s go-routine will call this method. If the server-side wants to terminate “normally”, then there is a special error value ErrNormalActorTermination which provides a non-nil error for triggering termination but does not cause any alarming details to be logged.

Some methods are always available to every actor: suitable code provided in both the base client and base server types. One of those is the client TerminateSync method. This sends a message into the mailbox, asking the actor to terminate. The default server-side handler for this message returns the ErrNormalActorTermination error which causes the actor to terminate. The client-side method waits until the actor has finished running its Terminate method before it returns. TerminateSync is also idempotent.

Other client-side methods that are always available include OnTermination(callback) which allows you to register a callback to be invoked when the actor terminates. Through this mechanism you can monitor actors and receive the reason they terminated.

When an actor terminates, any messages left in its mailbox are discarded. For any of these messages which required a reply, the client-side API is able to detect that the server-side terminated before it received the message. Similarly, if the client-side code attempts to post any message to a mailbox after the server-side has terminated, then it can immediately spot that the actor has already terminated and that the message can’t possibly be processed. But, successfully posting a message into a mailbox is no guarantee that the message will be received by the server-side and processed: the server-side could terminate before that occurs.

Shopping-Basket Example

Let’s build an actor that represents a shopping basket. Lots of people will be able to concurrently add to the shopping basket, and the shopping basket actor will be able to answer questions about the number of items in it, and the cost of the entire basket. First, the server-side:

import (
   "github.com/rs/zerolog"
   "wellquite.org/actors"
   "wellquite.org/actors/mailbox"
)

type basketServer struct {
   actors.ServerBase
   items map[*Item]uint
}

var _ actors.Server = (*basketServer)(nil)

func (self *basketServer) Init(log zerolog.Logger, mailboxReader *mailbox.MailboxReader, selfClient *actors.ClientBase) (err error) {
   self.items = make(map[*Item]uint)
   return self.ServerBase.Init(log, mailboxReader, selfClient)
}

type Item struct {
   Name string
   Cost uint
}

type summariseMsg struct {
   actors.MsgSyncBase
   ItemCount uint // reply field
   TotalCost uint // reply field
}

func (self *basketServer) HandleMsg(msg mailbox.Msg) (err error) {
   switch msgT := msg.(type) {
   case *Item:
      self.items[msgT] += 1
      return nil

   case *summariseMsg:
      totalCount := uint(0)
      totalCost := uint(0)
      for item, count := range self.items {
         totalCount += count
         totalCost += (count * item.Cost)
      }
      msgT.ItemCount = totalCount
      msgT.TotalCost = totalCost
      msgT.MarkProcessed()
      return nil

   default:
      return self.ServerBase.HandleMsg(msg)
   }
}

The basketServer embeds a ServerBase which provides default implementations of all the callbacks. I need to override two of the callbacks: Init and HandleMsg. Whenever I override one of these callbacks, I must make sure I also call the default embedded handler otherwise things will break. In HandleMsg there are two message types that this basket actor cares about:

  1. Receiving an *Item means adding the item to the basket. This is asynchronous; there is no reply to the caller. I just add the item into the map in the private server-side state.
  2. Receiving a *summariseMsg message. This needs a response: it is asking the actor to summarise what’s in the basket. The summariseMsg type has actors.MsgSyncBase embedded within it, which adds a little machinery to allow replies to be issued straight into the message itself. So the handler fills in a couple of fields in message, and then calls MarkProcessed on the message, which provides a signal that the client-side can now proceed and safely access the fields.

The client-side looks like this:

type BasketClient struct {
   *actors.ClientBase
}

func SpawnBasket(log zerolog.Logger) (*BasketClient, error) {
   clientBase, err := actors.Spawn(log, &basketServer{}, "basket")
   if err != nil {
      return nil, err
   }
   return &BasketClient{ClientBase: clientBase}, nil
}

func (self *BasketClient) AddItem(item *Item) {
   self.Send(item)
}

func (self *BasketClient) Summarise() (itemCount, totalCost uint, success bool) {
   msg := &summariseMsg{}
   if self.SendSync(msg, true) {
      return msg.ItemCount, msg.TotalCost, true
   } else {
      return 0, 0, false
   }
}

The BasketClient is the public API to this actor. It has two methods: AddItem and Summarise, neither of which expose the mailbox, or any implementation detail of the actor. The BasketClient type embeds a *ClientBase which is created by actors.Spawn. This *ClientBase value wraps the actor’s mailbox and provides useful methods like Send and SendSync. Whilst any value can be sent to an actor, only values which embed actors.MsgSyncBase can be passed to SendSync. SendSync takes a 2nd parameter, waitForReply, which if true causes the call to SendSync to block until the server has called MarkProcessed on the message. I set this to true so that when the call to SendSync returns, I know the server-side has processed the message and I can find the answers I’m looking for in the fields of the message. This is why it’s important that on the server-side, MarkProcessed is called after the fields have been filled in. SendSync also returns a boolean. If the value returned is false then it means the server terminated before it received and processed the message: i.e. the message was not processed.

Because BasketClient embeds an *actors.ClientBase value, it also gains TerminateSync and OnTermination methods.

Not a lot of code, but it shows all the key design points:

  • Client-side and server-side use different types (structs) so there’s no danger that private server-side state is publicly exposed in the client.
  • The server-side code and state is only ever run by a single go-routine, so there’s no need for any locks when manipulating private state.
  • The server-side embeds an actors.ServerBase (or similar) which makes my basketServer type a valid actor server.
  • The client-side uses the *actors.ClientBase it gets back from Spawn to post to the actor’s mailbox.
  • Any value of any type can be sent to an actor.
  • Values which embed actors.MsgSyncBase can also be sent to the actor using SendSync. This allows the client-side to block until the server-side has received the message, processed it, and called MarkProcessed on the message. This makes it safe to use a single message to both send values to, and receive values from the actor.

There is a little boilerplate, particularly on the server-side; and you have to remember to call the embedded default implementation of a callback when you override it. But there are only 3 callbacks, so hopefully it’s not too onerous nor the API too wide. The semantics around the synchronisation for both Spawn/Init and TerminateSync/Terminated provide important guarantees about the state of the actor.

Hierarchies of Actors

One very common pattern is to use an actor to manage a set of other (child) actors. I can ensure that:

  • If a child actor terminates with ErrNormalActorTermination (normal termination) then that is no cause for alarm and everything else continues to work.
  • If a child actor terminates for any other reason (abnormal termination) then the manager actor itself terminates.
  • Whenever the manager terminates, it makes sure that all its child actors have terminated.
  • Given the synchronous design of TerminateSync, calling TerminateSync on the manager will not return until all its children have also fully terminated too.

My actors library provides exactly these properties with its ManagerClient type, and SpawnManager function. I can adjust the spawning of baskets so that a new basket is a child of a manager:

func SpawnBasket(manager actors.ManagerClient, name string) (*BasketClient, error) {
   clientBase, err := manager.Spawn(&basketServer{}, name)
   if err != nil {
      return nil, err
   }
   return &BasketClient{ClientBase: clientBase}, nil
}

Instead of calling actors.Spawn, I call manager.Spawn. I can now create some sort of registry actor, which allows me to access baskets by basket identifier. Again, I’ll do the server-side first:

type basketRegistryServer struct {
   actors.ServerBase
   baskets map[uint64]*BasketClient
   manager *actors.ManagerClientBase
}

var _ actors.Server = (*basketRegistryServer)(nil)

func (self *basketRegistryServer) Init(log zerolog.Logger, mailboxReader *mailbox.MailboxReader, selfClient *actors.ClientBase) (err error) {
   self.baskets = make(map[uint64]*BasketClient)
   manager, err := actors.SpawnManager(log, "basket manager")
   if err != nil {
      return err
   }
   self.manager = manager
   return self.ServerBase.Init(log, mailboxReader, selfClient)
}

func (self *basketRegistryServer) Terminated(err error, caughtPanic interface{}) {
   self.manager.TerminateSync()
   self.ServerBase.Terminated(err, caughtPanic)
}

type ensureBasketMsg struct {
   actors.MsgSyncBase
   basketId uint64        // query field
   basket   *BasketClient // reply field
}

func (self *basketRegistryServer) HandleMsg(msg mailbox.Msg) (err error) {
   switch msgT := msg.(type) {
   case *ensureBasketMsg:
      basket, err := self.ensureBasket(msgT.basketId)
      if err != nil {
         return err
      }
      msgT.basket = basket
      msgT.MarkProcessed()
      return nil
   default:
      return self.ServerBase.HandleMsg(msg)
   }
}

func (self *basketRegistryServer) ensureBasket(basketId uint64) (basket *BasketClient, err error) {
   basket, found := self.baskets[basketId]
   if !found {
      basket, err = SpawnBasket(self.manager, fmt.Sprintf("basket(%d)", basketId))
      if err == nil {
         self.baskets[basketId] = basket
      }
   }
   return basket, err
}

The basketRegistryServer has two bits of private state this time, both of which get set up in Init: a map to allow me to find baskets by identifier, and the manager itself. I think of the manager as a child of the registry, and then the baskets will be children of the manager, as shown in figure 3.

The Basket Registry owns the Manager, which owns the Basket actors

Figure 3: Hierarchy of actors

Because there’s a manager and other child actors involved, I override Terminated too to make sure than when the registry terminates, it terminates the manager (which will in turn terminate all its children). As normal, I have to make sure I also call the default base implementation.

There’s only one message type the registry cares about currently: ensureBasketMsg which returns the basket for the given identifier, creating it if it doesn’t already exist. This is a message which gets a reply, hence embedding actors.MsgSyncBase. In the ensureBasket method, I spawn the new basket as a child of the registry’s manager. I also provide a name for the new basket actor which includes its identifier. This will show up in logs and generally makes tracing easier.

Next the client-side:

type BasketRegistryClient struct {
   *actors.ClientBase
}

func SpawnBasketRegistry(log zerolog.Logger) (*BasketRegistryClient, error) {
   clientBase, err := actors.Spawn(log, &basketRegistryServer{}, "basket registry")
   if err != nil {
      return nil, err
   }
   return &BasketRegistryClient{ClientBase: clientBase}, nil
}

func (self *BasketRegistryClient) EnsureBasket(basketId uint64) *BasketClient {
   msg := &ensureBasketMsg{}
   if self.SendSync(msg, true) {
      return msg.basket
   }
   return nil
}

A couple of improvements are needed to the server though:

  1. If a basket terminates abnormally, then the manager will terminate, which will cause all the baskets to be terminated. It would be good if this cascade continued up so that the registry terminates too. So I want the registry to observe the termination of the manager.
  2. If a basket terminates normally (perhaps the customer goes through the checkout and pays), there is currently no way for the registry to delete its reference to the basket from its map. So the registry should also observe the normal termination of each basket.

For both of these I need to use the OnTermination facility to create a subscription to observe the termination of an actor. Firstly for the manager, I can make a few changes to Init function in the registry:

func (self *basketRegistryServer) Init(log zerolog.Logger, mailboxReader *mailbox.MailboxReader, selfClient *actors.ClientBase) (err error) {
   self.baskets = make(map[uint64]*BasketClient)
   manager, err := actors.SpawnManager(log, "basket manager")
   if err != nil {
      return err
   }
   subscription := manager.OnTermination(func(subscription *actors.TerminationSubscription, err error, caughtPanic interface{}) {
      selfClient.TerminateSync()
   })
   if subscription == nil {
      return errors.New("Unable to create subscription to manager.")
   }
   self.manager = manager
   return self.ServerBase.Init(log, mailboxReader, selfClient)
}

The function that I pass to OnTermination will get run when the manager terminates. It will run in its own new go-routine. So to get it to terminate the registry, it uses the client for the registry to ask the registry to terminate. The effect is that if the manager terminates for any reason (which could be caused by a basket terminating abnormally) then the registry will be asked to terminate too. In this way, errors can propagate between actors, and you can make sure that if something goes wrong, actors can be terminated in a controlled and deterministic fashion.

Solving the second problem looks very similar, except I need to subscribe to each new basket. If the basket terminates, the callback will be run in a new go-routine as before, which means to tidy up the registry I need to send it a suitable message:

type deleteBasketMsg uint64

func (self *basketRegistryServer) ensureBasket(basketId uint64) (basket *BasketClient, err error) {
   basket, found := self.baskets[basketId]
   if !found {
      basket, err = SpawnBasket(self.manager, fmt.Sprintf("basket(%d)", basketId))
      if err == nil {
         self.baskets[basketId] = basket
         subscription := basket.OnTermination(func(subscription *actors.TerminationSubscription, err error, caughtPanic interface{}) {
            if err == actors.ErrNormalActorTermination {
               self.SelfClient.Send(deleteBasketMsg(basketId))
            }
         })
         if subscription == nil {
            return nil, errors.New("Unable to create subscription to basket")
         }
      }
   }
   return basket, err
}

I only bother sending the new deleteBasketMsg if the basket terminated normally: in all abnormal cases, the manager would also terminate, which I already detect and handle suitably. Lastly, I need to add the extra case to HandleMsg:

func (self *basketRegistryServer) HandleMsg(msg mailbox.Msg) (err error) {
   switch msgT := msg.(type) {
   case *ensureBasketMsg:
      basket, err := self.ensureBasket(msgT.basketId)
      if err != nil {
         return err
      }
      msgT.basket = basket
      msgT.MarkProcessed()
      return nil
   case deleteBasketMsg:
      delete(self.baskets, uint64(msgT))
      return nil
   default:
      return self.ServerBase.HandleMsg(msg)
   }
}

Managing child actors and observing their termination takes a little more code and effort. The reward is a sensible hierarchy of actors, and control over the way the program behaves when errors occur and propagate.

Back to the Distributed Editor

With all that covered, I can now return to the distributed editor I’ve been building and explain how I’ve used my Actors library to structure the server.

Back in part 3 I briefly talked about the URL structure for the HTTP server. I said:

The WebSockets will be available under /documents/$documentName.

What I want to achieve is:

  • One actor manages one document.
  • All of these actors are children of some registry actor which allows me to find such actors by document name.
  • Each of these actors only gets created when a WebSocket to that document name is created.
  • When there are no more WebSockets open for a particular document name, the corresponding actor terminates normally.
  • As long as a WebSocket connection stays alive, it sends messages it receives from the browser to the document’s actor, and updates that it receives from the document’s actor it sends down to the browser over the WebSocket.

The document.go file I linked to last week. Hopefully it’s now clear how that file is divided into the client-side API and server-side. The implementations of Init and HandleMsg should no longer be mysterious. There are 3 different messages the document actor understands:

  1. Messages from a browser, received via a WebSocket. These include edits, undo, and redo messages.
  2. documentSubscribeMsg and documentUnsubscribeMsg messages. These are used by the code that handles the WebSockets to subscribe to a particular document actor. You can see in the code for unsubscribe how if the number of subscribers drops to 0, then the actor will exit normally.

To create a subscription you provide a callback function. The document actor will invoke this function with the bytes that should be sent out of the WebSocket and down to the browser.

The document registry is structured almost exactly the same way as the basket registry from earlier.

That really just leaves the code that handles each WebSocket. It creates the subscription to the document actor and also sends to the document actor any messages it receives from the WebSocket that it’s able to decode. Defers make sure that whatever causes the WebSocket to close, the subscription to the document actor will be cancelled.


That covers pretty much all the interesting bits of the client and server. In about 850 lines of TypeScript and 1800 lines of Go I have a pretty reasonably engineered implementation of a distributed text editor that should be enough for Alice and Bob to start playing with.

What remains to be covered is testing. Testing a distributed system is always fun: as I mentioned earlier, bugs frequently only show themselves if certain events happen in a particular order. I find it very effective to create tests that can generate a random stream of events and feed that into the system. How to do that and how to verify the effect of those events is a subject for next week.