The Database for Social Software

By Lawrence Lundy‑Bryan on 03 March 2022

Moving from one-at-a-time to all-at-the-same-time software. Note that this post was originally published on the Lunar Ventures medium blog.

Single-player and one-at-a-time software

How often have you lost data when collaborating with colleagues on a Microsoft document? Words just sort of go missing. Or, when more than a few people want to edit a Google slide, you get the “this document is overloaded, so editing is disabled” message. Most of the software we have today is designed for one-at-a-time use. This is a bit frustrating, sure, but actually this one-at-a-time software paradigm severely limits what we can do online.

Do your work, then email it to someone else to do theirs. That was the workflow (still is for lawyers). Cloud services and SaaS like Google Docs removed the need for email, but the software paradigm didn’t change. New tools like Miro and Figma push the limits on collaboration; they are closer to few-at-a-time. And for users, it feels like magic. There is something different about seeing other people’s cursors move around the screen with yours. Technologies like OT (operational transform) and CRDTs (conflict-free replicated data types) allow developers to resolve data conflicts and incorporate some collaborative features into their applications but are quite limited. CRDTs in particular are complicated, hard to prove they are working correctly, and difficult to get to play nicely with storage and other server-side services. Because getting them to work is so hard, we only see distributed systems experts even attempting to use them.

Why does single-player software dominate? Well fundamentally, because of the CAP theorem). To oversimplify, assuming the network is not perfect and sometimes has “hiccups”, developers must choose between consistency; all clients receive the most recent write or receive an error, and availability; all reads return data, but it might not be the most recent data. As the Internet got faster and computers more powerful, users demanded websites, apps and games to work immediately. This demand led most applications to ensure data was available as quickly as possible, so developers chose availability and traded off consistency. However, some sectors chose differently: finance and e-commerce applications need to make sure accounts and balances are always in sync and must prioritise consistency. Neither side is quite happy with the outcome. Fundamentally, we still live in a single-player world, which tries to look like multiplayer, and fails in subtle ways.

To move beyond single-player, we need infrastructure for strong consistency and high availability; concurrency at low latency.

Multiplayer and all-at-the-same-time software

What if developers didn’t have to choose between availability or consistency?

Our bet is that deeply integrating CRDTs into the backend is the way forward. This is what Vaxine, our latest portfolio company, is planning to do. Founded by James Arthur and supported by academics including Annette Bieniusa as well as Mark Shapiro, the co-inventor of CRDTs, Vaxine is a rich-CDRT database for collaborative backend applications. Vaxine ’s mission is to make it easy for developers to integrate collaborative functionality into their apps. The aim is to minimise network latency on both the read and write paths for geo-distributed applications without compromising on data consistency, session guarantees or real-world constraints and invariants. The app developer specifies the business logic, and Vaxine takes care of the backend CRDT stuff.

This is useful, for example, in computer games, where you need real-time collaboration on a structured model that requires an invariant. Say two Minecraft players both try to move the same block simultaneously. Or two players in an MMORPG both pick up a sword at the same time, during a “network hiccup”. In a traditional geo-distributed database, there are replication consensus and locking constraints: both players will think they picked up the sword, but a minute later, one of them will discover they no longer have the sword, and that the monsters they killed with this sword have come back to life. Or that a minecraft block has disappeared from the inside of the new building they constructed. If the game was instead implemented as a conflict-free system, this wouldn’t happen: the game would only show one of the players they picked up a sword: but it would take several seconds to pick up any sword or block, in order for the database to make 100% sure that no conflicts arose. Both solutions suck.

But if the game used Vaxine, you get the best of both worlds: conflict-free collaboration with standard database guarantees. The game developer specifies what objects and behaviours must be synced across the system, and which ones may be kept “tentative”. The dev might specify that players can pick up a sword immediately, but can only equip the sword once the database made sure there are no “extra copies” of it lying around. Or the dev can specify that two players may slay monsters with two :”copies” of a sword, and if a duplication occurs, one of them will get a “consolation prize”, and all monsters slain in the meantime will stay slain. Vaxine helps the developer specify these behaviours, and the “business logic” for dealing with conflicts, enabling developers to get the best of both worlds,

Social software

Vaxine will make it much easier to build strong collaborative software. First up is creation software. We expect a wave of new applications by lowering the barrier to building collaborative features. Users are used to data just sort of getting lost when using collaborative creation tools. But that will be a thing of the past. By itself, this will make our lives like 2% better: nothing to write home about. But imagine the new possibilities when applications can handle thousands of concurrent editors. We have no idea what can be done when we give communities real-time collaboration tools. I think we will be surprised at just how limiting our one-at-a-time software has been for a whole host of processes. It’s a step-change in functionality that will support the movement to a distributed and remote workforce. The idea that you have to get around a whiteboard to figure things out will be a relic of the past by combining tools like Vaxine and 3D interfaces.

If you zoom out, we’re currently in the process of replicating our physical world digitally. We’re sort of at a weird moment where our backend and frontend technologies are limiting the fidelity of our digital worlds. Our interfaces are 2D, and our 3D interfaces have extremely low fidelity. And our backend can’t give us real-time global shared experiences. So we are left with this frustrating feeling when we interact digitally. The low fidelity makes virtual spaces lame, but the lack of high-quality social experiences makes them isolating.

We’re investing in Vaxine because we recognise that collaboration software is hard to build. We think a database that makes deploying CRDTs easier lowers the barriers to creating richer collaborative experiences. This has clear benefits for productivity software, gaming and virtual worlds. But more than that, we think it enables something more profound, the ability to be social in virtual spaces.

Consistency, availability, latency, concurrency, and CRDTs are technical terms hiding a deeper reality. Most of our digital tools today limit us to one–at-a-time experiences, an inherently isolating experience. Vaxine supports a new paradigm: all-at-the-same-time software; more interactive, more engaging and deeply social.


Introducing Rich-CRDTs

By Valter Balegas on 03 May 2022

Rich-CRDTs are conflict-free data types (“CRDTs”) extended to provide additional (“Rich”) database guarantees. These guarantees, such as constraints and referential integrity, make building applications under eventual consistency much simpler.

Subscribe to the Vaxine blog and newsletter.

Get technical updates and articles straight to your inbox.

We will never share your email.

Or go old-school with some really simple syndication.

Subscribe

Subscribe using RSS

Go old-school with some really simple syndication.

Subscribe