Skip to content

Instantly share code, notes, and snippets.

@pesterhazy
Last active October 25, 2025 03:18
Show Gist options
  • Save pesterhazy/3e039677f2e314cb77ffe3497ebca07b to your computer and use it in GitHub Desktop.
Save pesterhazy/3e039677f2e314cb77ffe3497ebca07b to your computer and use it in GitHub Desktop.

Revisions

  1. pesterhazy revised this gist Nov 14, 2024. 1 changed file with 4 additions and 0 deletions.
    4 changes: 4 additions & 0 deletions building-sync-systems.md
    Original file line number Diff line number Diff line change
    @@ -75,6 +75,10 @@ Every app has slightly different needs. So my guess is that you will need to bui
    - [MongoDB Device Sync](https://www.mongodb.com/products/platform/atlas-app-services/device-sync)

    Syncs a MongoDB database with local devices based on the [Realm embedded database](https://www.mongodb.com/docs/realm/)

    - [Instant](https://github.com/instantdb/instant)

    Sync system as a service inspired by Clojure and Datomic

    ## Databases

  2. pesterhazy revised this gist Mar 28, 2024. 1 changed file with 4 additions and 0 deletions.
    4 changes: 4 additions & 0 deletions building-sync-systems.md
    Original file line number Diff line number Diff line change
    @@ -71,6 +71,10 @@ Every app has slightly different needs. So my guess is that you will need to bui
    - [PowerSync](https://docs.powersync.com/overview/powersync-philosophy)

    Another offering for syncing Postgres with a local SQLite.

    - [MongoDB Device Sync](https://www.mongodb.com/products/platform/atlas-app-services/device-sync)

    Syncs a MongoDB database with local devices based on the [Realm embedded database](https://www.mongodb.com/docs/realm/)

    ## Databases

  3. pesterhazy revised this gist Mar 28, 2024. 1 changed file with 8 additions and 0 deletions.
    8 changes: 8 additions & 0 deletions building-sync-systems.md
    Original file line number Diff line number Diff line change
    @@ -64,6 +64,14 @@ Every app has slightly different needs. So my guess is that you will need to bui

    > Fireproof is an embedded database for collaborative applications. Install it in your front-end app, or use it in any serverless cloud or edge function. Fireproof’s document API includes live updates, flexible queries, binary attachments, encrypted block replication, and multi-user sync. Fireproof enables developers to ship interactive features faster in any deployment environment.
    - [Electric SQL](https://electric-sql.com/)

    SQLite in the browser automatically and seemlessly synced with Postgres. That's a big promise – can ElectricSQL deliver?

    - [PowerSync](https://docs.powersync.com/overview/powersync-philosophy)

    Another offering for syncing Postgres with a local SQLite.

    ## Databases

    When you're building a sync engine, you're essentially building a database with replication - whether you realize it or not. So it's a good idea to review some of the literature on databases and replication.
  4. pesterhazy revised this gist Mar 21, 2024. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion building-sync-systems.md
    Original file line number Diff line number Diff line change
    @@ -22,7 +22,7 @@ So you want to write a sync system for a web app with offline and realtime suppo

    More recently, Tuomas described things the team learned in 3 years of building a sync engine in a [Twitter thread](https://twitter.com/artman/status/1558081796914483201).

    - [Scaling Linear's sync system](Scaling the Linear Sync Engine)
    - [Scaling the Linear Sync Engine](https://linear.app/blog/scaling-the-linear-sync-engine))](https://linear.app/blog/scaling-the-linear-sync-engine)

    Follow-up to the previous video

  5. pesterhazy revised this gist Mar 13, 2024. 1 changed file with 4 additions and 0 deletions.
    4 changes: 4 additions & 0 deletions building-sync-systems.md
    Original file line number Diff line number Diff line change
    @@ -122,6 +122,10 @@ While CRDTs offer advantages like operation without a centralized server, they o
    - Martin Kleppmann, Alastair R. Beresford: "A Conflict-Free Replicated JSON Datatype". IEEE Transactions on Parallel and Distributed Systems 28(10):2733–2746, April 2017 ([link](https://arxiv.org/abs/1608.03960))

    Very accessible paper on what ended up being published as [automerge](https://github.com/automerge/automerge)

    - Marijn Haverbeke, [Collaborative Editing in Codemirror](https://marijnhaverbeke.nl/blog/collaborative-editing-cm.html)

    Pragmatic discussion of the (according to the author) false dichotomy between OT and CRDTs, in the context of collaborative text editing

    Many more papers and other resources are avaialble on [crdt.tech](https://crdt.tech/resources).

  6. pesterhazy revised this gist Jan 24, 2024. 1 changed file with 4 additions and 0 deletions.
    4 changes: 4 additions & 0 deletions building-sync-systems.md
    Original file line number Diff line number Diff line change
    @@ -44,6 +44,10 @@ Every app has slightly different needs. So my guess is that you will need to bui

    [Pouchdb](https://pouchdb.com/) offers an in-browser database with server replication, and did so before it was cool. It's still pretty impressive today, and free software.

    - RxDB

    [RxDB](https://rxdb.info/) is a local-first, NoSQL-database for JavaScript Applications like Websites, hybrid Apps, Electron-Apps, Progressive Web Apps, Deno and Node.js.

    - [Fluid Framework](https://fluidframework.com/docs/)

    Fluid Framework is a "collection of client libraries for distributing and synchronizing shared state. These libraries allow multiple clients to simultaneously create and operate on shared data structures using coding patterns similar to those used to work with local data." Fluid is developed by Microsoft.
  7. pesterhazy revised this gist Jan 24, 2024. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions building-sync-systems.md
    Original file line number Diff line number Diff line change
    @@ -81,6 +81,7 @@ When you're building a sync engine, you're essentially building a database with
    Some SaaS are trying to offer multiplayer, syncing, caching etc as a service. No clear winner yet. But even if you're not going to use these, it's worth reading the API docs for inspiration.

    - [replicache](https://replicache.dev/) "is a JavaScript framework for building high-performance, offline-capable, collaborative web apps"
    - [reflect](https://reflect.net/) "is a high-performance sync for multiplayer web apps"
    - [Liveblocks](https://liveblocks.io/) "is a set of APIs and tools that helps you create performant and reliable multiplayer experiences in minutes"
    - [Supabase](https://supabase.com/docs/guides/realtime)

  8. pesterhazy revised this gist Jan 24, 2024. 1 changed file with 5 additions and 1 deletion.
    6 changes: 5 additions & 1 deletion building-sync-systems.md
    Original file line number Diff line number Diff line change
    @@ -54,7 +54,11 @@ Every app has slightly different needs. So my guess is that you will need to bui

    - [AWS AppSync](https://docs.aws.amazon.com/appsync/latest/devguide/aws-appsync-real-time-data.html)

    AppSync is AWS attempt at offering automated realtime GraphQL updates. It's one of the weirder AWS offerings. As is often the case with AWS marketing-driven producs, it's hard to figure out from the docs if it's a viable alternative for a given problem.
    AppSync is AWS's attempt at offering automated realtime GraphQL updates. It's one of the weirder AWS offerings. As is often the case with AWS marketing-driven producs, it's hard to figure out from the docs if it's a viable alternative for a given problem.

    - [fireproof](https://use-fireproof.com/)

    > Fireproof is an embedded database for collaborative applications. Install it in your front-end app, or use it in any serverless cloud or edge function. Fireproof’s document API includes live updates, flexible queries, binary attachments, encrypted block replication, and multi-user sync. Fireproof enables developers to ship interactive features faster in any deployment environment.
    ## Databases

  9. pesterhazy revised this gist Jun 30, 2023. 1 changed file with 4 additions and 0 deletions.
    4 changes: 4 additions & 0 deletions building-sync-systems.md
    Original file line number Diff line number Diff line change
    @@ -22,6 +22,10 @@ So you want to write a sync system for a web app with offline and realtime suppo

    More recently, Tuomas described things the team learned in 3 years of building a sync engine in a [Twitter thread](https://twitter.com/artman/status/1558081796914483201).

    - [Scaling Linear's sync system](Scaling the Linear Sync Engine)

    Follow-up to the previous video

    - [Muse's Sync](https://museapp.com/podcast/56-sync/) (Adan Wulf, Addam Wiggins, Mark Mcgranaghan)

    In-depth discussion of the why and how behind Muse's realtime sync system
  10. pesterhazy revised this gist Oct 17, 2022. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion building-sync-systems.md
    Original file line number Diff line number Diff line change
    @@ -72,7 +72,7 @@ When you're building a sync engine, you're essentially building a database with

    Some SaaS are trying to offer multiplayer, syncing, caching etc as a service. No clear winner yet. But even if you're not going to use these, it's worth reading the API docs for inspiration.

    - [replicache](https://replicache.dev/)
    - [replicache](https://replicache.dev/) "is a JavaScript framework for building high-performance, offline-capable, collaborative web apps"
    - [Liveblocks](https://liveblocks.io/) "is a set of APIs and tools that helps you create performant and reliable multiplayer experiences in minutes"
    - [Supabase](https://supabase.com/docs/guides/realtime)

  11. pesterhazy revised this gist Oct 17, 2022. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions building-sync-systems.md
    Original file line number Diff line number Diff line change
    @@ -73,6 +73,7 @@ When you're building a sync engine, you're essentially building a database with
    Some SaaS are trying to offer multiplayer, syncing, caching etc as a service. No clear winner yet. But even if you're not going to use these, it's worth reading the API docs for inspiration.

    - [replicache](https://replicache.dev/)
    - [Liveblocks](https://liveblocks.io/) "is a set of APIs and tools that helps you create performant and reliable multiplayer experiences in minutes"
    - [Supabase](https://supabase.com/docs/guides/realtime)

    ## Realtime updates
  12. pesterhazy revised this gist Aug 31, 2022. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion building-sync-systems.md
    Original file line number Diff line number Diff line change
    @@ -66,7 +66,7 @@ When you're building a sync engine, you're essentially building a database with

    [Datascript](https://github.com/tonsky/datascript) is a client-side version of Datomic (but there's no built-in sync engine).

    - Postgres is the most mature RDBMS out there (but doesn't help you replicate data to the client). Learn as much as you can from the decades of research and practical wisdom that went into Postgres. The [chapter in the Postgres manual on Isolation levels](https://www.postgresql.org/docs/current/mvcc.html) is excellent
    - Postgres is the most mature RDBMS out there (but doesn't by itself help you replicate data to the client). There's a lot to learn from from decades of research and production experience. The [chapter in the Postgres manual on Isolation levels](https://www.postgresql.org/docs/current/mvcc.html) is excellent.

    ## SaaS in the sync space

  13. pesterhazy revised this gist Aug 31, 2022. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion building-sync-systems.md
    Original file line number Diff line number Diff line change
    @@ -18,7 +18,7 @@ So you want to write a sync system for a web app with offline and realtime suppo

    - [Linear's realtime sync system](https://www.youtube.com/watch?v=WxK11RsLqp4&t=2169s&ab_channel=Fraktio) (Tuomas Artman)

    Linear has a sophisticated sync system with offline capabilities. This talks describes how it works.
    Linear has a sophisticated sync system with offline capabilities. This 2020 talk describes how it works.

    More recently, Tuomas described things the team learned in 3 years of building a sync engine in a [Twitter thread](https://twitter.com/artman/status/1558081796914483201).

  14. pesterhazy revised this gist Aug 31, 2022. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions building-sync-systems.md
    Original file line number Diff line number Diff line change
    @@ -20,6 +20,8 @@ So you want to write a sync system for a web app with offline and realtime suppo

    Linear has a sophisticated sync system with offline capabilities. This talks describes how it works.

    More recently, Tuomas described things the team learned in 3 years of building a sync engine in a [Twitter thread](https://twitter.com/artman/status/1558081796914483201).

    - [Muse's Sync](https://museapp.com/podcast/56-sync/) (Adan Wulf, Addam Wiggins, Mark Mcgranaghan)

    In-depth discussion of the why and how behind Muse's realtime sync system
  15. pesterhazy revised this gist Aug 31, 2022. 1 changed file with 4 additions and 0 deletions.
    4 changes: 4 additions & 0 deletions building-sync-systems.md
    Original file line number Diff line number Diff line change
    @@ -38,6 +38,10 @@ Every app has slightly different needs. So my guess is that you will need to bui

    [Pouchdb](https://pouchdb.com/) offers an in-browser database with server replication, and did so before it was cool. It's still pretty impressive today, and free software.

    - [Fluid Framework](https://fluidframework.com/docs/)

    Fluid Framework is a "collection of client libraries for distributing and synchronizing shared state. These libraries allow multiple clients to simultaneously create and operate on shared data structures using coding patterns similar to those used to work with local data." Fluid is developed by Microsoft.

    - [WatermelonDB](https://github.com/Nozbe/WatermelonDB) - an offline-first local db (React and ReactNative) with capabilities based on IndexedDB/SQLite

    WatermelonDB offers a [client-side sync implementation and API](https://nozbe.github.io/WatermelonDB/Advanced/Sync.html); you still need to write your own backend.
  16. pesterhazy revised this gist Aug 31, 2022. 1 changed file with 4 additions and 0 deletions.
    4 changes: 4 additions & 0 deletions building-sync-systems.md
    Original file line number Diff line number Diff line change
    @@ -81,6 +81,10 @@ Instead of rolling your own websocket server, consider using a hosted service li

    ## Literature

    - Differential Synchronization, Neil Fraser, 2009 ([link](https://neil.fraser.name/writing/sync/eng047-fraser.pdf))

    There's also a [youtube talk by the author](https://www.youtube.com/watch?v=S2Hp_1jqpY8&ab_channel=GoogleTechTalks) on the same topic

    - Kleppmann, [Designing Data-Intensive Applications](https://dataintensive.net/), O'Reilly (2017)

    If you don't have this book yet, drop everything and read it now. Every chapter is full of insights and summaries of how to use and build databases, much of it applicable to sync engines.
  17. pesterhazy revised this gist Aug 31, 2022. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions building-sync-systems.md
    Original file line number Diff line number Diff line change
    @@ -93,6 +93,8 @@ Instead of rolling your own websocket server, consider using a hosted service li

    I recommend learning about CRDTs. Not because a sync system should be built on CRDTs (it probably shouldn't unless you're building a truly decentralized, peer-to-peer system), but because everyone's talking about them and it's useful to understand their limitations (and strengths). Also the literature contains many concepts that are helpful even if you don't need costly decentralization because can rely on a centralized server.

    While CRDTs offer advantages like operation without a centralized server, they often suffer from problems with regard to performance, memory usage and implementation complexity.

    - Martin Kleppmann, Alastair R. Beresford: "A Conflict-Free Replicated JSON Datatype". IEEE Transactions on Parallel and Distributed Systems 28(10):2733–2746, April 2017 ([link](https://arxiv.org/abs/1608.03960))

    Very accessible paper on what ended up being published as [automerge](https://github.com/automerge/automerge)
  18. pesterhazy revised this gist Aug 31, 2022. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion building-sync-systems.md
    Original file line number Diff line number Diff line change
    @@ -38,7 +38,7 @@ Every app has slightly different needs. So my guess is that you will need to bui

    [Pouchdb](https://pouchdb.com/) offers an in-browser database with server replication, and did so before it was cool. It's still pretty impressive today, and free software.

    - [WatermelonDB] - an offline-first local db (React and ReactNative) with capabilities based on IndexedDB/SQLite
    - [WatermelonDB](https://github.com/Nozbe/WatermelonDB) - an offline-first local db (React and ReactNative) with capabilities based on IndexedDB/SQLite

    WatermelonDB offers a [client-side sync implementation and API](https://nozbe.github.io/WatermelonDB/Advanced/Sync.html); you still need to write your own backend.

  19. pesterhazy revised this gist Aug 31, 2022. 1 changed file with 4 additions and 0 deletions.
    4 changes: 4 additions & 0 deletions building-sync-systems.md
    Original file line number Diff line number Diff line change
    @@ -38,6 +38,10 @@ Every app has slightly different needs. So my guess is that you will need to bui

    [Pouchdb](https://pouchdb.com/) offers an in-browser database with server replication, and did so before it was cool. It's still pretty impressive today, and free software.

    - [WatermelonDB] - an offline-first local db (React and ReactNative) with capabilities based on IndexedDB/SQLite

    WatermelonDB offers a [client-side sync implementation and API](https://nozbe.github.io/WatermelonDB/Advanced/Sync.html); you still need to write your own backend.

    - [AWS AppSync](https://docs.aws.amazon.com/appsync/latest/devguide/aws-appsync-real-time-data.html)

    AppSync is AWS attempt at offering automated realtime GraphQL updates. It's one of the weirder AWS offerings. As is often the case with AWS marketing-driven producs, it's hard to figure out from the docs if it's a viable alternative for a given problem.
  20. pesterhazy revised this gist Aug 31, 2022. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions building-sync-systems.md
    Original file line number Diff line number Diff line change
    @@ -69,6 +69,8 @@ Some SaaS are trying to offer multiplayer, syncing, caching etc as a service. No

    The simplest way to replicate data is for client to poll for updates every second or so. But that introduces unnecessary server load and >1 s latencies, so eventually you will want to switch to a push model. On the web that usually means using [Websockets](https://developer.mozilla.org/en-US/docs/Web/API/WebSocket) or [Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events) to allow the server to push updates to the clients as they arrive.

    Notice that realtime updates are conceptually distinct from data replication - you can simplify the system by building a polling based solution first and adding push updates as a separate step. The realtime update can be in the form of a lightweight _shoulder tap_, a message that wakes up the client, causing it to pull the latest updates from the server. Realtime notificaitons can be seen as optional - when the websocket connection is not available, you can fall back to polling.

    Instead of rolling your own websocket server, consider using a hosted service like [pusher.com](https://pusher.com/) or [ably.com](https://ably.com/). They do all the heavy lifting for you (like keeping thousands of simultaneous TCP connections open) and provide a request/response style interface for your server to send messages to connected clients.

    - [LiveGraph: real-time data fetching at Figma](https://www.figma.com/blog/livegraph-real-time-data-fetching-at-figma/)
  21. pesterhazy revised this gist Aug 31, 2022. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions building-sync-systems.md
    Original file line number Diff line number Diff line change
    @@ -65,9 +65,9 @@ Some SaaS are trying to offer multiplayer, syncing, caching etc as a service. No
    - [replicache](https://replicache.dev/)
    - [Supabase](https://supabase.com/docs/guides/realtime)

    ## Realtime sync
    ## Realtime updates

    The simplest way to replicate data is for client to poll for updates every second or so. But that introduces unnecessary server load and latency, so eventually you will want to switch to push based updates. On the web that usually means using [Websockets](https://developer.mozilla.org/en-US/docs/Web/API/WebSocket) or [Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events) to allow the server to push updates to the clients as they arrive.
    The simplest way to replicate data is for client to poll for updates every second or so. But that introduces unnecessary server load and >1 s latencies, so eventually you will want to switch to a push model. On the web that usually means using [Websockets](https://developer.mozilla.org/en-US/docs/Web/API/WebSocket) or [Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events) to allow the server to push updates to the clients as they arrive.

    Instead of rolling your own websocket server, consider using a hosted service like [pusher.com](https://pusher.com/) or [ably.com](https://ably.com/). They do all the heavy lifting for you (like keeping thousands of simultaneous TCP connections open) and provide a request/response style interface for your server to send messages to connected clients.

  22. pesterhazy revised this gist Aug 31, 2022. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion building-sync-systems.md
    Original file line number Diff line number Diff line change
    @@ -77,7 +77,7 @@ Instead of rolling your own websocket server, consider using a hosted service li

    - Kleppmann, [Designing Data-Intensive Applications](https://dataintensive.net/), O'Reilly (2017)

    If you haven't read this book, drop everything and start by reading it. Every chapter is full of insights and summaries of how to use and build databases (which is what you're doing essentially).
    If you don't have this book yet, drop everything and read it now. Every chapter is full of insights and summaries of how to use and build databases, much of it applicable to sync engines.

    [Local-first software](https://www.inkandswitch.com/local-first/) is a rebranding of offline-first systems. The name emphasizes the philosophical stance that data lives on your device first and foremost, with implications for privacy, ownership and decentralization.

  23. pesterhazy revised this gist Aug 31, 2022. 1 changed file with 5 additions and 1 deletion.
    6 changes: 5 additions & 1 deletion building-sync-systems.md
    Original file line number Diff line number Diff line change
    @@ -36,7 +36,11 @@ Every app has slightly different needs. So my guess is that you will need to bui

    Couchdb is a database based around replication. The [CouchDB book](https://guide.couchdb.org/), while a bit dated, is well written and worth reading.

    [Pouchdb](https://pouchdb.com/) offered a local Javascript database with server replication before it was cool. It's still pretty impressive today, and free software.
    [Pouchdb](https://pouchdb.com/) offers an in-browser database with server replication, and did so before it was cool. It's still pretty impressive today, and free software.

    - [AWS AppSync](https://docs.aws.amazon.com/appsync/latest/devguide/aws-appsync-real-time-data.html)

    AppSync is AWS attempt at offering automated realtime GraphQL updates. It's one of the weirder AWS offerings. As is often the case with AWS marketing-driven producs, it's hard to figure out from the docs if it's a viable alternative for a given problem.

    ## Databases

  24. pesterhazy revised this gist Aug 31, 2022. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion building-sync-systems.md
    Original file line number Diff line number Diff line change
    @@ -75,7 +75,7 @@ Instead of rolling your own websocket server, consider using a hosted service li

    If you haven't read this book, drop everything and start by reading it. Every chapter is full of insights and summaries of how to use and build databases (which is what you're doing essentially).

    [Local-first software](https://www.inkandswitch.com/local-first/) is a new name for offline-first systems. The name emphasizes the philosophical stance that data lives on your device first and foremost, with privacy and decentralization advantages.
    [Local-first software](https://www.inkandswitch.com/local-first/) is a rebranding of offline-first systems. The name emphasizes the philosophical stance that data lives on your device first and foremost, with implications for privacy, ownership and decentralization.

    - Martin Kleppmann, Adam Wiggins, Peter van Hardenberg, and Mark McGranaghan: “Local-first software: You own your data, in spite of the cloud”. ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward! ’19), October 2019 ([link](https://martin.kleppmann.com/papers/local-first.pdf))

  25. pesterhazy revised this gist Aug 31, 2022. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions building-sync-systems.md
    Original file line number Diff line number Diff line change
    @@ -67,6 +67,8 @@ The simplest way to replicate data is for client to poll for updates every secon

    Instead of rolling your own websocket server, consider using a hosted service like [pusher.com](https://pusher.com/) or [ably.com](https://ably.com/). They do all the heavy lifting for you (like keeping thousands of simultaneous TCP connections open) and provide a request/response style interface for your server to send messages to connected clients.

    - [LiveGraph: real-time data fetching at Figma](https://www.figma.com/blog/livegraph-real-time-data-fetching-at-figma/)

    ## Literature

    - Kleppmann, [Designing Data-Intensive Applications](https://dataintensive.net/), O'Reilly (2017)
  26. pesterhazy revised this gist Aug 31, 2022. 1 changed file with 3 additions and 3 deletions.
    6 changes: 3 additions & 3 deletions building-sync-systems.md
    Original file line number Diff line number Diff line change
    @@ -67,13 +67,13 @@ The simplest way to replicate data is for client to poll for updates every secon

    Instead of rolling your own websocket server, consider using a hosted service like [pusher.com](https://pusher.com/) or [ably.com](https://ably.com/). They do all the heavy lifting for you (like keeping thousands of simultaneous TCP connections open) and provide a request/response style interface for your server to send messages to connected clients.

    ## DDIA
    ## Literature

    - Kleppmann, [Designing Data-Intensive Applications](https://dataintensive.net/), O'Reilly (2017)

    If you haven't read this book, drop everything and start by reading it. Every chapter is full of insights and summaries of how to use and build databases (which is what you're doing essentially).

    ## Local-first software
    [Local-first software](https://www.inkandswitch.com/local-first/) is a new name for offline-first systems. The name emphasizes the philosophical stance that data lives on your device first and foremost, with privacy and decentralization advantages.

    - Martin Kleppmann, Adam Wiggins, Peter van Hardenberg, and Mark McGranaghan: “Local-first software: You own your data, in spite of the cloud”. ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward! ’19), October 2019 ([link](https://martin.kleppmann.com/papers/local-first.pdf))

  27. pesterhazy revised this gist Aug 31, 2022. 1 changed file with 6 additions and 0 deletions.
    6 changes: 6 additions & 0 deletions building-sync-systems.md
    Original file line number Diff line number Diff line change
    @@ -61,6 +61,12 @@ Some SaaS are trying to offer multiplayer, syncing, caching etc as a service. No
    - [replicache](https://replicache.dev/)
    - [Supabase](https://supabase.com/docs/guides/realtime)

    ## Realtime sync

    The simplest way to replicate data is for client to poll for updates every second or so. But that introduces unnecessary server load and latency, so eventually you will want to switch to push based updates. On the web that usually means using [Websockets](https://developer.mozilla.org/en-US/docs/Web/API/WebSocket) or [Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events) to allow the server to push updates to the clients as they arrive.

    Instead of rolling your own websocket server, consider using a hosted service like [pusher.com](https://pusher.com/) or [ably.com](https://ably.com/). They do all the heavy lifting for you (like keeping thousands of simultaneous TCP connections open) and provide a request/response style interface for your server to send messages to connected clients.

    ## DDIA

    - Kleppmann, [Designing Data-Intensive Applications](https://dataintensive.net/), O'Reilly (2017)
  28. pesterhazy revised this gist Aug 31, 2022. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions building-sync-systems.md
    Original file line number Diff line number Diff line change
    @@ -24,9 +24,9 @@ So you want to write a sync system for a web app with offline and realtime suppo

    In-depth discussion of the why and how behind Muse's realtime sync system

    ## Exisiting sync systems
    ## Off-the-shelf sync systems

    Every app has slightly different needs. So my guess is that you will need to build your own system. But it's worth taking a look at pre-built sync engines, either as buy-over-build or to steal ideas:
    Every app has slightly different needs. So my guess is that you will need to build your own system. But it's worth taking a look at off-the-shelf sync engines, either as buy-over-build or to steal ideas:

    - Firebase

  29. pesterhazy revised this gist Aug 31, 2022. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions building-sync-systems.md
    Original file line number Diff line number Diff line change
    @@ -20,9 +20,9 @@ So you want to write a sync system for a web app with offline and realtime suppo

    Linear has a sophisticated sync system with offline capabilities. This talks describes how it works.

    - [Muse's Sync](https://museapp.com/podcast/56-sync/)
    - [Muse's Sync](https://museapp.com/podcast/56-sync/) (Adan Wulf, Addam Wiggins, Mark Mcgranaghan)

    In-depth discussion of the why and how of the Muse's realtime sync system
    In-depth discussion of the why and how behind Muse's realtime sync system

    ## Exisiting sync systems

  30. pesterhazy revised this gist Aug 31, 2022. 1 changed file with 9 additions and 4 deletions.
    13 changes: 9 additions & 4 deletions building-sync-systems.md
    Original file line number Diff line number Diff line change
    @@ -1,15 +1,20 @@
    So you want to write a sync system for a web app with offline and realtime support? Good luck. You might find the following resources useful.

    ## Sync systems in the industry
    ## Overview articles

    - [How Figma’s multiplayer technology works](https://www.figma.com/blog/how-figmas-multiplayer-technology-works/) (Evan Wallace)
    - [Database in a browser, a spec](https://stopa.io/post/279) (Stepan Parunashvili)

    Fantastic summary of Figma's sync system and why they ultimately rejected fully decentralized CRDTs while still using some ideas from the CRDT literature.
    What problem are we trying to solve with a sync system?

    - [Database in a browser, a spec](https://stopa.io/post/279) (Stepan Parunashvili)
    - [The web of tomorrow](https://tonsky.me/blog/the-web-after-tomorrow/) (Nikita Prokopov)

    Lots of ideas. Published in 2015, maybe it was ahead of its time?

    ## Sync engines in the industry

    - [How Figma’s multiplayer technology works](https://www.figma.com/blog/how-figmas-multiplayer-technology-works/) (Evan Wallace)

    Fantastic summary of Figma's sync system and why they ultimately rejected fully decentralized CRDTs while still using some ideas from the CRDT literature.

    - [Linear's realtime sync system](https://www.youtube.com/watch?v=WxK11RsLqp4&t=2169s&ab_channel=Fraktio) (Tuomas Artman)