@rikishi/watermelondb

<!DOCTYPE HTML> <html lang="en" class="sidebar-visible no-js light"> <head>  <meta charset="UTF-8"> <title>Sync implementation - WatermelonDB documentation</title>  <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> <meta name="description" content=""> <meta name="viewport" content="width=device-width, initial-scale=1"> <meta name="theme-color" content="#ffffff" /> <link rel="icon" href="../favicon.svg"> <link rel="shortcut icon" href="../favicon.png"> <link rel="stylesheet" href="../css/variables.css"> <link rel="stylesheet" href="../css/general.css"> <link rel="stylesheet" href="../css/chrome.css"> <link rel="stylesheet" href="../css/print.css" media="print">  <link rel="stylesheet" href="../FontAwesome/css/font-awesome.css"> <link rel="stylesheet" href="../fonts/fonts.css">  <link rel="stylesheet" href="../highlight.css"> <link rel="stylesheet" href="../tomorrow-night.css"> <link rel="stylesheet" href="../ayu-highlight.css">  </head> <body>  <script type="text/javascript"> var path_to_root = "../"; var default_theme = window.matchMedia("(prefers-color-scheme: dark)").matches ? "navy" : "light"; </script>  <script type="text/javascript"> try { var theme = localStorage.getItem('mdbook-theme'); var sidebar = localStorage.getItem('mdbook-sidebar'); if (theme.startsWith('"') && theme.endsWith('"')) { localStorage.setItem('mdbook-theme', theme.slice(1, theme.length - 1)); } if (sidebar.startsWith('"') && sidebar.endsWith('"')) { localStorage.setItem('mdbook-sidebar', sidebar.slice(1, sidebar.length - 1)); } } catch (e) { } </script>  <script type="text/javascript"> var theme; try { theme = localStorage.getItem('mdbook-theme'); } catch(e) { } if (theme === null || theme === undefined) { theme = default_theme; } var html = document.querySelector('html'); html.classList.remove('no-js') html.classList.remove('light') html.classList.add(theme); html.classList.add('js'); </script>  <script type="text/javascript"> var html = document.querySelector('html'); var sidebar = 'hidden'; if (document.body.clientWidth >= 1080) { try { sidebar = localStorage.getItem('mdbook-sidebar'); } catch(e) { } sidebar = sidebar || 'visible'; } html.classList.remove('sidebar-visible'); html.classList.add("sidebar-" + sidebar); </script> <nav id="sidebar" class="sidebar" aria-label="Table of contents"> <div class="sidebar-scrollbox"> <ol class="chapter"><li class="chapter-item expanded "><a href="../ch01-00-get-excited.html"><strong aria-hidden="true">1.</strong> Get excited</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="../index.html"><strong aria-hidden="true">1.1.</strong> Check out the README</a></li><li class="chapter-item expanded "><a href="../Demo.html"><strong aria-hidden="true">1.2.</strong> See the demo</a></li></ol></li><li class="chapter-item expanded "><a href="../ch02-00-learn-to-use.html"><strong aria-hidden="true">2.</strong> Learn to use Watermelon</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="../Installation.html"><strong aria-hidden="true">2.1.</strong> Installation</a></li><li class="chapter-item expanded "><a href="../Setup.html"><strong aria-hidden="true">2.2.</strong> Setup</a></li><li class="chapter-item expanded "><a href="../Schema.html"><strong aria-hidden="true">2.3.</strong> Schema</a></li><li class="chapter-item expanded "><a href="../Model.html"><strong aria-hidden="true">2.4.</strong> Defining Models</a></li><li class="chapter-item expanded "><a href="../CRUD.html"><strong aria-hidden="true">2.5.</strong> Create, Read, Update, Delete</a></li><li class="chapter-item expanded "><a href="../Components.html"><strong aria-hidden="true">2.6.</strong> Connecting to React Components</a></li><li class="chapter-item expanded "><a href="../Query.html"><strong aria-hidden="true">2.7.</strong> Querying</a></li><li class="chapter-item expanded "><a href="../Relation.html"><strong aria-hidden="true">2.8.</strong> Relations</a></li><li class="chapter-item expanded "><a href="../Writers.html"><strong aria-hidden="true">2.9.</strong> Writers, Readers, batching</a></li></ol></li><li class="chapter-item expanded "><a href="../ch03-00-advanced.html"><strong aria-hidden="true">3.</strong> Advanced guides</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="../Advanced/Migrations.html"><strong aria-hidden="true">3.1.</strong> Migrations</a></li><li class="chapter-item expanded "><a href="../Advanced/Sync.html"><strong aria-hidden="true">3.2.</strong> Sync</a></li><li class="chapter-item expanded "><a href="../Advanced/CreateUpdateTracking.html"><strong aria-hidden="true">3.3.</strong> Automatic create/update tracking</a></li><li class="chapter-item expanded "><a href="../Advanced/AdvancedFields.html"><strong aria-hidden="true">3.4.</strong> Advanced fields</a></li><li class="chapter-item expanded "><a href="../Advanced/Flow.html"><strong aria-hidden="true">3.5.</strong> Flow</a></li><li class="chapter-item expanded "><a href="../Advanced/LocalStorage.html"><strong aria-hidden="true">3.6.</strong> LocalStorage</a></li><li class="chapter-item expanded "><a href="../Advanced/ProTips.html"><strong aria-hidden="true">3.7.</strong> Pro tips</a></li><li class="chapter-item expanded "><a href="../Advanced/Performance.html"><strong aria-hidden="true">3.8.</strong> Performance tips</a></li><li class="chapter-item expanded "><a href="../Advanced/SharingDatabaseAcrossTargets.html"><strong aria-hidden="true">3.9.</strong> iOS - Sharing database across targets</a></li></ol></li><li class="chapter-item expanded "><a href="../ch04-00-deeper.html"><strong aria-hidden="true">4.</strong> Dig deeper into WatermelonDB</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="../Implementation/Architecture.html"><strong aria-hidden="true">4.1.</strong> Architecture</a></li><li class="chapter-item expanded "><a href="../Implementation/Adapters.html"><strong aria-hidden="true">4.2.</strong> Adapters</a></li><li class="chapter-item expanded "><a href="../Implementation/SyncImpl.html" class="active"><strong aria-hidden="true">4.3.</strong> Sync implementation</a></li></ol></li><li class="chapter-item expanded "><a href="../ch04-00-deeper.html"><strong aria-hidden="true">5.</strong> Other</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="../Roadmap.html"><strong aria-hidden="true">5.1.</strong> Roadmap</a></li><li class="chapter-item expanded "><a href="../CONTRIBUTING.html"><strong aria-hidden="true">5.2.</strong> Contributing</a></li><li class="chapter-item expanded "><a href="../CHANGELOG.html"><strong aria-hidden="true">5.3.</strong> Changelog</a></li></ol></li></ol> </div> <div id="sidebar-resize-handle" class="sidebar-resize-handle"></div> </nav> <div id="page-wrapper" class="page-wrapper"> <div class="page"> <div id="menu-bar-hover-placeholder"></div> <div id="menu-bar" class="menu-bar sticky bordered"> <div class="left-buttons"> <button id="sidebar-toggle" class="icon-button" type="button" title="Toggle Table of Contents" aria-label="Toggle Table of Contents" aria-controls="sidebar"> <i class="fa fa-bars"></i> </button> <button id="theme-toggle" class="icon-button" type="button" title="Change theme" aria-label="Change theme" aria-haspopup="true" aria-expanded="false" aria-controls="theme-list"> <i class="fa fa-paint-brush"></i> </button> <ul id="theme-list" class="theme-popup" aria-label="Themes" role="menu"> <li role="none"><button role="menuitem" class="theme" id="light">Light (default)</button></li> <li role="none"><button role="menuitem" class="theme" id="rust">Rust</button></li> <li role="none"><button role="menuitem" class="theme" id="coal">Coal</button></li> <li role="none"><button role="menuitem" class="theme" id="navy">Navy</button></li> <li role="none"><button role="menuitem" class="theme" id="ayu">Ayu</button></li> </ul> <button id="search-toggle" class="icon-button" type="button" title="Search. (Shortkey: s)" aria-label="Toggle Searchbar" aria-expanded="false" aria-keyshortcuts="S" aria-controls="searchbar"> <i class="fa fa-search"></i> </button> </div> <h1 class="menu-title">WatermelonDB documentation</h1> <div class="right-buttons"> <a href="../print.html" title="Print this book" aria-label="Print this book"> <i id="print-button" class="fa fa-print"></i> </a> </div> </div> <div id="search-wrapper" class="hidden"> <form id="searchbar-outer" class="searchbar-outer"> <input type="search" name="search" id="searchbar" name="searchbar" placeholder="Search this book ..." aria-controls="searchresults-outer" aria-describedby="searchresults-header"> </form> <div id="searchresults-outer" class="searchresults-outer hidden"> <div id="searchresults-header" class="searchresults-header"></div> <ul id="searchresults"> </ul> </div> </div>  <script type="text/javascript"> document.getElementById('sidebar-toggle').setAttribute('aria-expanded', sidebar === 'visible'); document.getElementById('sidebar').setAttribute('aria-hidden', sidebar !== 'visible'); Array.from(document.querySelectorAll('#sidebar a')).forEach(function(link) { link.setAttribute('tabIndex', sidebar === 'visible' ? 0 : -1); }); </script> <div id="content" class="content"> <main> <h1 id="sync-implementation-details"><a class="header" href="#sync-implementation-details">Sync implementation details</a></h1> <p>If you're looking for a guide to implement Watermelon Sync in your app, see <a href="../Advanced/Sync.html"><strong>Synchronization</strong></a>.</p> <p>If you want to contribute to Watermelon Sync, or implement your own synchronization engine from scratch, read this.</p> <h2 id="implementing-your-own-sync-from-scratch"><a class="header" href="#implementing-your-own-sync-from-scratch">Implementing your own sync from scratch</a></h2> <p>For basic details about how changes tracking works, see: <a href="https://www.youtube.com/watch?v=uFvHURTRLxQ">📺 Digging deeper into WatermelonDB</a></p> <p>Why you might want to implement a custom sync engine? If you have an existing remote server architecture that's difficult to adapt to Watermelon sync protocol, or you specifically want a different architecture (e.g. single HTTP request -- server resolves conflicts). Be warned, however, that <strong>implementing sync that works reliably</strong> is a hard problem, so we recommend sticking to Watermelon Sync and tweaking it as needed.</p> <p>The rest of this document contains details about how Watermelon Sync works - you can use that as a blueprint for your own work.</p> <p>If possible, please use sync implementation helpers from <code>sync/*.js</code> to keep your custom sync implementation have as much commonality as possible with the standard implementation. This is good both for you and for the rest of WatermelonDB community, as we get to share improvements and bug fixes. If the helpers are <em>almost</em> what you need, but not quite, please send pull requests with improvements!</p> <h2 id="watermelon-sync----details"><a class="header" href="#watermelon-sync----details">Watermelon Sync -- Details</a></h2> <h3 id="general-design"><a class="header" href="#general-design">General design</a></h3> <ul> <li>master/replica - server is the source of truth, client has a full copy and syncs back to server (no peer-to-peer syncs)</li> <li>two phase sync: first pull remote changes to local app, then push local changes to server</li> <li>client resolves conflicts</li> <li>content-based, not time-based conflict resolution</li> <li>conflicts are resolved using per-column client-wins strategy: in conflict, server version is taken except for any column that was changed locally since last sync.</li> <li>local app tracks its changes using a _status (synced/created/updated/deleted) field and _changes field (which specifies columns changed since last sync)</li> <li>server only tracks timestamps (or version numbers) of every record, not specific changes</li> <li>sync is performed for the entire database at once, not per-collection</li> <li>eventual consistency (client and server are consistent at the moment of successful pull if no local changes need to be pushed)</li> <li>non-blocking: local database writes (but not reads) are only momentarily locked when writing data but user can safely make new changes throughout the process</li> </ul> <h3 id="sync-procedure"><a class="header" href="#sync-procedure">Sync procedure</a></h3> <ol> <li>Pull phase</li> </ol> <ul> <li>get <code>lastPulledAt</code> timestamp locally (null if first sync)</li> <li>call <code>pullChanges</code> function, passing <code>lastPulledAt</code> <ul> <li>server responds with all changes (create/update/delete) that occured since <code>lastPulledAt</code></li> <li>server serves us with its current timestamp</li> </ul> </li> <li>IN ACTION (lock local writes): <ul> <li>ensure no concurrent syncs</li> <li>apply remote changes locally <ul> <li>insert new records <ul> <li>if already exists (error), update</li> <li>if locally marked as deleted (error), un-delete and update</li> </ul> </li> <li>update records <ul> <li>if synced, just replace contents with server version</li> <li>if locally updated, we have a conflict! <ul> <li>take remote version, apply local fields that have been changed locally since last sync (per-column client wins strategy)</li> <li>record stays marked as updated, because local changes still need to be pushed</li> </ul> </li> <li>if locally marked as deleted, ignore (deletion will be pushed later)</li> <li>if doesn't exist locally (error), create</li> </ul> </li> <li>destroy records <ul> <li>if alredy deleted, ignore</li> <li>if locally changed, destroy anyway</li> <li>ignore children (server ought to schedule children to be destroyed)</li> </ul> </li> </ul> </li> <li>if successful, save server's timestamp as new <code>lastPulledAt</code></li> </ul> </li> </ul> <ol start="2"> <li>Push phase</li> </ol> <ul> <li>Fetch local changes <ul> <li>Find all locally changed records (created/updated record + deleted IDs) for all collections</li> <li>Strip _status, _changed</li> </ul> </li> <li>Call <code>pushChanges</code> function, passing local changes object, and the new <code>lastPulledAt</code> timestamp <ul> <li>Server applies local changes to database, and sends OK</li> <li>If one of the pushed records has changed <em>on the server</em> since <code>lastPulledAt</code>, push is aborted, all changes reverted, and server responds with an error</li> </ul> </li> <li>IN ACTION (lock local writes): <ul> <li>markLocalChangesAsSynced: <ul> <li>take local changes fetched in previous step, and:</li> <li>permanently destroy records marked as deleted</li> <li>mark created/updated records as synced and reset their _changed field</li> <li>note: <em>do not</em> mark record as synced if it changed locally since <code>fetch local changes</code> step (user could have made new changes that need syncing)</li> </ul> </li> </ul> </li> </ul> <h3 id="notes"><a class="header" href="#notes">Notes</a></h3> <ul> <li>This procedure is designed such that if sync fails at any moment, and even leaves local app in inconsistent (not fully synced) state, we should still achieve consistency with the next sync: <ul> <li>applyRemoteChanges is designed such that if all changes are applied, but <code>lastPulledAt</code> doesn't get saved — so during next pull server will serve us the same changes, second applyRemoteChanges will arrive at the same result</li> <li>local changes before "fetch local changes" step don't matter at all - user can do anything</li> <li>local changes between "fetch local changes" and "mark local changes as synced" will be ignored (won't be marked as synced) - will be pushed during next sync</li> <li>if changes don't get marked as synced, and are pushed again, server should apply them the same way</li> <li>remote changes between pull and push phase will be locally ignored (will be pulled next sync) unless there's a per-record conflict (then push fails, but next sync resolves both pull and push)</li> </ul> </li> </ul> <h3 id="migration-syncs"><a class="header" href="#migration-syncs">Migration Syncs</a></h3> <p>Schema versioning and migrations complicate sync, because a client might not be able to sync some tables and columns, but after upgrade to the newest version, it should be able to get consistent sync. To be able to do that, we need to know what's the schema version at which the last sync occured. Unfortunately, Watermelon Sync didn't track that from the first version, so backwards-compat is required.</p> <pre><code>synchronize({ migrationsEnabledAtVersion: XXX }) . . . . LPA = last pulled at MEA = migrationsEnabledAtVersion, schema version at which future migration support was introduced LS = last synced schema version (may be null due to backwards compat) CV = current schema version LPA MEA LS CV migration set LS=CV? comment null X X 10 null YES first sync. regardless of whether the app is migration sync aware, we can note LS=CV to fetch all migrations once available 100 null X X null NO indicates app is not migration sync aware so we're not setting LS to allow future migration sync 100 X 10 10 null NO up to date, no migration 100 9 9 10 {9-10} YES correct migration sync 100 9 null 10 {9-10} YES fallback migration. might not contain all necessary migrations, since we can't know for sure that user logged in at then-current-version==MEA 100 9 11 10 ERROR NO LS > CV indicates programmer error 100 11 X 10 ERROR NO MEA > CV indicates programmer error </code></pre> <h3 id="reference"><a class="header" href="#reference">Reference</a></h3> <p>This design has been informed by:</p> <ul> <li>10 years of experience building synchronization at Nozbe</li> <li>Kinto & Kinto.js <ul> <li>https://github.com/Kinto/kinto.js/blob/master/src/collection.js</li> <li>https://kintojs.readthedocs.io/en/latest/api/#fetching-and-publishing-changes</li> </ul> </li> <li>Histo - https://github.com/mirkokiefer/syncing-thesis</li> </ul> </main> <nav class="nav-wrapper" aria-label="Page navigation">  <a rel="prev" href="../Implementation/Adapters.html" class="mobile-nav-chapters previous" title="Previous chapter" aria-label="Previous chapter" aria-keyshortcuts="Left"> <i class="fa fa-angle-left"></i> </a> <a rel="next" href="../ch04-00-deeper.html" class="mobile-nav-chapters next" title="Next chapter" aria-label="Next chapter" aria-keyshortcuts="Right"> <i class="fa fa-angle-right"></i> </a> <div style="clear: both"></div> </nav> </div> </div> <nav class="nav-wide-wrapper" aria-label="Page navigation"> <a rel="prev" href="../Implementation/Adapters.html" class="nav-chapters previous" title="Previous chapter" aria-label="Previous chapter" aria-keyshortcuts="Left"> <i class="fa fa-angle-left"></i> </a> <a rel="next" href="../ch04-00-deeper.html" class="nav-chapters next" title="Next chapter" aria-label="Next chapter" aria-keyshortcuts="Right"> <i class="fa fa-angle-right"></i> </a> </nav> </div> <script type="text/javascript"> window.playground_copyable = true; </script> <script src="../elasticlunr.min.js" type="text/javascript" charset="utf-8"></script> <script src="../mark.min.js" type="text/javascript" charset="utf-8"></script> <script src="../searcher.js" type="text/javascript" charset="utf-8"></script> <script src="../clipboard.min.js" type="text/javascript" charset="utf-8"></script> <script src="../highlight.js" type="text/javascript" charset="utf-8"></script> <script src="../book.js" type="text/javascript" charset="utf-8"></script>  </body> </html>