@rikishi/watermelondb

<!DOCTYPE HTML> <html lang="en" class="sidebar-visible no-js light"> <head>  <meta charset="UTF-8"> <title>Sync - WatermelonDB documentation</title>  <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> <meta name="description" content=""> <meta name="viewport" content="width=device-width, initial-scale=1"> <meta name="theme-color" content="#ffffff" /> <link rel="icon" href="../favicon.svg"> <link rel="shortcut icon" href="../favicon.png"> <link rel="stylesheet" href="../css/variables.css"> <link rel="stylesheet" href="../css/general.css"> <link rel="stylesheet" href="../css/chrome.css"> <link rel="stylesheet" href="../css/print.css" media="print">  <link rel="stylesheet" href="../FontAwesome/css/font-awesome.css"> <link rel="stylesheet" href="../fonts/fonts.css">  <link rel="stylesheet" href="../highlight.css"> <link rel="stylesheet" href="../tomorrow-night.css"> <link rel="stylesheet" href="../ayu-highlight.css">  </head> <body>  <script type="text/javascript"> var path_to_root = "../"; var default_theme = window.matchMedia("(prefers-color-scheme: dark)").matches ? "navy" : "light"; </script>  <script type="text/javascript"> try { var theme = localStorage.getItem('mdbook-theme'); var sidebar = localStorage.getItem('mdbook-sidebar'); if (theme.startsWith('"') && theme.endsWith('"')) { localStorage.setItem('mdbook-theme', theme.slice(1, theme.length - 1)); } if (sidebar.startsWith('"') && sidebar.endsWith('"')) { localStorage.setItem('mdbook-sidebar', sidebar.slice(1, sidebar.length - 1)); } } catch (e) { } </script>  <script type="text/javascript"> var theme; try { theme = localStorage.getItem('mdbook-theme'); } catch(e) { } if (theme === null || theme === undefined) { theme = default_theme; } var html = document.querySelector('html'); html.classList.remove('no-js') html.classList.remove('light') html.classList.add(theme); html.classList.add('js'); </script>  <script type="text/javascript"> var html = document.querySelector('html'); var sidebar = 'hidden'; if (document.body.clientWidth >= 1080) { try { sidebar = localStorage.getItem('mdbook-sidebar'); } catch(e) { } sidebar = sidebar || 'visible'; } html.classList.remove('sidebar-visible'); html.classList.add("sidebar-" + sidebar); </script> <nav id="sidebar" class="sidebar" aria-label="Table of contents"> <div class="sidebar-scrollbox"> <ol class="chapter"><li class="chapter-item expanded "><a href="../ch01-00-get-excited.html"><strong aria-hidden="true">1.</strong> Get excited</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="../index.html"><strong aria-hidden="true">1.1.</strong> Check out the README</a></li><li class="chapter-item expanded "><a href="../Demo.html"><strong aria-hidden="true">1.2.</strong> See the demo</a></li></ol></li><li class="chapter-item expanded "><a href="../ch02-00-learn-to-use.html"><strong aria-hidden="true">2.</strong> Learn to use Watermelon</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="../Installation.html"><strong aria-hidden="true">2.1.</strong> Installation</a></li><li class="chapter-item expanded "><a href="../Setup.html"><strong aria-hidden="true">2.2.</strong> Setup</a></li><li class="chapter-item expanded "><a href="../Schema.html"><strong aria-hidden="true">2.3.</strong> Schema</a></li><li class="chapter-item expanded "><a href="../Model.html"><strong aria-hidden="true">2.4.</strong> Defining Models</a></li><li class="chapter-item expanded "><a href="../CRUD.html"><strong aria-hidden="true">2.5.</strong> Create, Read, Update, Delete</a></li><li class="chapter-item expanded "><a href="../Components.html"><strong aria-hidden="true">2.6.</strong> Connecting to React Components</a></li><li class="chapter-item expanded "><a href="../Query.html"><strong aria-hidden="true">2.7.</strong> Querying</a></li><li class="chapter-item expanded "><a href="../Relation.html"><strong aria-hidden="true">2.8.</strong> Relations</a></li><li class="chapter-item expanded "><a href="../Writers.html"><strong aria-hidden="true">2.9.</strong> Writers, Readers, batching</a></li></ol></li><li class="chapter-item expanded "><a href="../ch03-00-advanced.html"><strong aria-hidden="true">3.</strong> Advanced guides</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="../Advanced/Migrations.html"><strong aria-hidden="true">3.1.</strong> Migrations</a></li><li class="chapter-item expanded "><a href="../Advanced/Sync.html" class="active"><strong aria-hidden="true">3.2.</strong> Sync</a></li><li class="chapter-item expanded "><a href="../Advanced/CreateUpdateTracking.html"><strong aria-hidden="true">3.3.</strong> Automatic create/update tracking</a></li><li class="chapter-item expanded "><a href="../Advanced/AdvancedFields.html"><strong aria-hidden="true">3.4.</strong> Advanced fields</a></li><li class="chapter-item expanded "><a href="../Advanced/Flow.html"><strong aria-hidden="true">3.5.</strong> Flow</a></li><li class="chapter-item expanded "><a href="../Advanced/LocalStorage.html"><strong aria-hidden="true">3.6.</strong> LocalStorage</a></li><li class="chapter-item expanded "><a href="../Advanced/ProTips.html"><strong aria-hidden="true">3.7.</strong> Pro tips</a></li><li class="chapter-item expanded "><a href="../Advanced/Performance.html"><strong aria-hidden="true">3.8.</strong> Performance tips</a></li><li class="chapter-item expanded "><a href="../Advanced/SharingDatabaseAcrossTargets.html"><strong aria-hidden="true">3.9.</strong> iOS - Sharing database across targets</a></li></ol></li><li class="chapter-item expanded "><a href="../ch04-00-deeper.html"><strong aria-hidden="true">4.</strong> Dig deeper into WatermelonDB</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="../Implementation/Architecture.html"><strong aria-hidden="true">4.1.</strong> Architecture</a></li><li class="chapter-item expanded "><a href="../Implementation/Adapters.html"><strong aria-hidden="true">4.2.</strong> Adapters</a></li><li class="chapter-item expanded "><a href="../Implementation/SyncImpl.html"><strong aria-hidden="true">4.3.</strong> Sync implementation</a></li></ol></li><li class="chapter-item expanded "><a href="../ch04-00-deeper.html"><strong aria-hidden="true">5.</strong> Other</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="../Roadmap.html"><strong aria-hidden="true">5.1.</strong> Roadmap</a></li><li class="chapter-item expanded "><a href="../CONTRIBUTING.html"><strong aria-hidden="true">5.2.</strong> Contributing</a></li><li class="chapter-item expanded "><a href="../CHANGELOG.html"><strong aria-hidden="true">5.3.</strong> Changelog</a></li></ol></li></ol> </div> <div id="sidebar-resize-handle" class="sidebar-resize-handle"></div> </nav> <div id="page-wrapper" class="page-wrapper"> <div class="page"> <div id="menu-bar-hover-placeholder"></div> <div id="menu-bar" class="menu-bar sticky bordered"> <div class="left-buttons"> <button id="sidebar-toggle" class="icon-button" type="button" title="Toggle Table of Contents" aria-label="Toggle Table of Contents" aria-controls="sidebar"> <i class="fa fa-bars"></i> </button> <button id="theme-toggle" class="icon-button" type="button" title="Change theme" aria-label="Change theme" aria-haspopup="true" aria-expanded="false" aria-controls="theme-list"> <i class="fa fa-paint-brush"></i> </button> <ul id="theme-list" class="theme-popup" aria-label="Themes" role="menu"> <li role="none"><button role="menuitem" class="theme" id="light">Light (default)</button></li> <li role="none"><button role="menuitem" class="theme" id="rust">Rust</button></li> <li role="none"><button role="menuitem" class="theme" id="coal">Coal</button></li> <li role="none"><button role="menuitem" class="theme" id="navy">Navy</button></li> <li role="none"><button role="menuitem" class="theme" id="ayu">Ayu</button></li> </ul> <button id="search-toggle" class="icon-button" type="button" title="Search. (Shortkey: s)" aria-label="Toggle Searchbar" aria-expanded="false" aria-keyshortcuts="S" aria-controls="searchbar"> <i class="fa fa-search"></i> </button> </div> <h1 class="menu-title">WatermelonDB documentation</h1> <div class="right-buttons"> <a href="../print.html" title="Print this book" aria-label="Print this book"> <i id="print-button" class="fa fa-print"></i> </a> </div> </div> <div id="search-wrapper" class="hidden"> <form id="searchbar-outer" class="searchbar-outer"> <input type="search" name="search" id="searchbar" name="searchbar" placeholder="Search this book ..." aria-controls="searchresults-outer" aria-describedby="searchresults-header"> </form> <div id="searchresults-outer" class="searchresults-outer hidden"> <div id="searchresults-header" class="searchresults-header"></div> <ul id="searchresults"> </ul> </div> </div>  <script type="text/javascript"> document.getElementById('sidebar-toggle').setAttribute('aria-expanded', sidebar === 'visible'); document.getElementById('sidebar').setAttribute('aria-hidden', sidebar !== 'visible'); Array.from(document.querySelectorAll('#sidebar a')).forEach(function(link) { link.setAttribute('tabIndex', sidebar === 'visible' ? 0 : -1); }); </script> <div id="content" class="content"> <main> <h1 id="synchronization"><a class="header" href="#synchronization">Synchronization</a></h1> <p>WatermelonDB has been designed from scratch to be able to seamlessly synchronize with a remote database (and, therefore, keep multiple copies of data synced with each other).</p> <p>Note that Watermelon is only a local database — you need to <strong>bring your own backend</strong>. What Watermelon provides are:</p> <ul> <li><strong>Synchronization primitives</strong> — information about which records were created, updated, or deleted locally since the last sync — and which columns exactly were modified. You can build your own custom sync engine using those primitives</li> <li><strong>Built-in sync adapter</strong> — You can use the sync engine Watermelon provides out of the box, and you only need to provide two API endpoints on your backend that conform to Watermelon sync protocol</li> </ul> <h2 id="using-synchronize-in-your-app"><a class="header" href="#using-synchronize-in-your-app">Using <code>synchronize()</code> in your app</a></h2> <p>To synchronize, you need to pass <code>pullChanges</code> and <code>pushChanges</code> <em>(optional)</em> that talk to your backend and are compatible with Watermelon Sync Protocol. The frontend code will look something like this:</p> <pre><code class="language-js">import { synchronize } from '@rikishi/watermelondb/sync' async function mySync() { await synchronize({ database, pullChanges: async ({ lastPulledAt, schemaVersion, migration }) => { const urlParams = `last_pulled_at=${lastPulledAt}&schema_version=${schemaVersion}&migration=${encodeURIComponent(JSON.stringify(migration))}` const response = await fetch(`https://my.backend/sync?${urlParams}`) if (!response.ok) { throw new Error(await response.text()) } const { changes, timestamp } = await response.json() return { changes, timestamp } }, pushChanges: async ({ changes, lastPulledAt }) => { const response = await fetch(`https://my.backend/sync?last_pulled_at=${lastPulledAt}`, { method: 'POST', body: JSON.stringify(changes) }) if (!response.ok) { throw new Error(await response.text()) } }, migrationsEnabledAtVersion: 1, }) } </code></pre> <h4 id="who-calls-synchronize"><a class="header" href="#who-calls-synchronize">Who calls <code>synchronize()</code>?</a></h4> <p>Upon looking at the example above, one question that may arise is who will call <code>synchronize()</code> -- or, in the example above <code>mySync()</code>. WatermelonDB does not manage the moment of invocation of the <code>synchronize()</code> function in any way. The database assumes every call of <code>pullChanges</code> will return <em>all</em> the changes that haven't yet been replicated (up to <code>last_pulled_at</code>). The application code is responsible for calling <code>synchronize()</code> in the frequence it deems necessary.</p> <h3 id="troubleshooting"><a class="header" href="#troubleshooting">Troubleshooting</a></h3> <p><strong>⚠️ Note about a React Native / UglifyES bug</strong>. When you import Watermelon Sync, your app might fail to compile in release mode. To fix this, configure Metro bundler to use Terser instead of UglifyES. Run:</p> <pre><code class="language-bash">yarn add metro-minify-terser </code></pre> <p>Then, update <code>metro.config.js</code>:</p> <pre><code class="language-js">module.exports = { // ... transformer: { // ... minifierPath: 'metro-minify-terser', }, } </code></pre> <p>You might also need to switch to Terser in Webpack if you use Watermelon for web.</p> <h3 id="implementing-pullchanges"><a class="header" href="#implementing-pullchanges">Implementing <code>pullChanges()</code></a></h3> <p>Watermelon will call this function to ask for changes that happened on the server since the last pull.</p> <p>Arguments:</p> <ul> <li><code>lastPulledAt</code> is a timestamp for the last time client pulled changes from server (or <code>null</code> if first sync)</li> <li><code>schemaVersion</code> is the current schema version of the local database</li> <li><code>migration</code> is an object representing schema changes since last sync (or <code>null</code> if up to date or not supported)</li> </ul> <p>This function should fetch from the server the list of ALL changes in all collections since <code>lastPulledAt</code>.</p> <ol> <li>You MUST pass an async function or return a Promise that eventually resolves or rejects</li> <li>You MUST pass <code>lastPulledAt</code>, <code>schemaVersion</code>, and <code>migration</code> to an endpoint that conforms to Watermelon Sync Protocol</li> <li>You MUST return a promise resolving to an object of this shape (your backend SHOULD return this shape already): <pre><code class="language-js">{ changes: { ... }, // valid changes object timestamp: 100000, // integer with *server's* current time } </code></pre> </li> <li>You MUST NOT store the object returned in <code>pullChanges()</code>. If you need to do any processing on it, do it before returning the object. Watermelon treats this object as "consumable" and can mutate it (for performance reasons)</li> </ol> <h3 id="implementing-pushchanges"><a class="header" href="#implementing-pushchanges">Implementing <code>pushChanges()</code></a></h3> <p>Watermelon will call this function with a list of changes that happened locally since the last push so you can post it to your backend.</p> <p>Arguments passed:</p> <pre><code class="language-js">{ changes: { ... }, // valid changes object lastPulledAt: 10000, // the timestamp of the last successful pull (timestamp returned in pullChanges) } </code></pre> <ol> <li>You MUST pass <code>changes</code> and <code>lastPulledAt</code> to a push sync endpoint conforming to Watermelon Sync Protocol</li> <li>You MUST pass an async function or return a Promise from <code>pushChanges()</code></li> <li><code>pushChanges()</code> MUST resolve after and only after the backend confirms it successfully received local changes</li> <li><code>pushChanges()</code> MUST reject if backend failed to apply local changes</li> <li>You MUST NOT resolve sync prematurely or in case of backend failure</li> <li>You MUST NOT mutate or store arguments passed to <code>pushChanges()</code>. If you need to do any processing on it, do it before returning the object. Watermelon treats this object as "consumable" and can mutate it (for performance reasons)</li> </ol> <h3 id="checking-unsynced-changes"><a class="header" href="#checking-unsynced-changes">Checking unsynced changes</a></h3> <p>WatermelonDB has a built in function to check whether there are any unsynced changes. The frontend code will look something like this</p> <pre><code class="language-js">import { hasUnsyncedChanges } from '@rikishi/watermelondb/sync' async function checkUnsyncedChanges() { await hasUnsyncedChanges({ database }) } </code></pre> <h3 id="general-information-and-tips"><a class="header" href="#general-information-and-tips">General information and tips</a></h3> <ol> <li>You MUST NOT connect to backend endpoints you don't control using <code>synchronize()</code>. WatermelonDB assumes pullChanges/pushChanges are friendly and correct and does not guarantee secure behavior if data returned is malformed.</li> <li>You SHOULD NOT call <code>synchronize()</code> while synchronization is already in progress (it will safely abort)</li> <li>You MUST NOT reset local database while synchronization is in progress (push to server will be safely aborted, but consistency of the local database may be compromised)</li> <li>You SHOULD wrap <code>synchronize()</code> in a "retry once" block - if sync fails, try again once. This will resolve push failures due to server-side conflicts by pulling once again before pushing.</li> <li>You can use <code>database.withChangesForTables</code> to detect when local changes occured to call sync. If you do this, you should debounce (or throttle) this signal to avoid calling <code>synchronize()</code> too often.</li> </ol> <h3 id="adopting-migration-syncs"><a class="header" href="#adopting-migration-syncs">Adopting Migration Syncs</a></h3> <p>For Watermelon Sync to maintain consistency after <a href="./Migrations.html">migrations</a>, you must support Migration Syncs (introduced in WatermelonDB v0.17). This allows Watermelon to request from backend the tables and columns it needs to have all the data.</p> <ol> <li>For new apps, pass <code>{migrationsEnabledAtVersion: 1}</code> to <code>synchronize()</code> (or the first schema version that shipped / the oldest schema version from which it's possible to migrate to the current version)</li> <li>To enable migration syncs, the database MUST be configured with <a href="./Migrations.html">migrations spec</a> (even if it's empty)</li> <li>For existing apps, set <code>migrationsEnabledAtVersion</code> to the current schema version before making any schema changes. In other words, this version should be the last schema version BEFORE the first migration that should support migration syncs.</li> <li>Note that for apps that shipped before WatermelonDB v0.17, it's not possible to determine what was the last schema version at which the sync happened. <code>migrationsEnabledAtVersion</code> is used as a placeholder in this case. It's not possible to guarantee that all necessary tables and columns will be requested. (If user logged in when schema version was lower than <code>migrationsEnabledAtVersion</code>, tables or columns were later added, and new records in those tables/changes in those columns occured on the server before user updated to an app version that has them, those records won't sync). To work around this, you may specify <code>migrationsEnabledAtVersion</code> to be the oldest schema version from which it's possible to migrate to the current version. However, this means that users, after updating to an app version that supports Migration Syncs, will request from the server all the records in new tables. This may be unacceptably inefficient.</li> <li>WatermelonDB >=0.17 will note the schema version at which the user logged in, even if migrations are not enabled, so it's possible for app to request from backend changes from schema version lower than <code>migrationsEnabledAtVersion</code></li> <li>You MUST NOT delete old <a href="./Migrations.html">migrations</a>, otherwise it's possible that the app is permanently unable to sync.</li> </ol> <h3 id="adopting-turbo-login"><a class="header" href="#adopting-turbo-login">Adopting Turbo Login</a></h3> <p>WatermelonDB v0.23 introduced an experimental optimization called "Turbo Login". Syncing using Turbo is up to 5.3x faster than the traditional method and uses a lot less memory, so it's suitable for even very large syncs. Keep in mind:</p> <ol> <li>This can only be used for the initial (login) sync, not for incremental syncs. It is a serious programmer error to run sync in Turbo mode if the database is not empty. Syncs with <code>deleted: []</code> fields not empty will fail.</li> <li>This only withs with SQLiteAdapter with JSI enabled and running - it does not work on web, or if e.g. Chrome Remote Debugging is enabled</li> <li>As of writing this, Turbo Login is considered experimental, so the exact API may change in a future version</li> </ol> <p>Here's basic usage:</p> <pre><code class="language-js">const isFirstSync = ... const useTurbo = isFirstSync await synchronize({ database, pullChanges: async ({ lastPulledAt, schemaVersion, migration }) => { const response = await fetch(`https://my.backend/sync?${...}`) if (!response.ok) { throw new Error(await response.text()) } if (useTurbo) { // NOTE: DO NOT parse JSON, we want raw text const json = await response.text() return { syncJson: json } } else { const { changes, timestamp } = await response.json() return { changes, timestamp } } }, unsafeTurbo: useTurbo, // ... }) </code></pre> <p>Raw JSON text is required, so it is not expected that you need to do any processing in pullChanges() - doing that defeats much of the point of using Turbo Login!</p> <p>If you're using pullChanges to send additional data to your app other than Watermelon Sync's <code>changes</code> and <code>timestamp</code>, you won't be able to process it in pullChanges. However, WatermelonDB can still pass extra keys in sync response back to the app - you can process them using <code>onDidPullChanges</code>. This works both with and without turbo mode:</p> <pre><code class="language-js">await synchronize({ database, pullChanges: async ({ lastPulledAt, schemaVersion, migration }) => { // ... }, unsafeTurbo: useTurbo, onDidPullChanges: async ({ messages }) => { if (messages) { messages.forEach(message => { alert(message) }) } } // ... }) </code></pre> <p>There's a way to make Turbo Login even more <em>turbo</em>! However, it requires native development skills. You need to develop your own networking native code, so that raw JSON can go straight from your native code to WatermelonDB's native code - skipping JavaScript processing altogether.</p> <pre><code class="language-js">await synchronize({ database, pullChanges: async ({ lastPulledAt, schemaVersion, migration }) => { // NOTE: You need the standard JS code path for incremental syncs // Create a unique id for this sync request const syncId = Math.floor(Math.random() * 1000000000) await NativeModules.MyNetworkingPlugin.pullSyncChanges( // Pass the id syncId, // Pass whatever information your plugin needs to make the request lastPulledAt, schemaVersion, migration ) // If successful, return the sync id return { syncJsonId: syncId } }, unsafeTurbo: true, // ... }) </code></pre> <p>In native code, perform network request and if successful, extract raw response body data - <code>NSData *</code> on iOS, <code>byte[]</code> on Android. Avoid extracting the response as a string or parsing the JSON. Then pass it to WatermelonDB's native code:</p> <pre><code class="language-java">// On Android (Java): import com.nozbe.watermelondb.jsi.WatermelonJSI; WatermelonJSI.provideSyncJson(/* id */ syncId, /* byte[] */ data); </code></pre> <pre><code class="language-objc">// On iOS (Objective-C): extern void watermelondbProvideSyncJson(int id, NSData *json, NSError **errorPtr); watermelondbProvideSyncJson(syncId, data, &error) </code></pre> <h3 id="adding-logging-to-your-sync"><a class="header" href="#adding-logging-to-your-sync">Adding logging to your sync</a></h3> <p>You can add basic sync logs to the sync process by passing an empty object to <code>synchronize()</code>. Sync will then mutate the object, populating it with diagnostic information (start/finish time, resolved conflicts, number of remote/local changes, any errors that occured, and more):</p> <pre><code class="language-js">// Using built-in SyncLogger import SyncLogger from '@rikishi/watermelondb/sync/SyncLogger' const logger = new SyncLogger(10 /* limit of sync logs to keep in memory */ ) await synchronize({ database, log: logger.newLog(), ... }) // this returns all logs (censored and safe to use in production code) console.log(logger.logs) // same, but pretty-formatted to a string (a user can easy copy this for diagnostic purposes) console.log(logger.formattedLogs) // You don't have to use SyncLogger, just pass a plain object to synchronize() const log = {} await synchronize({ database, log, ... }) console.log(log.startedAt) console.log(log.finishedAt) </code></pre> <p>⚠️ Remember to act responsibly with logs, since they might contain your user's private information. Don't display, save, or send the log unless you censor the log.</p> <h3 id="debugging-changes"><a class="header" href="#debugging-changes">Debugging <code>changes</code></a></h3> <p>If you want to conveniently see incoming and outgoing changes in sync in the console, add these lines to your pullChanges/pushChanges:</p> <p>⚠️ Leaving such logging committed and running in production is a huge security vulnerability and a performance hog.</p> <pre><code class="language-js">// UNDER NO CIRCUMSTANCES SHOULD YOU COMMIT THESE LINES UNCOMMENTED!!! require('@rikishi/watermelondb/sync/debugPrintChanges').default(changes, isPush) </code></pre> <p>Pass <code>true</code> for second parameter if you're checking outgoing changes (pushChanges), <code>false</code> otherwise. Make absolutely sure you don't commit this debug tool. For best experience, run this on web (Chrome) -- the React Native experience is not as good.</p> <h3 id="additional-synchronize-flags"><a class="header" href="#additional-synchronize-flags">Additional <code>synchronize()</code> flags</a></h3> <ul> <li><code>_unsafeBatchPerCollection: boolean</code> - if true, changes will be saved to the database in multiple batches. This is unsafe and breaks transactionality, however may be required for very large syncs due to memory issues</li> <li><code>sendCreatedAsUpdated: boolean</code> - if your backend can't differentiate between created and updated records, set this to <code>true</code> to supress warnings. Sync will still work well, however error reporting, and some edge cases will not be handled as well.</li> <li><code>conflictResolver: (TableName, local: DirtyRaw, remote: DirtyRaw, resolved: DirtyRaw) => DirtyRaw</code> - can be passed to customize how records are updated when they change during sync. See <code>src/sync/index.js</code> for details.</li> </ul> <h2 id="implementing-your-sync-backend"><a class="header" href="#implementing-your-sync-backend">Implementing your Sync backend</a></h2> <h3 id="understanding-changes-objects"><a class="header" href="#understanding-changes-objects">Understanding <code>changes</code> objects</a></h3> <p>Synchronized changes (received by the app in <code>pullChanges</code> and sent to the backend in <code>pushChanges</code>) are represented as an object with <em>raw records</em>. Those only use raw table and column names, and raw values (strings/numbers/booleans) — the same as in <a href="../Schema.html">Schema</a>.</p> <p>Deleted objects are always only represented by their IDs.</p> <p>Example:</p> <pre><code class="language-js">{ projects: { created: [ { id: 'aaaa', name: 'Foo', is_favorite: true }, { id: 'bbbb', name: 'Bar', is_favorite: false }, ], updated: [ { id: 'ccc', name: 'Baz', is_favorite: true }, ], deleted: ['ddd'], }, tasks: { created: [], updated: [ { id: 'tttt', name: 'Buy eggs' }, ], deleted: [], }, ... } </code></pre> <p>Again, notice the properties returned have the format defined in the <a href="../Schema.html">Schema</a> (e.g. <code>is_favorite</code>, not <code>isFavorite</code>).</p> <p>Valid changes objects MUST conform to this shape:</p> <pre><code class="language-js">Changes = { [table_name: string]: { created: RawRecord[], updated: RawRecord[], deleted: string[], } } </code></pre> <h3 id="implementing-pull-endpoint"><a class="header" href="#implementing-pull-endpoint">Implementing pull endpoint</a></h3> <p>Expected parameters:</p> <pre><code class="language-js">{ lastPulledAt: Timestamp, schemaVersion: int, migration: null | { from: int, tables: string[], columns: { table: string, columns: string[] }[] } } </code></pre> <p>Expected response:</p> <pre><code class="language-js">{ changes: Changes, timestamp: Timestamp } </code></pre> <ol> <li>The pull endpoint SHOULD take parameters and return a response matching the shape specified above. This shape MAY be different if negotiated with the frontend (however, frontend-side <code>pullChanges()</code> MUST conform to this)</li> <li>The pull endpoint MUST return all record changes in all collections since <code>lastPulledAt</code>, specifically: <ul> <li>all records that were created on the server since <code>lastPulledAt</code></li> <li>all records that were updated on the server since <code>lastPulledAt</code></li> <li>IDs of all records that were deleted on the server since <code>lastPulledAt</code></li> <li>record IDs MUST NOT be duplicated</li> </ul> </li> <li>If <code>lastPulledAt</code> is null or 0, you MUST return all accessible records (first sync)</li> <li>The timestamp returned by the server MUST be a value that, if passed again to <code>pullChanges()</code> as <code>lastPulledAt</code>, will return all changes that happened since this moment.</li> <li>The pull endpoint MUST provide a consistent view of changes since <code>lastPulledAt</code> <ul> <li>You should perform all queries synchronously or in a write lock to ensure that returned changes are consistent</li> <li>You should also mark the current server time synchronously with the queries</li> <li>This is to ensure that no changes are made to the database while you're fetching changes (otherwise some records would never be returned in a pull query)</li> <li>If it's absolutely not possible to do so, and you have to query each collection separately, be sure to return a <code>lastPulledAt</code> timestamp marked BEFORE querying starts. You still risk inconsistent responses (that may break app's consistency assumptions), but the next pull will fetch whatever changes occured during previous pull.</li> <li>An alternative solution is to check for the newest change before and after all queries are made, and if there's been a change during the pull, return an error code, or retry.</li> </ul> </li> <li>If <code>migration</code> is not null, you MUST include records needed to get a consistent view after a local database migration <ul> <li>Specifically, you MUST include all records in tables that were added to the local database between the last user sync and <code>schemaVersion</code></li> <li>For all columns that were added to the local app database between the last sync and <code>schemaVersion</code>, you MUST include all records for which the added column has a value other than the default value (<code>0</code>, <code>''</code>, <code>false</code>, or <code>null</code> depending on column type and nullability)</li> <li>You can determine what schema changes were made to the local app in two ways: <ul> <li>You can compare <code>migration.from</code> (local schema version at the time of the last sync) and <code>schemaVersion</code> (current local schema version). This requires you to negotiate with the frontend what schema changes are made at which schema versions, but gives you more control</li> <li>Or you can ignore <code>migration.from</code> and only look at <code>migration.tables</code> (which indicates which tables were added to the local database since the last sync) and <code>migration.columns</code> (which indicates which columns were added to the local database to which tables since last sync).</li> <li>If you use <code>migration.tables</code> and <code>migration.columns</code>, you MUST whitelist values a client can request. Take care not to leak any internal fields to the client.</li> </ul> </li> </ul> </li> <li>Returned raw records MUST match your app's <a href="../Schema.html">Schema</a></li> <li>Returned raw records MUST NOT not contain special <code>_status</code>, <code>_changed</code> fields.</li> <li>Returned raw records MAY contain fields (columns) that are not yet present in the local app (at <code>schemaVersion</code> -- but added in a later version). They will be safely ignored.</li> <li>Returned raw records MUST NOT contain arbitrary column names, as they may be unsafe (e.g. <code>__proto__</code> or <code>constructor</code>). You should whitelist acceptable column names.</li> <li>Returned record IDs MUST only contain safe characters <ul> <li>Default WatermelonDB IDs conform to <code>/^[a-zA-Z0-9]{16}$/</code></li> <li><code>_-.</code> are also allowed if you override default ID generator, but <code>'"\/$</code> are unsafe</li> </ul> </li> <li>Changes SHOULD NOT contain collections that are not yet present in the local app (at <code>schemaVersion</code>). They will, however, be safely ignored. <ul> <li>NOTE: This is true for WatermelonDB v0.17 and above. If you support clients using earlier versions, you MUST NOT return collections not known by them.</li> </ul> </li> <li>Changes MUST NOT contain collections with arbitrary names, as they may be unsafe. You should whitelist acceptable collection names.</li> </ol> <h3 id="implementing-push-endpoint"><a class="header" href="#implementing-push-endpoint">Implementing push endpoint</a></h3> <ol> <li>The push endpoint MUST apply local changes (passed as a <code>changes</code> object) to the database. Specifically: <ul> <li>create new records as specified by the changes object</li> <li>update existing records as specified by the changes object</li> <li>delete records by the specified IDs</li> </ul> </li> <li>If the <code>changes</code> object contains a new record with an ID that already exists, you MUST update it, and MUST NOT return an error code. <ul> <li>(This happens if previous push succeeded on the backend, but not on frontend)</li> </ul> </li> <li>If the <code>changes</code> object contains an update to a record that does not exist, then: <ul> <li>If you can determine that this record no longer exists because it was deleted, you SHOULD return an error code (to force frontend to pull the information about this deleted ID)</li> <li>Otherwise, you MUST create it, and MUST NOT return an error code. (This scenario should not happen, but in case of frontend or backend bugs, it would keep sync from ever succeeding.)</li> </ul> </li> <li>If the <code>changes</code> object contains a record to delete that doesn't exist, you MUST ignore it and MUST NOT return an error code <ul> <li>(This may happen if previous push succeeded on the backend, but not on frontend, or if another user deleted this record in between user's pull and push calls)</li> </ul> </li> <li>If the <code>changes</code> object contains a record that has been modified on the server after <code>lastPulledAt</code>, you MUST abort push and return an error code <ul> <li>This scenario means that there's a conflict, and record was updated remotely between user's pull and push calls. Returning an error forces frontend to call pull endpoint again to resolve the conflict</li> </ul> </li> <li>If application of all local changes succeeds, the endpoint MUST return a success status code.</li> <li>The push endpoint MUST be fully transactional. If there is an error, all local changes MUST be reverted on the server, and en error code MUST be returned.</li> <li>You MUST ignore <code>_status</code> and <code>_changed</code> fields contained in records in <code>changes</code> object</li> <li>You SHOULD validate data passed to the endpoint. In particular, collection and column names ought to be whitelisted, as well as ID format — and of course any application-specific invariants, such as permissions to access and modify records</li> <li>You SHOULD sanitize record fields passed to the endpoint. If there's something slightly wrong with the contents (but not shape) of the data (e.g. <code>user.role</code> should be <code>owner</code>, <code>admin</code>, or <code>member</code>, but user sent empty string or <code>abcdef</code>), you SHOULD NOT send an error code. Instead, prefer to "fix" errors (sanitize to correct format). <ul> <li>Rationale: Synchronization should be reliable, and should not fail other than transiently, or for serious programming errors. Otherwise, the user will have a permanently unsyncable app, and may have to log out/delete it and lose unsynced data. You don't want a bug 5 versions ago to create a persistently failing sync.</li> </ul> </li> <li>You SHOULD delete all descendants of deleted records <ul> <li>Frontend should ask the push endpoint to do so as well, but if it's buggy, you may end up with permanent orphans</li> </ul> </li> </ol> <h3 id="tips-on-implementing-server-side-changes-tracking"><a class="header" href="#tips-on-implementing-server-side-changes-tracking">Tips on implementing server-side changes tracking</a></h3> <p>If you're wondering how to <em>actually</em> implement consistent pulling of all changes since the last pull, or how to detect that a record being pushed by the user changed after <code>lastPulledAt</code>, here's what we recommend:</p> <ul> <li>Add a <code>last_modified</code> field to all your server database tables, and bump it to <code>NOW()</code> every time you create or update a record.</li> <li>This way, when you want to get all changes since <code>lastPulledAt</code>, you query records whose <code>last_modified > lastPulledAt</code>.</li> <li>The timestamp should be at least millisecond resolution, and you should add (for extra safety) a MySQL/PostgreSQL procedure that will ensure <code>last_modified</code> uniqueness and monotonicity <ul> <li>Specificaly, check that there is no record with a <code>last_modified</code> equal to or greater than <code>NOW()</code>, and if there is, increment the new timestamp by 1 (or however much you need to ensure it's the greatest number)</li> <li><a href="https://github.com/Kinto/kinto/blob/814c30c5dd745717b8ea50d708d9163a38d2a9ec/kinto/core/storage/postgresql/schema.sql#L64-L116">An example of this for PostgreSQL can be found in Kinto</a></li> <li>This protects against weird edge cases - such as records being lost due to server clock time changes (NTP time sync, leap seconds, etc.)</li> </ul> </li> <li>Of course, remember to ignore <code>last_modified</code> from the user if you do it this way.</li> <li>An alternative to using timestamps is to use an auto-incrementing counter sequence, but you must ensure that this sequence is consistent across all collections. You also leak to users the amount of traffic to your sync server (number of changes in the sequence)</li> <li>To distinguish between <code>created</code> and <code>updated</code> records, you can also store server-side <code>server_created_at</code> timestamp (if it's greater than <code>last_pulled_at</code> supplied to sync, then record is to be <code>created</code> on client, if less than — client already has it and it is to be <code>updated</code> on client). Note that this timestamp must be consistent with last_modified — and you must not use client-created <code>created_at</code> field, since you can never trust local timestamps. <ul> <li>Alternatively, you can send all non-deleted records as all <code>updated</code> and Watermelon will do the right thing in 99% of cases (you will be slightly less protected against weird edge cases — treatment of locally deleted records is different). If you do this, pass <code>sendCreatedAsUpdated: true</code> to <code>synchronize()</code> to supress warnings about records to be updated not existing locally.</li> </ul> </li> <li>You do need to implement a mechanism to track when records were deleted on the server, otherwise you wouldn't know to push them <ul> <li>One possible implementation is to not fully delete records, but mark them as DELETED=true</li> <li>Or, you can have a <code>deleted_xxx</code> table with just the record ID and timestamp (consistent with last_modified)</li> <li>Or, you can treat it the same way as "revoked permissions"</li> </ul> </li> <li>If you have a collaborative app with any sort of permissions, you also need to track granting and revoking of permissions the same way as changes to records <ul> <li>If permission to access records has been granted, the pull endpoint must add those records to <code>created</code></li> <li>If permission to access records has been revoked, the pull endpoint must add those records to <code>deleted</code></li> <li>Remember to also return all descendants of a record in those cases</li> </ul> </li> </ul> <h2 id="local-vs-remote-ids"><a class="header" href="#local-vs-remote-ids">Local vs Remote IDs</a></h2> <p>WatermelonDB has been designed with the assumption that there is no difference between Local IDs (IDs of records and their relations in a WatermelonDB database) and Remote IDs (IDs on the backend server). So a local app can create new records, generating their IDs, and the backend server will use this ID as the true ID. This greatly simplifies synchronization, as you don't have to replace local with remote IDs on the record and all records that point to it.</p> <p>We highly recommend that you adopt this practice.</p> <p>Some people are skeptical about this approach due to conflicts, since backend can guarantee unique IDs, and the local app can't. However, in practice, a standard Watermelon ID has 8,000,000,000,000,000,000,000,000 possible combinations. That's enough entropy to make conflicts extremely unlikely. At <a href="https://nozbe.com">Nozbe</a>, we've done it this way at scale for more than a decade, and not once did we encounter a genuine ID conflict or had other issues due to this approach.</p> <blockquote> <p>Using the birthday problem, we can calculate that for 36^16 possible IDs, if your system grows to a billion records, the probability of a single conflict is 6e-8. At 100B records, the probability grows to 0.06%. But if you grow to that many records, you're probably a very rich company and can start worrying about things like this <em>then</em>.</p> </blockquote> <p>If you absolutely can't adopt this practice, there's a number of production apps using WatermelonDB that keep local and remote IDs separate — however, more work is required this way. Search Issues to find discussions about this topic — and consider contributing to WatermelonDB to make managing separate local IDs easier for everyone!</p> <h2 id="existing-backend-implementations-for-watermelondb"><a class="header" href="#existing-backend-implementations-for-watermelondb">Existing backend implementations for WatermelonDB</a></h2> <p>Note that those are not maintained by WatermelonDB, and we make no endorsements about quality of these projects:</p> <ul> <li><a href="https://fahri.id/posts/how-to-build-watermelondb-sync-backend-in-elixir/">How to Build WatermelonDB Sync Backend in Elixir</a></li> <li><a href="https://github.com/AliAllaf/firemelon">Firemelon</a></li> <li><a href="https://github.com/nathanheffley/laravel-watermelon">Laravel Watermelon</a></li> <li>Did you make one? Please contribute a link!</li> </ul> <h2 id="current-sync-limitations"><a class="header" href="#current-sync-limitations">Current Sync limitations</a></h2> <ol> <li>If a record being pushed changes between pull and push, push will just fail. It would be better if it failed with a list of conflicts, so that <code>synchronize()</code> can automatically respond. Alternatively, sync could only send changed fields and server could automatically always just apply those changed fields to the server version (since that's what per-column client-wins resolver will do anyway)</li> <li>During next sync pull, changes we've just pushed will be pulled again, which is unnecessary. It would be better if server, during push, also pulled local changes since <code>lastPulledAt</code> and responded with NEW timestamp to be treated as <code>lastPulledAt</code>.</li> <li>It shouldn't be necessary to push the whole updated record — just changed fields + ID should be enough</li> </ol> <blockquote> <p>Note: That might conflict with "If client wants to update a record that doesn’t exist, create it"</p> </blockquote> <p>You don't like these limitations? Good, neither do we! Please contribute - we'll give you guidance.</p> <h2 id="contributing"><a class="header" href="#contributing">Contributing</a></h2> <ol> <li>If you implement Watermelon sync but found this guide confusing, please contribute improvements!</li> <li>Please help out with solving the current limitations!</li> <li>If you write server-side code made to be compatible with Watermelon, especially for popular platforms (Node, Ruby on Rails, Kinto, etc.) - please open source it and let us know! This would dramatically simplify implementing sync for people</li> <li>If you find Watermelon sync bugs, please report the issue! And if possible, write regression tests to make sure it never happens again</li> </ol> <h2 id="sync-primitives-and-implementing-your-own-sync-entirely-from-scratch"><a class="header" href="#sync-primitives-and-implementing-your-own-sync-entirely-from-scratch">Sync primitives and implementing your own sync entirely from scratch</a></h2> <p>See: <a href="../Implementation/SyncImpl.html">Sync implementation details</a></p> </main> <nav class="nav-wrapper" aria-label="Page navigation">  <a rel="prev" href="../Advanced/Migrations.html" class="mobile-nav-chapters previous" title="Previous chapter" aria-label="Previous chapter" aria-keyshortcuts="Left"> <i class="fa fa-angle-left"></i> </a> <a rel="next" href="../Advanced/CreateUpdateTracking.html" class="mobile-nav-chapters next" title="Next chapter" aria-label="Next chapter" aria-keyshortcuts="Right"> <i class="fa fa-angle-right"></i> </a> <div style="clear: both"></div> </nav> </div> </div> <nav class="nav-wide-wrapper" aria-label="Page navigation"> <a rel="prev" href="../Advanced/Migrations.html" class="nav-chapters previous" title="Previous chapter" aria-label="Previous chapter" aria-keyshortcuts="Left"> <i class="fa fa-angle-left"></i> </a> <a rel="next" href="../Advanced/CreateUpdateTracking.html" class="nav-chapters next" title="Next chapter" aria-label="Next chapter" aria-keyshortcuts="Right"> <i class="fa fa-angle-right"></i> </a> </nav> </div> <script type="text/javascript"> window.playground_copyable = true; </script> <script src="../elasticlunr.min.js" type="text/javascript" charset="utf-8"></script> <script src="../mark.min.js"