Last active
October 21, 2025 03:41
-
Star
(148)
You must be signed in to star a gist -
Fork
(2)
You must be signed in to fork a gist
-
-
Save pesterhazy/4de96193af89a6dd5ce682ce2adff49a to your computer and use it in GitHub Desktop.
Revisions
-
pesterhazy revised this gist
Nov 10, 2024 . 1 changed file with 2 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -77,6 +77,8 @@ The Web Platform lacks a standardized way - or any proper way, really - to lock Attempts have been made to builds locks on top of existing APIs like localStorage and postMessage, but clearly locking requires a new primitive and cannot be polyfilled. Proposed by Google, the [Web Locks API](https://wicg.github.io/web-locks/) adds the capabilities desparately needed to implement safe concurrent use of IndexedDB. However, while [Chrome shipped Web Locks in 2018](https://www.chromestatus.com/feature/5712361335816192), Safari and Firefox haven't implemented the API yet. Similar things can be said about the new [Atomics API](https://v8.dev/features/atomics). Update 2024: all major browsers [now support Web Locks](https://caniuse.com/mdn-api_lock). ## Private Browsing Mode (Firefox) Firefox is the only major browser without support for IndexedDB in Private Browsing mode. In a Private Browsing window (or if Private Browsing is enabled as a global setting), Firefox [throws with a cryptic error](https://bugzilla.mozilla.org/show_bug.cgi?id=781982) when you call indexeddb.open: -
pesterhazy revised this gist
Oct 27, 2021 . 1 changed file with 1 addition and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1,4 +1,4 @@ This gist lists challenges you run into when building offline-first applications based on IndexedDB, including open-source libraries like [Firebase](https://github.com/firebase/firebase-js-sdk), [pouchdb](https://github.com/pouchdb/pouchdb) and [AWS amplify](https://github.com/aws-amplify/amplify-js/blob/8fa348b8ba708434992d97557b0fceebbf7abe9a/packages/datastore/src/storage/adapter/IndexedDBAdapter.ts) ([more](https://github.com/pubkey/client-side-databases)). Note that some of the following issues affect only Safari. Out of the major browsers, Chrome's IndexedDB implementation is the best. -
pesterhazy revised this gist
Aug 13, 2021 . No changes.There are no files selected for viewing
-
pesterhazy revised this gist
Jun 15, 2021 . 1 changed file with 1 addition and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -103,7 +103,7 @@ In a similar vein, wiki service Notion [discusses problems](https://www.notion.s > Before SQLite, we relied on IndexedDB for client-side storage. But we encountered storage quotas, a number of bugs, and performance concerns on Windows machines in particular, which meant IndexedDB wouldn’t scale with Notion’s growing user base. Their solution? To move to an SQLite db bundled with their Electron app running outside the browser context. ## Bugs, bugs, bugs (Safari) -
pesterhazy revised this gist
Jun 15, 2021 . 1 changed file with 5 additions and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -103,4 +103,8 @@ In a similar vein, wiki service Notion [discusses problems](https://www.notion.s > Before SQLite, we relied on IndexedDB for client-side storage. But we encountered storage quotas, a number of bugs, and performance concerns on Windows machines in particular, which meant IndexedDB wouldn’t scale with Notion’s growing user base. Their solution? To move to a SQLite db bundled with their Electron app outside the browser context. ## Bugs, bugs, bugs (Safari) The Webkit team keeps shipping critical storage-related bugs into production, again and again. In May 2021, Safari 14.1.1 was released with a bug where [IndexedDB databases would fail to `open()`](https://bugs.webkit.org/show_bug.cgi?id=226547) when the browser is loaded for the first time ([discussion](https://news.ycombinator.com/item?id=27509206)). This came a month after the Safari 14.1 release, which [fundamentally broke localStorage](https://bugs.webkit.org/show_bug.cgi?id=225344) ([discussion](https://twitter.com/jaffathecake/status/1389493762129375232?lang=en)). -
pesterhazy revised this gist
May 4, 2021 . 1 changed file with 1 addition and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -101,6 +101,6 @@ As it turns out, Mozilla objected to the Web SQL interface as having no alternat In a similar vein, wiki service Notion [discusses problems](https://www.notion.so/blog/faster-page-load-navigation) with their IndexedDb backend: > Before SQLite, we relied on IndexedDB for client-side storage. But we encountered storage quotas, a number of bugs, and performance concerns on Windows machines in particular, which meant IndexedDB wouldn’t scale with Notion’s growing user base. Their solution? To move to a SQLite db bundled with their Electron app outside the browser context. -
pesterhazy revised this gist
May 4, 2021 . No changes.There are no files selected for viewing
-
pesterhazy revised this gist
Apr 23, 2021 . 1 changed file with 1 addition and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -103,4 +103,4 @@ In a similar vein, wiki service Notion [discusses problems](https://www.notion.s Before SQLite, we relied on IndexedDB for client-side storage. But we encountered storage quotas, a number of bugs, and performance concerns on Windows machines in particular, which meant IndexedDB wouldn’t scale with Notion’s growing user base. Their solution? To move to a SQLite db bundled with their Electron app outside the browser context. -
pesterhazy revised this gist
Apr 23, 2021 . 1 changed file with 7 additions and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -97,4 +97,10 @@ However, how quotas work is different from browser to browser. One surprising fi Before IndexedDB there was [Web SQL](https://www.w3.org/TR/webdatabase/), a thin wrapper around the legendary SQLite embedded database. Web SQL is more powerful (a proper superset of IndexedDB) and arguably better designed than IndexedDB. So why did browser vendors end up favoring the inferior IndexedDB standard? As it turns out, Mozilla objected to the Web SQL interface as having no alternative implementation available. So IndexedDB, an API with less surface, was chosen instead. Ironically, IndexedDB is internally implemented based on SQLite in Firefox (Chrome uses the simpler LevelDB instead). In a similar vein, wiki service Notion [discusses problems](https://www.notion.so/blog/faster-page-load-navigation) with their IndexedDb backend: Before SQLite, we relied on IndexedDB for client-side storage. But we encountered storage quotas, a number of bugs, and performance concerns on Windows machines in particular, which meant IndexedDB wouldn’t scale with Notion’s growing user base. Their solution? To move to SQLite hosted outside the browser context, in Electron. -
pesterhazy revised this gist
Apr 9, 2021 . 1 changed file with 8 additions and 2 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1,6 +1,6 @@ This gist lists challenges you run into when building offline-first applications based on IndexedDB, including open-source libraries like [Firebase](https://github.com/firebase/firebase-js-sdk), [pouchdb](https://github.com/pouchdb/pouchdb) and [AWS amplify](https://github.com/aws-amplify/amplify-js/blob/8fa348b8ba708434992d97557b0fceebbf7abe9a/packages/datastore/src/storage/adapter/IndexedDBAdapter.ts). Note that some of the following issues affect only Safari. Out of the major browsers, Chrome's IndexedDB implementation is the best. ## Backing file on disk (WAL file) keeps growing (Safari) @@ -91,4 +91,10 @@ Because there is no API to tell if Private Browsing is active, the only workarou Browser vendors [enforce qutoas](https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API/Browser_storage_limits_and_eviction_criteria) for all browser storage, including IndexedDB. When the quota is exceeded, writes will fail wih `QuotaExceededError` or similar exceptions. However, how quotas work is different from browser to browser. One surprising finding is that in Firefox, quotas seem to apply, not per subdomain, but to all subdomains belonging to a certain site name (more precisely, an [eTLD+1](https://web.dev/same-site-same-origin/)) taken together. So for the purposes of quota, `a.example.com` and `b.examples.com` will be counted together. As a result, clearing storage data for the current subdomain _may not prevent quota exceptions_ because the quota may be used up by other subdomains. ## Web SQL - the database that could have been Before IndexedDB there was [Web SQL](https://www.w3.org/TR/webdatabase/), a thin wrapper around the legendary SQLite embedded database. Web SQL is more powerful (a proper superset of IndexedDB) and arguably better designed than IndexedDB. So why did browser vendors end up favoring the inferior IndexedDB standard? As it turns out, Mozilla objected to the Web SQL interface as having no alternative implementation available. So IndexedDB, an API with less surface, was chosen instead. Ironically, IndexedDB is internally implemented based on SQLite in Firefox (Chrome uses the simpler LevelDB instead). -
pesterhazy revised this gist
Jan 25, 2021 . 1 changed file with 2 additions and 2 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -52,9 +52,9 @@ IndexedDB has the concepts of transactions. Does this mean that IndexedDB is ACI There are semantic differences between how Safari interprets the IndexedDB spec compared to Chrome and Firefox. In Safari, IndexedDB transactions auto-close more aggressively when nothing is being done to a transaction in a stackframe. This problem is apparent when using indexeddb with promises, because the use of `Promise.resolve().then( () => ... )` can lead to the transaction being closed prematurely, causing an exception - especially in Safari and Firefox. The idb promise wrapper [contains an warning](https://github.com/jakearchibald/idb#transaction-lifetime) but transaction auto-commit semantics in combination with promises are still difficult to get right. For example, Firebase went so far as to implement [their own version of Promise](https://github.com/firebase/firebase-js-sdk/blob/352eb475c000fb5e36eadad6d2a18f784818d7db/packages/firestore/src/local/persistence_promise.ts#L30-L37) for the purpose of working around these issues. Safari is most finicky, though operating within the spec. In any case the behavior differences compared to other implementations are confusing. In fact it's fair to say that it's hard to understand when transactions are supposed to auto-close, either as a application developer or browser implementer (see [this bug report](https://bugs.webkit.org/show_bug.cgi?id=216769)) In general, asynchronous operations or promises mixed with IndexedDB transactions are fraught with peril. Consider the case of deleting a number of keys from an index. This will require two steps: -
pesterhazy revised this gist
Jan 22, 2021 . 1 changed file with 2 additions and 2 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -40,15 +40,15 @@ At first we thought that the problem was in our code, but we discovered that sim As far as we can tell, the issue seems to occur when the Safari tab is sent to or comes back from the background. We know that Safari throttles tab activity aggressively to safe energy. But of course, this shouldn't lead to [indexeddb operations being delayed indefinitely](https://bugs.webkit.org/show_bug.cgi?id=197050). ## Is IndexedDB ACID compliant? IndexedDB has the concepts of transactions. Does this mean that IndexedDB is ACID compliant? Let's look at atomicity (A) and isolation (I) in particular: - IndexedDB provides atomic transactions. When multiple writes are wrapped in a transaction, IndexedDB guarantees that either all writes will go through or none of them will. This avoids inconsistencies due to writes interrupted mid-way. - IndexedDB does _not_ provide transaction isolation. As far as I know, concurrent transactions (in multiple tabs or even in a single tab) are never rolled back because they touch the same object stores. The only exception, as far as I can tell, is exceptions thrown when a primary key constraint is violated. This can be used to achieve some level of transaction isolation. However, the guarantees are nowhere near as extensive as those provided by SQL databases. ## Confusing transaction auto-commit semantics There are semantic differences between how Safari interprets the IndexedDB spec compared to Chrome and Firefox. In Safari, IndexedDB transactions auto-close more aggressively when nothing is being done to a transaction in a stackframe. This problem is apparent when using indexeddb with promises, because the use of `Promise.resolve().then( () => ... )` can lead to the transaction being closed prematurely, causing an exception - especially in Safari and Firefox. -
pesterhazy revised this gist
Jan 22, 2021 . 1 changed file with 4 additions and 4 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -50,13 +50,13 @@ IndexedDB has the concepts of transactions. Does this mean that IndexedDB is ACI ## Confusing transaction auto-closing semantics There are semantic differences between how Safari interprets the IndexedDB spec compared to Chrome and Firefox. In Safari, IndexedDB transactions auto-close more aggressively when nothing is being done to a transaction in a stackframe. This problem is apparent when using indexeddb with promises, because the use of `Promise.resolve().then( () => ... )` can lead to the transaction being closed prematurely, causing an exception - especially in Safari and Firefox. The idb promise wrapper [contains an warning](https://github.com/jakearchibald/idb#transaction-lifetime) but transaction auto-commit semantics in combination with promises are still difficult to get right. Safari is operating within the spec here, but the behavior differences compared to other implementations are confusing. In fact it's fair to say that it's hard to understand when transactions are supposed to auto-close, either as a application developer or browser implementer (see [this bug report](https://bugs.webkit.org/show_bug.cgi?id=216769)) In general, asynchronous operations or promises mixed with IndexedDB transactions are fraught with peril. Consider the case of deleting a number of keys from an index. This will require two steps: 1. Call `.getAllKeys()` on the index 2. Iterate over all the keys and call `.delete()` on the ObjectStore for each key @@ -65,7 +65,7 @@ It's tempting to do all of this in a single transaction - after all, they two pa Unfortunately, this will result in a latent bug that only manifests itself in rare circumstances, depending on timing. For example, Firefox will sometimes throw an exception with the message `Transaction is already committing or done`, but not in a reproducible manner. The solution is to perform all write operations (`.put`, `.delete`, etc) synchronously in a single stackframe, the same stackframe that created the transaction. In other words, it's not safe to perform steps 1 and 2 in a single transaction. Instead, each step should be a done in its own transaction (the first one being readonly, the second readwrite). ## No support for IDBTransaction.commit (Safari) -
pesterhazy revised this gist
Jan 22, 2021 . 1 changed file with 8 additions and 8 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -40,6 +40,14 @@ At first we thought that the problem was in our code, but we discovered that sim As far as we can tell, the issue seems to occur when the Safari tab is sent to or comes back from the background. We know that Safari throttles tab activity aggressively to safe energy. But of course, this shouldn't lead to [indexeddb operations being delayed indefinitely](https://bugs.webkit.org/show_bug.cgi?id=197050). ## Is an IndexedDB database ACID compliant? IndexedDB has the concepts of transactions. Does this mean that IndexedDB is ACID compliant? Let's look at atomicity (A) and isolation (I) in particular: - IndexedDB provides atomic transactions. When multiple writes are wrapped in a transaction, IndexedDB guarantees that either all writes will go through or none of them will. This avoids inconsistencies due to writes interrupted mid-way. - IndexedDB does _not_ provide transaction isolation. As far as I know, concurrent transactions (in multiple tabs or even in a single tab) are never rolled back because they touch the same object stores. The only exception, as far as I can tell, is exceptions thrown when a primary key constraint is violated. This can be used to achieve some level of transaction isolation. However, the guarantees are nowhere near as extensive as those provided by SQL databases. ## Confusing transaction auto-closing semantics There are semantic differences between how Safari interprets the IndexedDB spec compared to Chrome and Firefox. In Safari, IndexedDB transactions auto-close more aggressively when nothing is being done to a transaction in a stackframe. This problem is apparent when using indexeddb with promises, because the use of `Promise.resolve().then( () => ... )` can lead to the transaction being closed prematurely, causing an exception - in Safari, but not in other browsers. @@ -63,14 +71,6 @@ The solution is to perform all write operations (`.put`, `.delete`, etc) synchro Safari is the only major browser that doesn't support [IDBTransaction.commit()](https://developer.mozilla.org/en-US/docs/Web/API/IDBTransaction/commit) ## No locking primitives (Safari, Firefox) The Web Platform lacks a standardized way - or any proper way, really - to lock a resource across multiple tabs of the same browser. Multiple tabs of an app using IndexedDB will invariably write to the same IndexedDb database. Without cross-tab locking, database corruption is hard to avoid. -
pesterhazy revised this gist
Jan 22, 2021 . 1 changed file with 8 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -63,6 +63,14 @@ The solution is to perform all write operations (`.put`, `.delete`, etc) synchro Safari is the only major browser that doesn't support [IDBTransaction.commit()](https://developer.mozilla.org/en-US/docs/Web/API/IDBTransaction/commit) ## Is an IndexedDB database ACID compliant? IndexedDB has the concepts of transactions. Does this mean that IndexedDB is ACID compliant? Let's look at atomicity (A) and isolation (I) in particular: - IndexedDB provides atomic transactions. When multiple writes are wrapped in a transaction, IndexedDB guarantees that either all writes will go through or none of them will. This avoids incosistencies due to writes interrupted mid-way. - IndexedDB does _not_ provide transaction isolation. As far as I know, concurrent transactions (in multiple tabs or even in a single tab) are never rolled back because they touch the same object stores. The only exception, as far as I can tell, is exceptions thrown when a primary key constraint is violated. This can be used to achieve some level of transaction isolation. However, the guarantees are nowhere near as extensive as those provided by SQL databases. ## No locking primitives (Safari, Firefox) The Web Platform lacks a standardized way - or any proper way, really - to lock a resource across multiple tabs of the same browser. Multiple tabs of an app using IndexedDB will invariably write to the same IndexedDb database. Without cross-tab locking, database corruption is hard to avoid. -
pesterhazy revised this gist
Jan 22, 2021 . 1 changed file with 12 additions and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -40,14 +40,25 @@ At first we thought that the problem was in our code, but we discovered that sim As far as we can tell, the issue seems to occur when the Safari tab is sent to or comes back from the background. We know that Safari throttles tab activity aggressively to safe energy. But of course, this shouldn't lead to [indexeddb operations being delayed indefinitely](https://bugs.webkit.org/show_bug.cgi?id=197050). ## Confusing transaction auto-closing semantics There are semantic differences between how Safari interprets the IndexedDB spec compared to Chrome and Firefox. In Safari, IndexedDB transactions auto-close more aggressively when nothing is being done to a transaction in a stackframe. This problem is apparent when using indexeddb with promises, because the use of `Promise.resolve().then( () => ... )` can lead to the transaction being closed prematurely, causing an exception - in Safari, but not in other browsers. The idb promise wrapper [contains an warning](https://github.com/jakearchibald/idb#transaction-lifetime) but transaction auto-close semantics in combination with promises are still difficult to get right. Safari is operating within the spec here, but the behavior differences compared to other implementations are confusing. In fact it's fair to say that it's hard to understand when transactions are supposed to auto-close, either as a application developer or browser implementer (see [this bug report](https://bugs.webkit.org/show_bug.cgi?id=216769)) In general, asynchronous operations or promises with IndexedDB transactions is fraught with peril. Consider the case of deleting a number of keys from an index. This will require two steps: 1. Call `.getAllKeys()` on the index 2. Iterate over all the keys and call `.delete()` on the ObjectStore for each key It's tempting to do all of this in a single transaction - after all, they two parts semantically belong together. However, the frist step is necessarily asynchronous. As a result, it's possible that once we get to step 2, the transaction has already auto-committed. Unfortunately, this will result in a latent bug that only manifests itself in rare circumstances, depending on timing. For example, Firefox will sometimes throw an exception with the message `Transaction is already committing or done`, but not in a reproducible manner. The solution is to perform all write operations (`.put`, `.delete`, etc) synchronously in the same stackframe where the transaction was created. In other words, it's not safe to perform steps 1 and 2 in a single transaction. Instead, each step should be a done in its own transaction (the first one being readonly, the second readwrite). ## No support for IDBTransaction.commit (Safari) Safari is the only major browser that doesn't support [IDBTransaction.commit()](https://developer.mozilla.org/en-US/docs/Web/API/IDBTransaction/commit) -
pesterhazy revised this gist
Jan 21, 2021 . 1 changed file with 1 addition and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -56,7 +56,7 @@ Safari is the only major browser that doesn't support [IDBTransaction.commit()]( The Web Platform lacks a standardized way - or any proper way, really - to lock a resource across multiple tabs of the same browser. Multiple tabs of an app using IndexedDB will invariably write to the same IndexedDb database. Without cross-tab locking, database corruption is hard to avoid. Attempts have been made to builds locks on top of existing APIs like localStorage and postMessage, but clearly locking requires a new primitive and cannot be polyfilled. Proposed by Google, the [Web Locks API](https://wicg.github.io/web-locks/) adds the capabilities desparately needed to implement safe concurrent use of IndexedDB. However, while [Chrome shipped Web Locks in 2018](https://www.chromestatus.com/feature/5712361335816192), Safari and Firefox haven't implemented the API yet. Similar things can be said about the new [Atomics API](https://v8.dev/features/atomics). ## Private Browsing Mode (Firefox) -
pesterhazy revised this gist
Jan 21, 2021 . 1 changed file with 1 addition and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -72,4 +72,4 @@ Because there is no API to tell if Private Browsing is active, the only workarou Browser vendors [enforce qutoas](https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API/Browser_storage_limits_and_eviction_criteria) for all browser storage, including IndexedDB. When the quota is exceeded, writes will fail wih `QuotaExceededError` or similar exceptions. However, how quotas work is different from browser to browser. One surprising finding is that in Firefox, quotas seem to apply, not per subdomain, but to all subdomains belonging to a certain site name (more precisely, an [eTLD+1](https://web.dev/same-site-same-origin/)) taken together. So for the purposes of quota, `a.example.com` and `b.examples.com` will be counted together. As a result, clearing storage data for the current subdomain _may not prevent quota exceptions_ because the quota may be used up by other subdomains. -
pesterhazy revised this gist
Jan 21, 2021 . 1 changed file with 7 additions and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -66,4 +66,10 @@ Firefox is the only major browser without support for IndexedDB in Private Brows A mutation operation was attempted on a database that did not allow mutations ``` Because there is no API to tell if Private Browsing is active, the only workaround is to try to [open a test database](https://github.com/aws-amplify/amplify-js/blob/7477d272587212c2a3cf0e86806f8ff4a03881e0/packages/datastore/src/util.ts#L337) and to regard the API as unavailable if this fails. ## Quotas are confusing Browser vendors [enforce qutoas](https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API/Browser_storage_limits_and_eviction_criteria) for all browser storage, including IndexedDB. When the quota is exceeded, writes will fail wih `QuotaExceededError` or similar exceptions. However, how quotas work is different from browser to browser. One surprising finding is that in Firefox, quotas seem to apply, not per subdomain, but to all subdomains belonging to a certain domain taken together. So for the purposes of quota, `a.example.com` and `b.examples.com` will be counted together. As a result, clearing storage data for the current subdomain _may not prevent quota exceptions_ because the quota may be used up by other subdomains. -
pesterhazy revised this gist
Jan 21, 2021 . 1 changed file with 1 addition and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -46,7 +46,7 @@ There are semantic differences between how Safari interprets the IndexedDB spec The idb promise wrapper [contains an warning](https://github.com/jakearchibald/idb#transaction-lifetime) but transaction auto-close semantics in combination with promises are still difficult to get right. Safari is operating within the spec here, but the behavior differences compared to other implementations are confusing. In fact it's fair to say that it's hard to understand when transactions are supposed to auto-close, either as a application developer or browser implementer (see [this bug report](https://bugs.webkit.org/show_bug.cgi?id=216769)) ## No support for IDBTransaction.commit (Safari) -
pesterhazy revised this gist
Nov 28, 2020 . 1 changed file with 1 addition and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -24,7 +24,7 @@ You can work around this problem by only creating a small number of indexeddb da After 7 days of inactivity, Safari [deletes all browser storage](https://webkit.org/blog/10218/full-third-party-cookie-blocking-and-more/), including cookies, localstorage, websql and indexeddb. Other browsers don't do this. The feature, called Intelligent Tracking Prevention (ITP), is meant to prevent advertisers from abusing indexeddb as a way to track users. But it also has the consequence that purely in-browser apps like [excalidraw](https://github.com/excalidraw/excalidraw) cannot treat browser storage as durable storage for the user's data; they can only use it as a disposable cache. The [IsLoggedIn API](https://github.com/privacycg/is-logged-in) might allow browsers to tell tracking from legitimate use of browser storage in the future but is still in the planning phase. -
pesterhazy revised this gist
Nov 28, 2020 . 1 changed file with 3 additions and 3 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -42,11 +42,11 @@ As far as we can tell, the issue seems to occur when the Safari tab is sent to o ## Transactions close more quickly (Safari) There are semantic differences between how Safari interprets the IndexedDB spec compared to Chrome and Firefox. In Safari, IndexedDB transactions auto-close more aggressively when nothing is being done to a transaction in a stackframe. This problem is apparent when using indexeddb with promises, because the use of `Promise.resolve().then( () => ... )` can lead to the transaction being closed prematurely, causing an exception - in Safari, but not in other browsers. The idb promise wrapper [contains an warning](https://github.com/jakearchibald/idb#transaction-lifetime) but transaction auto-close semantics in combination with promises are still difficult to get right. Safari is operating within the spec here, but the behavior differences compared to other implementations are confusing. ## No support for IDBTransaction.commit (Safari) -
pesterhazy revised this gist
Nov 28, 2020 . 1 changed file with 2 additions and 2 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1,6 +1,6 @@ This gist lists challenges you run into when building offline-first applications based on IndexedDB, including open-source libraries like [Firebase](https://github.com/firebase/firebase-js-sdk), [pouchdb](https://github.com/pouchdb/pouchdb) and [AWS amplify](https://github.com/aws-amplify/amplify-js/blob/8fa348b8ba708434992d97557b0fceebbf7abe9a/packages/datastore/src/storage/adapter/IndexedDBAdapter.ts). Note that most of the following issues primarily affect Safari. Out of the major browsers, Chrome's IndexedDB implementation is the best. ## Backing file on disk (WAL file) keeps growing (Safari) -
pesterhazy revised this gist
Nov 28, 2020 . 1 changed file with 2 additions and 2 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -60,10 +60,10 @@ Attempts have been made to builds locks on top of existing APIs like localStorag ## Private Browsing Mode (Firefox) Firefox is the only major browser without support for IndexedDB in Private Browsing mode. In a Private Browsing window (or if Private Browsing is enabled as a global setting), Firefox [throws with a cryptic error](https://bugzilla.mozilla.org/show_bug.cgi?id=781982) when you call indexeddb.open: ``` A mutation operation was attempted on a database that did not allow mutations ``` Because there is no API to tell if Private Browsing is active, the only workaround is to try to [open a test database](https://github.com/aws-amplify/amplify-js/blob/7477d272587212c2a3cf0e86806f8ff4a03881e0/packages/datastore/src/util.ts#L337) and to regard the API as unavailable if this fails. -
pesterhazy revised this gist
Nov 28, 2020 . 1 changed file with 1 addition and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -58,7 +58,7 @@ The Web Platform lacks a standardized way - or any proper way, really - to lock Attempts have been made to builds locks on top of existing APIs like localStorage and postMessage, but clearly locking requires a new primitive and cannot be polyfilled. Proposed by Google, the [Web Locks API](https://wicg.github.io/web-locks/) adds the capabilities desparately needed to implement safe concurrent use of IndexedDB. However, while [Chrome shipped Web Locks in 2018](https://www.chromestatus.com/feature/5712361335816192), Safari and Firefox haven't implemented the API yet. ## Private Browsing Mode (Firefox) Firefox is the only major browser without support for IndexedDB in Private Browsing mode. In a Private Browsing window (or if Private Browsing is enabled as a global setting), Firefox throws with a cryptic error when you call indexeddb.open: -
pesterhazy revised this gist
Nov 28, 2020 . 1 changed file with 11 additions and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -56,4 +56,14 @@ Safari is the only major browser that doesn't support [IDBTransaction.commit()]( The Web Platform lacks a standardized way - or any proper way, really - to lock a resource across multiple tabs of the same browser. Multiple tabs of an app using IndexedDB will invariably write to the same IndexedDb database. Without cross-tab locking, database corruption is hard to avoid. Attempts have been made to builds locks on top of existing APIs like localStorage and postMessage, but clearly locking requires a new primitive and cannot be polyfilled. Proposed by Google, the [Web Locks API](https://wicg.github.io/web-locks/) adds the capabilities desparately needed to implement safe concurrent use of IndexedDB. However, while [Chrome shipped Web Locks in 2018](https://www.chromestatus.com/feature/5712361335816192), Safari and Firefox haven't implemented the API yet. ## Exceptions when using IndexedDB in Private Browsing Mode (Firefox) Firefox is the only major browser without support for IndexedDB in Private Browsing mode. In a Private Browsing window (or if Private Browsing is enabled as a global setting), Firefox throws with a cryptic error when you call indexeddb.open: ``` A mutation operation was attempted on a database that did not allow mutations ``` Because there is no API to tell if Private Browsing is active, the only workaround is to try to open a test database and to regard IndexedDB unavailable when this fails. -
pesterhazy revised this gist
Nov 28, 2020 . 1 changed file with 2 additions and 2 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1,6 +1,6 @@ This gist lists challenges you run into when building offline-first applications based on IndexedDB. Open-source libraries like [Firebase](https://github.com/firebase/firebase-js-sdk), [pouchdb](https://github.com/pouchdb/pouchdb) and [AWS amplify](https://github.com/aws-amplify/amplify-js/blob/8fa348b8ba708434992d97557b0fceebbf7abe9a/packages/datastore/src/storage/adapter/IndexedDBAdapter.ts) all run into similar problems. Note that most of the issues listed below primarily affect Safari. Out of the major browsers, Chrome's IndexedDB implementation is the best. ## Backing file on disk (WAL file) keeps growing (Safari) -
pesterhazy renamed this gist
Nov 28, 2020 . 1 changed file with 0 additions and 0 deletions.There are no files selected for viewing
File renamed without changes. -
pesterhazy revised this gist
Nov 28, 2020 . 1 changed file with 9 additions and 9 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1,12 +1,12 @@ This gist lists challenges people encounter building offline-first applications based on IndexedDB (desktop and mobile). The challenges my team is facing are likely to be shared by other offline-first tools and libraries, including [Firebase](https://github.com/firebase/firebase-js-sdk), [pouchdb](https://github.com/pouchdb/pouchdb) and [AWS amplify](https://github.com/aws-amplify/amplify-js/blob/8fa348b8ba708434992d97557b0fceebbf7abe9a/packages/datastore/src/storage/adapter/IndexedDBAdapter.ts). Note that most of the problems below primarily affect Safari. Generally, Chrome's IndexedDB implementation is the best. ## Backing file on disk (WAL file) keeps growing (Safari) When this bug occurs, every time you use the indexeddb, the [WAL file grows](https://bugs.webkit.org/show_bug.cgi?id=191294). Garbage collection doesn't seem to be working, so after a while, you end up with gigabytes of data. ## Random exceptions when working with a large number of indexeddb databases (Safari) Sometimes Safari gets into a funky state, where exceptions like "Unable to open database file on disk" are thrown. Other exceptions related to indexeddb methods are also possible. @@ -20,15 +20,15 @@ issue by deleting your local state. You can work around this problem by only creating a small number of indexeddb databases. ## State deleted after 7 days of inactivity (Safari) After 7 days of inactivity, Safari [deletes all browser storage](https://webkit.org/blog/10218/full-third-party-cookie-blocking-and-more/), including cookies, localstorage, websql and indexeddb. Other browsers don't do this. The feature, called Intelligent Tracking Prevention (ITP), is meant to prevent advertisers from abusing indexeddb as a way to track users. But it also has the consequence that local-first apps or apps like [excalidraw](https://github.com/excalidraw/excalidraw) that work without a server cannot treat browser storage as durable storage for the user's data; they can only use it as a disposable cache. The [IsLoggedIn API](https://github.com/privacycg/is-logged-in) might allow browsers to tell tracking from legitimate use of browser storage in the future but is still in the planning phase. ## IndexedDB operations hang without progress or error (Safari) In our offline-first product, we've had reports from Safari users that sometimes data isn't persisted to disk even though no exceptions are printed to the console. The problem seems to be that in rare circumstances, the promise chain stalls, with no progress or error - the worst kind of failure mode, because changes are silently dropped. @@ -40,19 +40,19 @@ At first we thought that the problem was in our code, but we discovered that sim As far as we can tell, the issue seems to occur when the Safari tab is sent to or comes back from the background. We know that Safari throttles tab activity aggressively to safe energy. But of course, this shouldn't lead to [indexeddb operations being delayed indefinitely](https://bugs.webkit.org/show_bug.cgi?id=197050). ## Transactions close more quickly (Safari) There are semantic differences between how Safari interprets the IndexedDB spec. IndexedDB transaction auto-close when nothing is being done to a transaction in a stackframe. This problem is apparent when using indexeddb with promises, because the use of `Promise.resolve().then( () => ... )` can lead to the transaction being closed prematurely, causing an exception - in Safari, but not in other browsers. The idb promise wrapper [has an explanation](https://github.com/jakearchibald/idb#transaction-lifetime) but transaction auto-close semantics are still difficult to get exactly right. Safari may be operating within the spec here, but the behavior differences compared to other implementations are certainly confusing. ## No support for IDBTransaction.commit (Safari) Safari is the only major browser that doesn't support [IDBTransaction.commit()](https://developer.mozilla.org/en-US/docs/Web/API/IDBTransaction/commit) ## No locking primitives (Safari, Firefox) The Web Platform lacks a standardized way - or any proper way, really - to lock a resource across multiple tabs of the same browser. Multiple tabs of an app using IndexedDB will invariably write to the same IndexedDb database. Without cross-tab locking, database corruption is hard to avoid. -
pesterhazy revised this gist
Nov 27, 2020 . 1 changed file with 1 addition and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -24,7 +24,7 @@ You can work around this problem by only creating a small number of indexeddb da After 7 days of inactivity, Safari [deletes all browser storage](https://webkit.org/blog/10218/full-third-party-cookie-blocking-and-more/), including cookies, localstorage, websql and indexeddb. Other browsers don't do this. The feature, called Intelligent Tracking Prevention (ITP), is meant to prevent advertisers from abusing indexeddb as a way to track users. But it also has the consequence that local-first apps or apps like [excalidraw](https://github.com/excalidraw/excalidraw) that work without a server cannot treat browser storage as durable storage for the user's data; they can only use it as a disposable cache. The [IsLoggedIn API](https://github.com/privacycg/is-logged-in) might allow browsers to tell tracking from legitimate use of browser storage in the future but is still in the planning phase.
NewerOlder