Bug 1745683 - add documentation for the webext-storage implementation....

- [bookmark_sync](https://searchfox.org/mozilla-central/source/toolkit/components/places/bookmark_sync),

  which [merges](https://mozilla.github.io/dogear) bookmarks from Firefox Sync

  with bookmarks in the Places database.

  [There's also some docs on how Rust interacts with Sync](../services/sync/rust-engines.html)

- [webext_storage_bridge](https://searchfox.org/mozilla-central/source/toolkit/components/extensions/storage/webext_storage_bridge),

  which powers the WebExtension storage.sync API. It's a self-contained example

  that pulls in a crate from application-services for the heavy lifting, wraps

  that up in a Rust XPCOM component, and then wraps the component in a JS

  interface. There's also some boilerplate there around adding a

  `components.conf` file, and a dummy C++ header that declares the component

  constructor.

  constructor. [It has some in-depth documentation on how it hangs together](../toolkit/components/extensions/webextensions/webext-storage.html).

parts of the component (ie, the part that provides the functionality that's not

sync related) will have its own mechanism for this. For example, the

`webext-storage bridge <https://searchfox.org/mozilla-central/source/toolkit/components/extensions/storage/webext_storage_bridge/src>`_

uses a similar technique.

uses a similar technique `which has some in-depth documentation <../../toolkit/components/extensions/webextensions/webext-storage.html>`_.

   lifecycle

   incognito

   webidl_bindings

   webext-storage

   other

   reference

========================

How webext storage works

========================

This document describes the implementation of the the `storage.sync` part of the

`WebExtensions Storage APIs

<https://developer.mozilla.org/docs/Mozilla/Add-ons/WebExtensions/API/storage>`_.

The implementation lives in the `toolkit/components/extensions/storage folder <https://searchfox.org/mozilla-central/source/toolkit/components/extensions/storage>`_

Ideally you would already know about Rust and XPCOM - `see this doc for more details <../../../../writing-rust-code/index.html>`_

At a very high-level, the system looks like:

.. mermaid::

    graph LR

        A[Extensions API]

        A --> B[Storage JS API]

        B --> C{magic}

        C --> D[app-services component]

Where "magic" is actually the most interesting part and the primary focus of this document.

    Note: The general mechanism described below is also used for other Rust components from the

    app-services team - for example, "dogear" uses a similar mechanism, and the sync engines

    too (but with even more complexity) to manage the threads. Unfortunately, at time of writing,

    no code is shared and it's not clear how we would, but this might change as more Rust lands.

The app-services component `lives on github <https://github.com/mozilla/application-services/blob/main/components/webext-storage>`_.

There are docs that describe `how to update/vendor this (and all) external rust code <../../../../build/buildsystem/rust.html>`_ you might be interested in.

To set the scene, let's look at the parts exposed to WebExtensions first; there are lots of

moving part there too.

WebExtension API

################

The WebExtension API is owned by the addons team. The implementation of this API is quite complex

as it involves multiple processes, but for the sake of this document, we can consider the entry-point

into the WebExtension Storage API as being `parent/ext-storage.js <https://searchfox.org/mozilla-central/source/toolkit/components/extensions/parent/ext-storage.js>`_

This entry-point ends up using the implementation in the

`ExtensionStorageSync JS class <https://searchfox.org/mozilla-central/rev/9028b0458cc1f432870d2996b186b0938dda734a/toolkit/components/extensions/ExtensionStorageSync.jsm#84>`_.

This class/module has complexity for things like migration from the earlier Kinto-based backend,

but importantly, code to adapt a callback API into a promise based one.

Overview of the API

###################

At a high level, this API is quite simple - there are methods to "get/set/remove" extension

storage data. Note that the "external" API exposed to the addon has subtly changed the parameters

for this "internal" API, so there's an extension ID parameter and the JSON data has already been

converted to a string.

The semantics of the API are beyond this doc but are

`documented on MDN <https://developer.mozilla.org/docs/Mozilla/Add-ons/WebExtensions/API/storage/sync>`_.

As you will see in those docs, the API is promise-based, but the rust implementation is fully

synchronous and Rust knows nothing about Javascript promises - so this system converts

the callback-based API to a promise-based one.

xpcom as the interface to Rust

##############################

xpcom is old Mozilla technology that uses C++ "vtables" to implement "interfaces", which are

described in IDL files. While this traditionally was used to interface

C++ and Javascript, we are leveraging existing support for Rust. The interface we are

exposing is described in `mozIExtensionStorageArea.idl <https://searchfox.org/mozilla-central/source/toolkit/components/extensions/storage/mozIExtensionStorageArea.idl>`_

The main interface of interest in this IDL file is `mozIExtensionStorageArea`.

This interface defines the functionality - and is the first layer in the sync to async model.

For example, this interface defines the following method:

.. code-block:: rust

    interface mozIExtensionStorageArea : nsISupports {

        ...

        // Sets one or more key-value pairs specified in `json` for the

        // `extensionId`...

        void set(in AUTF8String extensionId,

                in AUTF8String json,

                in mozIExtensionStorageCallback callback);

As you will notice, the 3rd arg is another interface, `mozIExtensionStorageCallback`, also

defined in that IDL file. This is a small, generic interface defined as:

.. code-block::

    interface mozIExtensionStorageCallback : nsISupports {

        // Called when the operation completes. Operations that return a result,

        // like `get`, will pass a `UTF8String` variant. Those that don't return

        // anything, like `set` or `remove`, will pass a `null` variant.

        void handleSuccess(in nsIVariant result);

        // Called when the operation fails.

        void handleError(in nsresult code, in AUTF8String message);

    };

Note that this delivers all results and errors, so must be capable of handling

every result type, which for some APIs may be problematic - but we are very lucky with this API

that this simple XPCOM callback interface is capable of reasonably representing the return types

from every function in the `mozIExtensionStorageArea` interface.

(There's another interface, `mozIExtensionStorageListener` which is typically

also implemented by the actual callback to notify the extension about changes,

but that's beyond the scope of this doc.)

*Note the thread model here is async* - the `set` call will return immediately, and later, on

the main thread, we will call the callback param with the result of the operation.

So under the hood, what happens is something like:

.. mermaid::

    sequenceDiagram

        Extension->>ExtensionStorageSync: call `set` and give me a promise

        ExtensionStorageSync->>xpcom: call `set`, supplying new data and a callback

        ExtensionStorageSync-->>Extension: your promise

        xpcom->>xpcom: thread magic in the "bridge"

        xpcom-->>ExtensionStorageSync: callback!

        ExtensionStorageSync-->>Extension: promise resolved

So onto the thread magic in the bridge!

webext_storage_bridge

#####################

The `webext_storage_bridge <https://searchfox.org/mozilla-central/source/toolkit/components/extensions/storage/webext_storage_bridge>`_

is a Rust crate which, as implied by the name, is a "bridge" between this Javascript/XPCOM world to

the actual `webext-storage <https://github.com/mozilla/application-services/tree/main/components/webext-storage>`_ crate.

lib.rs

------

Is the entry-point - it defines the xpcom "factory function" -

an `extern "C"` function which is called by xpcom to create the Rust object

implementing `mozIExtensionStorageArea` using existing gecko support.

area.rs

-------

This module defines the interface itself. For example, inside that file you will find:

.. code-block:: rust

    impl StorageSyncArea {

        ...

        xpcom_method!(

            set => Set(

                ext_id: *const ::nsstring::nsACString,

                json: *const ::nsstring::nsACString,

                callback: *const mozIExtensionStorageCallback

            )

        );

        /// Sets one or more key-value pairs.

        fn set(

            &self,

            ext_id: &nsACString,

            json: &nsACString,

            callback: &mozIExtensionStorageCallback,

        ) -> Result<()> {

            self.dispatch(

                Punt::Set {

                    ext_id: str::from_utf8(&*ext_id)?.into(),

                    value: serde_json::from_str(str::from_utf8(&*json)?)?,

                },

                callback,

            )?;

            Ok(())

        }

Of interest here:

* `xpcom_method` is a Rust macro, and part of the existing xpcom integration which already exists

  in gecko. It declares the xpcom vtable method described in the IDL.

* The `set` function is the implementation - it does string conversions and the JSON parsing

  on the main thread, then does the work via the supplied callback param, `self.dispatch` and a `Punt`.

* The `dispatch` method dispatches to another thread, leveraging existing in-tree `moz_task <https://searchfox.org/mozilla-central/source/xpcom/rust/moz_task>`_ support, shifting the `Punt` to another thread and making the callback when done.

Punt

----

`Punt` is a whimsical name somewhat related to a "bridge" - it carries things across and back.

It is a fairly simple enum in `punt.rs <https://searchfox.org/mozilla-central/source/toolkit/components/extensions/storage/webext_storage_bridge/src/punt.rs>`_.

It's really just a restatement of the API we expose suitable for moving across threads. In short, the `Punt` is created on the main thread,

then sent to the background thread where the actual operation runs via a `PuntTask` and returns a `PuntResult`.

There's a few dances that go on, but the end result is that `inner_run() <https://searchfox.org/mozilla-central/source/toolkit/components/extensions/storage/webext_storage_bridge/src/punt.rs>`_

gets executed on the background thread - so for `Set`:

.. code-block:: rust

        Punt::Set { ext_id, value } => {

            PuntResult::with_change(&ext_id, self.store()?.get()?.set(&ext_id, value)?)?

        }

Here, `self.store()` is a wrapper around the actual Rust implementation from app-services with

various initialization and mutex dances involved - see `store.rs`.

ie, this function is calling our Rust implementation and stashing the result in a `PuntResult`

The `PuntResult` is private to that file but is a simple struct that encapsulates both

the actual result of the function (also a set of changes to send to observers, but that's

beyond this doc).

Ultimately, the `PuntResult` ends up back on the main thread once the call is complete

and arranges to callback the JS implementation, which in turn resolves the promise created in `ExtensionStorageSync.jsm`

End result:

-----------

.. mermaid::

    sequenceDiagram

        Extension->>ExtensionStorageSync: call `set` and give me a promise

        ExtensionStorageSync->>xpcom - bridge main thread: call `set`, supplying new data and a callback

        ExtensionStorageSync-->>Extension: your promise

        xpcom - bridge main thread->>moz_task worker thread: Punt this

        moz_task worker thread->>webext-storage: write this data to the database

        webext-storage->>webext-storage: done: result/error and observers

        webext-storage-->>moz_task worker thread: ...

        moz_task worker thread-->>xpcom - bridge main thread: PuntResult

        xpcom - bridge main thread-->>ExtensionStorageSync: callback!

        ExtensionStorageSync-->>Extension: promise resolved