Replicache Big Picture

Replicache

This is a framework that helps implement Local First Software. Git similarly helps organize synchronized tasks through push and pull.

Replicache synchronizes server data asynchronously behind the scenes, eliminating server round trips and enabling immediate UI changes.

Replica Parts

Replicache consists of several elements.

Replicache

Replicache can be viewed as an in-browser Key-Value store that includes git-like operations internally. Write to memory first and synchronize later.

Your application

This is an application we created, like a web application. It is the entity that stores the state in Replicache. Mutator and Subscription are implemented to change and respond to state.

Your server

It exists to store the most reliable data. The state stored in the database connected to the server takes priority over the state in the application.

The server must implement push (upstream) and pull (downstream) to communicate with the client's Replicache.

• push(upstream): Replicache sends changes to the push endpoint. Mutators are implemented on servers as well as applications, and this push endpoint executes this mutator to change the state of the database.

• pull(downstream): When periodically or explicitly requested, Replicache sends a pull request to the server. The server returns the changes necessary for the client to become identical to the server's state.

• poke: Although the client periodically sends a pull request, in order to display it in more real time, when there is a change on the server, it is a signal that the server gives a hint to the client to make a pull request. It does not carry any data.

Sync

The application and server are synchronized to the latest state. The picture below clearly shows this process. It shows the process of periodically pulling state changes from the server and updating the UI, and how state changes on the client first update the UI and are then pushed to the server.

Source

Clients, ClientGroups, Caches

Replicache in memory is called Client.

import {Replicache} from "replicache";

const rep = new Replicache({
  name: userID,
  ...
});

console.log(rep.clientID);

There is usually one client per tap. The client is volatile and follows the life cycle of the tab. Has a unique clientID.

Client Group is a set of clients that share local data. Clients within this client group share state even when offline.

Client Group uses an on-disk persistent cache that is distinguished by the name parameter of the Replicache constructor. All clients belonging to a Client Group with the same name share the same cache.

The Client View

Client has an ordered map of key value pairs in persistent cache, which is called Client View. Client View is application data and is synchronized with server data. The reason it is called Client View is because different clients may have server data in different Client Views. This means that each client sees the status of the server differently.

Accessing the Client View is very fast. Read latency is less than 1ms and most devices have a throughput of 500MB/s.

It is recommended that you read and use it directly from the Client View rather than copying the Client View separately using useState from a place like React and uploading it to memory. When the mutator changes the Client View, the subscription is triggered so that the UI is updated.

Subscriptions

The Subscribe function receives the ReadTransaction argument and implements reading from Replicache. Whenever this subscription goes out of date due to a change in replicache data, the subscribe function is performed again. If this result changes, the value is updated and the UI is also updated.

If you configure the UI through Subscription, you can always keep it up to date.

import {Replicache} from "replicache";

const rep = new Replicache({
  name: userID,
  ...
});

console.log(rep.clientID);

Mutations

Mutation refers to the task of changing Replicache data. The subject that receives mutations and actually changes data is called a mutator.

At startup, several Mutators are registered in Replicache, but they are actually just named functions. Both createTodo and markTodoComplete below are mutators that change Replicache data through WriteTransaction.

const todos = useSubscribe(rep, async tx => {
  return await tx.scan({prefix: 'todo/'}).toArray();
});
return (
  <ul>
    {todos.map(todo => (
      <li key={todo.id}>{todo.text}</li>
    ))}
  </ul>
);

Mutator works as follows. When the mutator operates, the data changes, and the subscriptions related to it are triggered, and the UI also changes.

const rep = new Replicache({
  ...
  mutators: {
    createTodo,
    markTodoComplete,
  },
});

async function createTodo(tx: WriteTransaction, todo: Todo) {
  await tx.set(`/todo/${todo.id}`, todo);
}

async function markTodoComplete(tx: WriteTransaction,
    {id, complete}: {id: string, complete: boolean}) {
  const key = `/todo/${id}`;
  const todo = await tx.get(key);
  if (!todo) {
    return;
  }
  todo.complete = complete;
  await tx.set(key, todo);
}

Internally, Mutator creates something called mutation. It's like an execution record, but Replicache creates the following mutation.

await rep.mutate.createTodo({id: nanoid(), text: "take out the trash"});

These mutations are marked as pending until they are pushed to the server and completely synced.

Sync Details

Now, here are the details of Sync, which can be said to be the core of Replicache. Sync is done on the server.

The Replica Sync Model

(From now on, the expression ‘state’ refers to the state of data (key value space) consisting of several key and value pairs.)

The Sync problem that Replicache is trying to solve occurs when multiple clients change the same state at the same time and the following conditions exist.

1. The state that the server has is the source of truth. It is expressed as canonical.
2. Changes in the client’s local state are reflected immediately. This is called speculative.
3. The server must apply changes exactly once and the results must be predictable. Changes applied to the server must be able to reasonably merge with local changes on the client.

The last item among these is 'reasonably merging' server changes with local state, which is an interesting topic. For ‘rational merger’, the following situations must be considered.

• If local changes have not yet been applied to the server. In this case, you need to ensure that local changes do not disappear from the app's UI even if new state is retrieved from the server. After receiving the new state from the server, any existing local changes must be re-executed on top of the server state.

• When local changes made on the client have already been sent to the server and reflected in the server state. In this case, be careful not to apply local changes twice. Local changes should not be reapplied

• If there are other clients that have changed the server state for the same state. In this case, as in the first case, local changes must be redone based on the status received from the server. However, because conflicts may occur over the same resource, the merge logic must be carefully planned. Write this logic inside the Mutator.

Let’s follow the Mutator’s operation process.

Local execution

The mutator operates locally and the value of the replicache changes according to the mutator logic. At the same time, this client creates a mutation with a mutationId that increases sequentially. Mutations are queued as pending mutations.

Push

Pending mutations are sent to the push endpoint (replicache-push) implemented on the server.

mutation changes the canonical state by executing the mutator implemented on the server. While applying the mutation, the last mutation id of this client is updated, and becomes a value that allows you to know which mutation to reapply from when this client does the next pull.

A pending mutation applied locally generates a speculative result, and a mutation applied to the server generates a canonical result. Mutations applied to the server are confirmed and will not be executed locally again. Even if the same mutation returns a different result, the server's canonical result takes precedence, so the client's result changes.

Pull

Replicache periodically sends a request to the pull endpoint (replicache-pull) to retrieve the latest state and update the UI.

Pull request includes cookie and clientGroupId, and returns new cookie, patch, and lastMutationIDChanges.

Cookie is used to distinguish the server status held by the client. Any value that can track how much the server and client states differ is sufficient. You can think of it as a global 'version' that changes whenever the state of the database changes. Alternatively, you can use a cookie strategy to track a more specific range of data.

lastMutationIdChanges is a value representing the mutation ID last applied by the server for each client. All mutations with a mutationID smaller than this value should no longer be considered pending but confirmed.

Rebase

When a client receives a pull, the patch must be applied to the local state. However, because the pending mutation would have affected the current local state, the patch cannot be applied directly to the local state. Instead, revert the local pending mutation, apply the patch received as a pull first, and then apply the local pending mutation again.

To enable this kind of undo and reapplication, Replicache was designed similarly to Git. You can think of the state of the server as the main branch, and the state changed by a locally pending mutation as the develop branch, receive a pull from the server to main, and rebase develop to main.

Conflicts that may arise during rebase will be discussed separately below.

Poke

Poke, as explained above, is a hint message that the server tells the client to pull.

Conflict Resolution

Merge conflicts are unavoidable in distributed systems such as Replicache. Merging is necessary during the pull and push process. Merging should be done in a way that makes the merge result predictable and fits the purpose of the app.

If it is a conference room reservation app, only one request should be approved when a conflict occurs. Therefore, you must adopt a merge method that only approves clients who have made reservations first.

On the other hand, if it is a Todo app, the purpose of the Todo list is that both changes are approved even if additions occur at the same time.

Merge Conflict occurs in the following two situations.

1. When local changes will be applied to the server. This is because the status when applied locally and the status when applied on the server may be different.

2. When rebasing. This is because the status may be different when applied.

Replicache recognizes that the merge method must be implemented differently depending on the purpose of the app, so it allows developers to implement it. Developers can implement this logic through Mutator.

The above is the detailed content of Replicache Big Picture. For more information, please follow other related articles on the PHP Chinese website!