# @dxos/feed

> Append-only replicatable log

Latest version **0.16.0** (published 2026-10-19) · FSL-1.1-Apache-2.0 license · 0 weekly downloads

## Install

```sh
npm install @dxos/feed
pnpm add @dxos/feed
yarn add @dxos/feed
bun add @dxos/feed
```

## Health

**Score 70/100 (B)** — status: active.

Positive: has types; esm support; no vulnerabilities; has provenance; recently updated; high maintenance score.

Warnings: low downloads; pre 1.0.

## Facts

| | |
|---|---|
| Version | 0.16.0 |
| Published | 2026-10-19 |
| First published | 2026-01-31 |
| Weekly downloads | 0 |
| License | FSL-1.1-Apache-2.0 |
| TypeScript types | bundled |
| Module format | ESM |
| Dependencies | 12 |
| Unpacked size | 721.4 KB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| Provenance | attested (GitHub Actions) |
| GitHub stars | 531 |
| Author | info@dxos.org |
| Maintainers | richburdon, marik_d, wittjosiah, dxos-bot, mykola-vrmchk |

## Links

- npm: https://www.npmjs.com/package/@dxos/feed
- Repository: https://github.com/dxos/dxos
- Homepage: https://dxos.org
- Issues: https://github.com/dxos/dxos/issues
- npm.io page: https://npm.io/package/@dxos/feed

## Dependencies (12)

- [effect](https://npm.io/package/effect.md) 4.0.4
- [@dxos/log](https://npm.io/package/@dxos/log.md) 0.16.0
- [@dxos/keys](https://npm.io/package/@dxos/keys.md) 0.16.0
- [@dxos/util](https://npm.io/package/@dxos/util.md) 0.16.0
- [@dxos/async](https://npm.io/package/@dxos/async.md) 0.16.0
- [@dxos/effect](https://npm.io/package/@dxos/effect.md) 0.16.0
- [@dxos/errors](https://npm.io/package/@dxos/errors.md) 0.16.0
- [@dxos/context](https://npm.io/package/@dxos/context.md) 0.16.0
- [@dxos/invariant](https://npm.io/package/@dxos/invariant.md) 0.16.0
- [@dxos/protocols](https://npm.io/package/@dxos/protocols.md) 0.16.0
- [@dxos/sql-sqlite](https://npm.io/package/@dxos/sql-sqlite.md) 0.16.0
- [@effect/sql-sqlite-node](https://npm.io/package/@effect/sql-sqlite-node.md) 4.0.4

## Recent versions

- 0.16.0 (latest) — 2026-10-19
- 0.16.0-next-60712ab191a99ba90316cc4be753e9e11b17100a (next) — 2026-10-19
- 0.9.1-main.c7dcc2e112 (main) — 2026-06-19
- 0.16.0-next-b440559823fb87ed1a4590dab21ae924d4d76c3a — 2026-10-19
- 0.14.0 — 2026-10-10
- 0.13.0 — 2026-10-07
- 0.12.0-next-73daef452f1464b62efdbca75fb0067c1066f264 — 2026-08-27
- 0.12.0-next-2800d0376abcc62d5544afdd744b16b2b960434e — 2026-08-27
- 0.12.0-next-2be9f24c465d98f4bd186aa03ece788902e6c17a — 2026-08-27
- 1.0.0-next-bbabc02363f18d7643ea4160e3a7a7809870fd0d — 2026-08-10
- 0.11.1 — 2026-08-05
- 0.11.0 — 2026-08-05
- 0.10.0 — 2026-07-02
- 0.9.1-staging.ee54ba693a — 2026-07-02
- 0.9.0 — 2026-06-15
- … 52 more at https://npm.io/package/@dxos/feed/versions

## README

# @dxos/feed

Append-only replicatable log consisting of blocks.

## Specification

### Concept

The feed is an append-only replicated log where data is stored in immutable blocks.
It is designed to run in multiple environments:

- **Browser**: Using `wasm-sqlite` in a Client Services Worker.
- **Node**: Native SQLite.
- **Cloudflare Workers**: Durable Objects SQL Storage.

All environments use `effect-sql` as the abstraction layer.

### Capacity

- **Feeds per Space**: ~1000
- **Blocks per Feed**: ~10,000

### Replication

- **Push**: Peers push blocks **without positions**.
- **Pull**: Peers pull blocks **with positions**, allowing them to reconstruct the global order locally.

#### Partial Replication

The protocol supports partial replication:

- **Per-Feed Configuration**: Each feed has independent replication settings.
  - **Default**: No replication (None).
  - **Replica Mode**: `replicate(fromPosition)` -> Replicates `[fromPosition, Infinity)`.
    - `fromPosition = 0`: Full History.
    - `fromPosition = current`: Live Tail only.
    - `fromPosition = now - 3days`: Time-based window (Future potential).
- **Metadata**: Replication settings are stored alongside feed metadata (locally).
- **Subset of Feeds**: Clients can choose to replicate only a specific subset of feeds within a Space.
  - _TODO: Define mechanism for selecting which feeds to replicate._
- **Range Selection**: Replication is strictly defined from a specific **start position** to the **end** (future blocks).
  - It is NOT possible to replicate only historical data without subscribing to new data.
  - Logic: `replicate(feedId, fromPosition) -> [fromPosition, Infinity)`

#### Recovery

Positions reach a replica only from the position authority, and the replica treats them as final:

- **Server swap.** Every response carries the serving store's `serverToken`. A token that differs from
  the one the client recorded means the store was replaced or wiped, so the client drops every position
  in the namespace and replays it from the start; local blocks are re-pushed and de-duplicated by
  `(actorId, sequence)`. The first token a client sees over progress it already holds is verified rather
  than acted on: the cursor steps back one slot, so the next pull re-fetches the block it was pulled from;
  the same block de-duplicates and progress is kept, a different one is a displacement and replays.
- **Server rollback.** A store that lost acknowledged rows keeps its token but re-issues their
  positions. The client notices when a position it is handed is already held by another local block,
  when a block it already holds arrives at a different position, when a pushed block is placed at or
  below its pull cursor, or when its cursor is above the store's `maxPosition`. Once other clients
  have written the store back past the cursor none of those fire, so every response also names the
  block the store holds at the requested position (`cursorBlock`), and a client holding a different
  block there replays as well. The displaced or above-mark blocks are unpositioned again (and so
  re-pushed), the handed position wins, and the pull cursor is rewound so the namespace is replayed —
  nothing is dropped, and the client never keeps retrying a position the store will not accept.

### Server Architecture (Cloudflare)

- **Initial Strategy**: **One Durable Object (DO) per Space**.
  - All feeds for a single space are managed by a single DO instance.
  - Simplifies consistency and coordination.
- **Scaling Strategy (Future)**: **Sharding by Feed ID**.
  - Feeds can be distributed across multiple DOs.
  - Shard Key: Prefix of `hash(feedId)` (e.g., first 2 chars).

### Ordering

Ordering is determined by the following rules:

1.  **Positioned Blocks**: Blocks that have been assigned a global `position` by the Server.
    - Ordered strictly by `position` (ascending).
2.  **Unpositioned Blocks**: Blocks pending position assignment (locally created or in-flight).
    - Ordered by **Lamport Timestamp** rules:
      1.  `sequence` (Ascending)
      2.  `actorId` (Lexicographical)

`sequence`: Assigned by the peer (Author). `actorId` + `sequence` forms the Lamport timestamp.

> `nextSeq = max(previous_seq_in_this_feed) + 1`

`position`: Assigned ONLY by the Server. Monotonic counter per Space.

### Idempotency

The protocol is idempotent. A block is uniquely identified by the tuple:

- `(spaceId, feedId, sequence, actorId)`
  OR
- `(spaceId, feedId, position)` (if positioned)

Processing the same block multiple times must be handled gracefully (e.g., `ON CONFLICT DO NOTHING`).

### Schema

#### Feeds Table (`feeds`)

Stores the mapping between global feed identifiers and local integer IDs.

| Field           | Type     | Description                                                                   |
| :-------------- | :------- | :---------------------------------------------------------------------------- |
| `feedPrivateId` | `number` | **Primary Key**. Local integer identifier for the feed (private to the peer). |
| `spaceId`       | `string` | The global Space ID (external).                                               |
| `feedId`        | `string` | The global Feed ID (ULID).                                                    |
| `feedNamespace` | `string` | **Optional**. Application specific namespace for filtering feeds.             |

**Indexes**:

- Unique Index: `(spaceId, feedId)`

#### Blocks Table (`blocks`)

| Field           | Type             | Description                                           |
| :-------------- | :--------------- | :---------------------------------------------------- |
| `feedPrivateId` | `number`         | Foreign key to `feeds.feedPrivateId`.                 |
| `position`      | `number \| null` | The global position index. `null` if unpositioned.    |
| `sequence`      | `number`         | **Sequence Number** (Assigned by Author).             |
| `actorId`       | `string`         | **Actor ID** (Public Key).                            |
| `prevSequence`  | `number`         | **Sequence Number** part of the Predecessor ID.       |
| `prevActorId`   | `string`         | **Actor ID** (Public Key) part of the Predecessor ID. |
| `timestamp`     | `number`         | Unix timestamp in milliseconds.                       |
| `data`          | `Uint8Array`     | The content of the block. **Immutable**.              |

**Indexes**:

- Unique Index: `(feedPrivateId, position)`
- Unique Index: `(feedPrivateId, sequence, actorId)`

#### Subscriptions Table (`subscriptions`)

Stores active subscriptions to optimize wire protocol.

| Field            | Type     | Description                               |
| :--------------- | :------- | :---------------------------------------- |
| `subscriptionId` | `string` | **Primary Key**. Unique ID.               |
| `expiresAt`      | `number` | Unix timestamp when subscription expires. |
| `feedPrivateIds` | `string` | JSON array of `feedPrivateId`s.           |

### Sync Protocol (Message-Based RPC)

The sync protocol is message-based. All messages include a `requestId`. Responses must copy the `requestId` from the request.

1.  **Query**
    - Input: `requestId`, `(feedIds: string[]) + cursor` OR `(subscriptionId: string) + cursor`
    - Output: `requestId`, `Block[]` (Stream or Batch)
    - Description: Returns blocks from the specified feeds (or subscription) with `position > cursor`.

2.  **Subscribe**
    - Input: `requestId`, `feedIds: string[]`
    - Output: `requestId`, `subscriptionId: string`
    - Description: Registers a subscription for specific feeds. Returns a `subscriptionId`.

3.  **Append**
    - Input: `requestId`, `namespace?: string`, `blocks: Block[]`
    - Output: `requestId`, `positions: number[]`
    - Description: Appends blocks to their respective feeds.
      - If `namespace` is provided and the feed is new, it sets the feed's namespace.
      - Returns the assigned global positions.

### Logic

#### Feed Identification

A feed is globally identified by the tuple `(spaceId, feedId)`.

- `spaceId`: String provided externally.
- `feedId`: ULID string.

#### Timestamps

All timestamps (e.g., `insertionTimestamp`) are Unix timestamps in milliseconds.

## Usage Examples

### 1. Append Blocks

**Request:**

```json
{
  "requestId": "req-1",
  "namespace": "my-app-data",
  "blocks": [
    {
      "actorId": "01H1...",
      "sequence": 100,
      "data": "...",
      "timestamp": 1234567890
    }
  ]
}
```

**Response:**

```json
{
  "requestId": "req-1",
  "positions": [42]
}
```

### 2. Subscribe and Query

**Subscribe Request:**

```json
{
  "requestId": "req-2",
  "feedIds": ["01H1..."]
}
```

**Subscribe Response:**

```json
{
  "requestId": "req-2",
  "subscriptionId": "sub-123",
  "expiresAt": 1234569999
}
```

### 3. List Feeds

**Request:**

```json
{
  "requestId": "req-4",
  "namespace": "data"
}
```

**Response:**

```json
{
  "requestId": "req-4",
  "feeds": [
    { "feedId": "01H1...", "namespace": "data" },
    { "feedId": "02H2...", "namespace": "data" }
  ]
}
```

**Query Request (Poll using Subscription):**

```json
{
  "requestId": "req-3",
  "subscriptionId": "sub-123",
  "cursor": 41
}
```

**Query Response:**

```json
{
  "requestId": "req-3",
  "blocks": [
    {
      "position": 42,
      "sequence": 100,
      "actorId": "01H1...",
      "data": "..."
    }
  ]
}
```

### TODOs

- [ ] Sync protocol only works if each namespace is synced separately.

---
_Source: https://npm.io/package/@dxos/feed · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._
