Matrix-static NPM

matrix-static

This is an experiement with a matrix.org static server. All files from the git repository, are served as they are, by a static HTTP server (gitlab pages), to represent the data of the matrix-static.gitlab.io matrix server.

community chat #matrix-static-server:matrix.org
code gitlab/matrix-static/matrix-static.gitlab.io
static deployment matrix-static.gitlab.io
original test https://gitlab.com/matrix-meetup/matrixmeetup.de
example static-server usage and test for a starting template
npm (very experimental & poc)

how it works

static files (json data & media)
static folders
published to gitlab pages
no backend processing or dynamic parts to resolve the API endpoints of a matrix server
gitlab pages _redirects are used to resolve .json static files as traditional matrix API endpoints
media are also hosted on the static server, and should resolve URLs mxc:// URI protocol scheme

idea & objectives

Be able to share content onto matrix, from an entirely static setup (no dynamic server content generation, no database, just files and folders).

For a starting point, try to server entirely static content; then try to generate this "exported static content", from a SSG style setup (jekyl, gohugo, gastby, astro etc.), and a netlify-cms/decap-cms configuration, Jamstack style.

The purpose is not yet to have DM communication, but more public content from rooms, displayed via other matrix clients, without being able to participate in it.

It is currently a bit tedious to make sure all files have the right keys, and that could be improved by defining the correct models of data needed to generate all the required files for a federated static matrix server that can "distribute" content.

Development & usages (installation)

The "easiest" way to get started, is in gitlab, to create a new-organization, and a repo new-organization/new-organization.gitlab.io.

That way the currect homeserver URL (without subfolder path) are used for the deployment in the manual server configuration (see deployment section).

To use this project as a javascript node script, we need to run the src/index.js file, which accepts passing in user arguments (in the form of URL query parameters).

npm install --save-dev matrix-static
add a build script into the project's package.json file

{
  "scripts": {
    "build": "matrix-static"
  }
}

This will output a ./build/ folder, with the static matrix server representation ready to be published (as static "pages" and redirect to json files).

The build command can be used once the project is setup with a configuration of the static matrix server. See the deployment section for example build and deploy.

use `npm link` to test script changes locally

It is possible to mock the installation of the project's script as npm package, when making local changes.

in this project's folder run npm link
in the test project folder run npm link matrix-static

As a result the local matrix-static project folder, will be used by the test project folder (see node_modules/{.bin,matrix-static}, which gives access to the matrix-static javascript executable).

the `matrix-static.json` static server configuration file

In the root directory of the project, create the matrix-static.json file, with the following minimal configuration for our new static matrix server.

{
  "m.server": "mx-static-test.gitlab.io:443"
}

Where:

m.server is the URL of the (home)server, as in the matrix user ID @user:homeserver.tld (without the user, the domain extension and the @:. characters) used. The port 443 part is not necessary, as this script will alway assume to be running over HTTPS.

the `./content/` folder

To represent the static homserver rooms and their events, we will use folder and files, orgnized as parents, children, and siblings in the ./content/ folder of the project.

For example the folder ./content/about/ will represent the room #about:homeserver.tld. In this case the name of the folder, about, will be the main room_alias, and the room_id for the static homeserver (as seen in the federation).

spaces are folders with children folders
rooms are folders with no children folder, and an index.json file (with minimal content {} empty object).
events are not yet represented

None of spaces, rooms, and events, can be customized, but currently their "alias" (also the ID), from the folder name. This part should not change, but will be customizable from the room/space state events (tbd).

deployment

How to deploy this project in different "free static pages providers". Gitlab pages are the recommanded way, as it only needs one account for the repository and deployment, and could be self hosted.

gitlab-pages

An example for a gitlab pages deployment, with the file .gitlab-ci.yml content:

image: node:latest

pages:
  stage: deploy
  script:
    - npm run build
    - mv build public
  artifacts:
    paths:
      - public
  only:
    - main

This ci/cd job should re-build/deploy a static matrix server (representation) for each commit on the main branch

github-pages

Not yet tested.

cloudflare, netlify and other "pages-providers"

Generally for other static pages providers (usually can link to the git(lab,hub) repository of a project).

create a new "pages project" at a provider
use npm build as the build command
use build as the build output folder, to be deployed (as "public folder")

For cloudflare and netlify, the _redirect file generated by the script, is the same as for gitlab pages, so it should work.

For other providers (vercel etc.), it has not yet been tested.

Docs:

npm package publication

In the gitlab-ci publish job definition, the project will be published as a npm package for each new tag created in the gitlab UI.

A new tag needs to start with v, followed by the matching package.json.version number, such as v0.0.1, in order to trigger the publication job.

The NPM_TOKEN is only available on protected branches (main) and tags (v*).

todo

figure what _matrix endpoint work, and are needed (and which cannot work at all)
server-server API https://spec.matrix.org/v1.8/server-server-api
client-server API https://spec.matrix.org/v1.8/client-server-api/
widgets API, for widgets
matrix users from an other homeserver should be able to join a room to subscribe to updates? (can that work)
other SDK and clients (ex: https://libli.org) should display the content correctly (without changes to their code)
try to write a digest of the matrix spec for the static context
improve to local/offline/url first matrix CMS for static content
test usage with SSG/decap-cms, and with web-components

notes

Things useful to know.

How to add a new "directory server" for room listing in element

On https://app.element.io:

open the "search for rooms" (Ctrl + K) modal interface
tap "Public Rooms"
above the suggestions, tap the "server selection" dropdown
tap "add new server" and add matrix-static.gitlab.io, click "add" (even if it seems to error)

At this point the modal interface should display the list of rooms staticaly served by the gitlab pages.

The element client is making this request:

https://matrix-client.matrix.org/_matrix/client/v3/publicRooms?server=matrix-static.gitlab.io&limit=1

some `r0` and `v1` and `v3` endpoints are needed

Not all endpoints used by clients are under the same version number.

`_matrix/v3/` folder required for element

A request is made to this endpoint, https://matrix-client.matrix.org/_matrix/client/v3/publicRooms?server=matrix-static.gitlab.io&limit=1

Otherwise cannot add matrix-static.gitlab.io as a new server in element for the "explore public room" function (from adding a new server for the room directory).

the `_redirects` file

This npm package script generates the file _redirects, so the "pages providers" can redirect the API endpoints expected by matrix clients (eg. /publicRooms), to the static files generated by this package (eg. /publicRooms.json).

All providers do not support this file and syntax, and not all providers supporting it, handle it the same way. Right now we're defaulting to gitlab's (inherited from netlify, and applied by cloudflare too).

download link for the latest gitlab pages artifacts

To see the resulting generated ./build/ folder and file structure, for this project ./content/ folder, check out the pages job artifacts.

room previews

In order to get room preview in element, we can implement different API endpoints.

docs https://spec.matrix.org/v1.8/client-server-api/#room-previews

gitlab pages deployment

Notes on using gitlab pages as the static host for the site/data.

pages URL and subfolders

Using a new gitlab organization, so we can use the root domain matrix-static.gitlab.io gitlab pages. That way we might avoid issues with having this projects content served on a subfolder example-org.gitlab.io/matrix-static/ for the _matrix api endpoints etc.

content resultion and JSON

On gitlab pages, for the static file /.well-known/matrix/server.json, to resolve the same file, with or without its extension, it is possible to use the pages redirect feature.

# example url resolving to the same file, if the redirects are setup
matrix-static.gitlab.io/.well-known/matrix/server.json
matrix-static.gitlab.io/.well-known/matrix/server

The purpose is to mimic the behavior of a traditional (dynamic) matrix server (such as synapse, dendrite, conduit etc.), from static data, moved from git to a gitlab pages.