@toa.io/filesystem

Toa

Runtime for low code eventually consistent distributed systems.

Status

The project is under heavy development, while being used in production by its authors. Non-scattered documentation is coming some time later.

Features

Runtime

• Interservice communications
  • Reliable RPC and Events (ComQ)
  • Flow control
  • Eventual consistency guarantee
  • Uniform interface
    • Input/query segregation
    • Successful rejections: a refusal is an answer rather than a failure
    • Readonly chains: a call that may only read is refused where it reaches an operation that may change state, however far down the chain
    • Distributed exception handling: a failure reaches whoever is waiting, or is tried again and then kept
    • Tasks: a call nobody waits for — accepted, kept, and run later by whichever process is free to take it
  • Contracts: a component is given what another one provides, at the version its deployment names
  • Multi-protocol transmission
    • built-in support for AMQP and HTTP
  • In-memory communications
  • Messages validation
• Persistent state management
  • Concurrency control
  • Data validation
  • Multi-document transactions
  • Transactional outbox: state and its events commit together
  • Transactional inbox: the same call arriving twice changes state once
  • Convergence: deployments of one context, each with its own database, converging on the same entity state
  • Invariants (guards)
  • Migrations: indexes and data changes, applied once per database
  • MongoDB
• Cadence: calls that happen on their own time
  • a pulse calls a component's own operation on a cadence, with no schedule stored anywhere
  • a delay hands one call over to be made later, and answers the id that cancels it
• Continuity: a task that raised picks up where it failed, rather than making again what had already answered
• Configuration with secrets
• API Gateway
  • Resource discovery: every route an application serves, at one path, and a page that reads it and calls it
  • Identity
    • Basis authentication
    • Identity federation (OIDC)
      • id_token authentication
      • Authorization code flow
    • Passkeys (WebAuthn)
    • OTP authentication
  • Access control
  • Cache control
  • File uploads, downloads, and processing.
  • Throttling: a distributed GCRA, with no per-request I/O
  • OAuth 2.1 authorization server: discovery, dynamic client registration, PKCE
  • JSON-RPC: every method the tree exposes, called as a procedure named after its route
  • Model Context Protocol: a method is a tool where it says so, and a model calls it as any other caller does
• Atomicity: what the replicas of a component decide together
  • Partitioning: an exclusive claim on one of a fixed number of slots
  • Distributed lock manager
  • Shared rate metering
• Telemetry
  • Structured logs: to stdout, to an OTLP endpoint, or both
  • Distributed tracing
  • Metrics: unsampled, and what the runtime records by zone
• Introspection: product topology collection and visualization
  • Halt: a deployment stops, holds nothing open, and comes back on its own — and is taken down only once the map says nothing was running
• Realtime streams
• BLOB storage
  • MIME type detection and validation
  • Providers: file system, Amazon S3, Cloudinary
• Transient state
• Built-in Fetch with retries and telemetry

Development

• Language interoperability
  • Node.js
  • Bash :)
• Service prototyping (inheritance)
  • Generic prototype
• Data and operations schemas
• Declarative API Gateway endpoints with authorization policies
• Integration tests SDK
• Runtime bootloader API
• Development environment configuration
• CLI
  • toa mono — composition and extension services in one process
• Extensibility:
  • Persistent storages
  • Communication protocols
  • Runtime core abstractions
  • Arbitrary services

Operations

• Standardized containerization
• Compositions: the components, and the extension services, deployed as one pod
• Resource management
• Environment variables
• Standalone infrastructure configuration
• One-command fully automated deployment
  • Preset multi-arch docker images
  • Helm deployment to kubernetes
  • Single-image (toa deploy --mono) deployment
• CLI utilities
  • Secrets management
  • Remote container shell