@kumori/kam NPM

= kam (Kumori artifact Manager)

This package implements kumori module management.

It installs the kam tool, as well as the cue program, needed to process the artifacts developed for the kumori platform.

In addition, kam also installs the kumori platform cli tools, kumorictl, and kumorimgr.

== Kumori artifacts, packages and modules A Kumori artifact describes a microservice in a way that allows the kumori platform to both manage its deployments, as well as integrate it within larger microservices.

A Kumori microservice can be described either as a component or as a service application. Descriptions or Kumori artifacts are carried out using the cue specification language, on which the kumori service model is defined.

The definition of a Kumori artifact consists of a cue package. In turn, cue packages ar organizaed within cue modules. Thus, kumori artifacts will be bundled within kumori modules themsekves.

A cue module is the unit of versioning and distribution in cue, and kumori leverages also this structure.

Ideally, kumori artifact management would piggyback on top of a cue module management, but at this moment, the cue ecosystem is lacking such a framework and set of tools. kam will fill this void, with a view to integrate with the cue ecosystem once it is more developed.

== cue module management At this point in time, the module/package definition in cue affects mostly to its loader. Thus, kam must take into account how that loader works in order to provide a proper organization of packages to work with the kumori tools.

=== The cue loader cue 's loader expects packages to live within a folder structure representing a module, where the root forlder of the module is recognized because it contains a cue.mod folder and within it, it contains a module.cue file with a module field whose string value is the name of the module.

If a cue package (e.g., package A) wants to use package B that is part of module with name M, it needs to use an import sentence that includes the package import path for B .

The package import path for package B is formed by concatenating its module name as stored within the special cue.mod/module.cue at the root folder of the module M to which B belongs, with the path from that root folder to the folder where the contents of package B are placed, and, finally, with the suffix :B.

Thus, A needs to have the following import sentence in order to use B.

source,cue

import ( "M/:B"

)

The suffix :B can be dropped from the import path if the basename of <path-to-B-from-root-of-M> is B itself..

Note that no special provision is made for imports between packages of the same module. Thus if in the above example A and B are within the same module M, the import sentence for A still needs to have the form shown above.

==== How does cue find the packages being imported.

When a cue file A imports package B, cue first finds A's module by traversing the folder structure upwards from A 's folder (until it finds the special cue.mod/module.cue, moment at which it determines it has reached the root of the module for A).

The loader then checks if A's module name is a prefix of the import path. In that case it assumes that the the module for B is the same as that of A and it finds the contents of B by traversing the rest of the path minus the prefix of the module's name.

Otherwise the loader starts searching within various subfolders within cue.mod, including the pkg, gen and usr subfolders. The search proceeds trying to match the package import path exactly with a path within pkg (and gen, usr), minus the :B suffix, if any.

Note that the assumption is that all leaf paths in cue.mod/pkg have two parts. An Mp part, with a path leading to a cue module's root directory, and a Pp part, leading to a package containing folder at the leaf. Thus, once the loader finds a path in pkg matching an import path, it ought to check that the module name found at the module root, M within the path corresponds with the prefix of the path at that root folder. I.e. Mp == M. When such condition is met the contents of the full path are loaded to satisfy the dependency, in accordance with how the loader actually picks up the contents of the package (see https://cuelang.org[])

Note, however, that the loader performs no check whatsoever on the imported package's module. It only pays attention to the folder structure...

Moreover, nested imports are not encapsulated. E.g., if A import B, which, in turn, imports C assuming all of them are in different modules, the logical structure would be for A's module to contain B 's module within A 's own cue.mod/pkg, and B 's module to contain C's module within B 's own cue.mod/pkg. However this does not work, as when processing packages from A all package imports, including those coming from B are searched within A 's cue.mod context. Which means that, in order for B to find C, C's module must be placed within the cue.mod/pkg forlder of A.

The above makes it difficult to bundle up a module and all its dependencies toghether to satisfy the way CUE wants to find imported packages.

==== Kumori's approach to circumvent CUE's loader

When a workspace is set up to work with Kumori modules, the workspace is configured as a CUE module, with its own cue.mod folder. Within that folder, a link is made from cue.mod/pkg to the module cache in a context of the user's choosing. By default, this context will be associated with the user's home directory in a specific subpath (e.g., .kumori/context).

Kumori modules are fully versioned, being the version part of the import path of the module. A complete module import path (its unambiguous name) has the following shape:

source,cue

//...//@

A Kumori module can then be installed (even multiple versions of the same module). When installing a module, it is placed, with the directory structure of its import path, within the user context's cache folder, making it available to any workspace linking to it.

To make this work, the author of a module, before publishing it, needs to build it. The process of building a kumori module consists of substituting all module import paths appearing in the cue files of the module for the import paths including the actual version of modules depended upon (the full module name as exemplified above).

The build process needs the module author to declare its dependencies somewhere. The place Kumori chose to keep metadata for kumori modules (for example, module dependencies) is the kmodule.cue file at the root of the CUE module of the workspace.

The kmodule.cue file declares the kmodule package which can be imported by the rest of the packages in the module.

To make it less cumbersome for packages in a module to depend on each other without needing to know the actual name/import path of the module they reside in, kumori modules' cue.mod/module.cue file contains exactly the same for any module: module: "...", that is, within the workspace the name of the module (its import path) is simply "...". Now, importing package B will only need an import path like so: import " .../B".

When building the kumori module, kam will substitute the references to ... with the full import path of the module being built, with its full semantic version qualification.

The result of the build process is a folder mirroring the structure and contents of the actual module's root folder, but where all import statements have been bound to both, the full name of the module (for paths starting with ...) and the full names of the imported modules. This build folder can finally be packaged within a tar tgz archive, and placed in suitable registries for distribution. To make it possible to verify the contents of a module, the build process also produces a checksum of the package.

This approach guarantees an effect similar to bundling a module with its exact dependencies guaranteeing that a module uses exactly the dependencies it declared when it was built.

Note that packaged module does not need to include a cue.mod folder, as it is not used for anything by the loader when importing from imported packages (as is the case with installed modules). An active workspace, by virtue of its link cue.mod/pkg to the context's cache will be able to access all

== Structure of a kumori workspace

A kumori workspace IS a cue module, with additional restrictions. As such, it is represented in a folder containing a cue.mod folder with a cue.mod/module.cue file, in which there is a module field whose value is "...".

In addition, a Kumori workspace also fulfills the following requirements.

It contains a cue.mod/usr subfolder, and a cue.mod/pkg symbolic link to a cache of installed kumori modules. There may be at times other subfolders of cue.mod, like cue.mod/gen, containing generated artifacts from the main files in the module folder. In particular, if the workspace ends up as a kumori module, cue.mod/gen will be used as the build folder.
When the worksapce is for a kumori module, there is a kumori.cue file, at the root folder, belonging to the kmodule package. This file contains metadata relevant to the kumori module, like version, name, and dependencies.
Each artifact resides within its own subfolder of the root module. The artifact's CUE package name must coincide with the basename of its name. Also ideally, the artifact's folder should contain an artifact.cue file and a ref.cue file, both within the artifact's package.
The artifact's directory path from the module's root directory is the artifact's name.
Every artifact package MUST provide a #Artifact definition. This definition should comply with the #Artifact definition of the Kumori service model for the kind of artifact it represents

source,cue

#Module: { // The versions of cue this module has been tested with cue: ...string

// the domain of the set of artifacts it contains domain: string // The name of the module minus the domain name name: string

// NOTE: the typical situation is that the CUE module // is formed by path joining the domain with the name.

// Sem version of this module semver: { version: uint,uint,uint prerelease?: string buildmetadata?: string }

// The dictionary of module dependencies. // The key is an alias for the module's import path // The value is a full module import path name (complete // semantic version value) dependencies: string: string // The dictionary of checksums // A key is a complete/full module name, including semantic versioning // the value is the checksum associated with the depended module. sums: string: string // Names of artifacts defined artifacts: ...#ArtifactRecord }

#ArtifactRecord : { name: string kind: "component" | "service"

}

=== Kumori redistributable modules

Kumori modules can be created from workspaces by building them as explained above. A Kumori does not contain a cue.mod folder, it just contains the kmodule.cue file.

== The kam tool

The kam tool is in charge of managing Kumori artifacts through manipulation of CUE modules and packages.

kam complements and uses kumorictl/kumorimgr. In addition, the kam package can be used as a library that can be used to implement other useful functions.

kam implements the following functionality:

Compute the sum of a file as per golang.
Compute the sha of a folder, also as done in golang
Edit the version in various ways
Tag the current version if on a git repo
Add dependencies
Add dependency imports to a file
Create/Init a module
Create/Init an artifact (within a module)
Register modules globally in a context (with location method and checksum)
Bring modules into the global cache for their use
- Modules can also be general CUE modules, although in this case, dependencies are not pursued (e.g., kumori.systems/kumori)
Building and packaging kumori modules
Setting up contexts
Publishing kumori module packages
Creating Deployment manifests and using kumorictl to launch them (TBD)
Managing multiple cluster tarhets for deployment (TBD)
Managing multiple kumori module registries. (TBD)

...

=== KAM, modules and workspaces

kam module init <name> creates a folder cue.mod within the folder where the tool is launched and initializes it with module.cue, so that the module name in module.cue is constant "...".

Also in links cue.mod/pkg to the user's kumori context's cache. It also creates the cue.mod/usr folder, to place there links from the dependency aliases to the actual modules they depend on.

Additionally, the command creates and initializes a kmodule.cue file at the module's root.

=== KAM and artifacts Artifacts defined within the module should fill their spec/ref fields.

kam module <artifact kind> init, creates a component/service artifact at the current folder. Artifact creation within a module proceeds by creating the files ref.cue and artifact.cue at the current folder (options exist to use a different folder).

ref.cue defines the spec/ref parts of the manifest, and should not be modified by the artifact author, as its contents are fully generated by kam.

On the other hand, kam fills artifact.cue with a skeleton implementation of the artifact, with only the parts that the author has to fill for the functionality it is implementing.

== Module management

The main purpose of kam is to help author kumori modules and artifacts within them, as well as prepare anc onfigure deployoments.

Part of the specification of a module is the set of dependencies it relies on. The main dependency is on the kumori model itself. The rest will typically be on other kumori modules implementing artifacts that play some role within the artifacts of the module being authored.

kam provides help to handle such dependencies, and it also helps manage installation of the depended modules, so that cue processing can be carried out for testing, before publishing the module.

Besides the current module being authored, kam also manages a kumori context. This context at the moment consists of a module registry, plus a cache of downloades modules. The registry serves the purpose of helping locate the module's contents. The cache stores the modules that have been downloaded.

There is also a weak reference count for modules that have been linked to by some module being authored.

The deafult directory context is in $HOME/.kumori. Within it one can find the context.json file, containing checksums and location information for registered modules, and the caching subdirectory. The caching subdirectory has been organized as if the context root was a CUE module.

Thus, the caching directory resides within the cue.mod/pkg subdirectory of the context directory. Within it cached modules are organized as expected by the CUE loader. The caching module receives the name deployments

The decission to adopt this organization will be made clear whe we introduce deployment support.

=== Adding a dependency to a module To add a dependency to a module, an alias for the dependency should be provided as well as some sort of version specification for it indicating which name/version pairs satisfy the dependency.

When adding such dependency, the context is queried to return a full name (vith semver) for the module as well a checksum. Then kam proceeds by adding an entry in kmodule.cue for the alias provided, to the full module name returned by the context. At the same time, it adds a checksum entry for that full module name.

Finally, it adds a link within the cue.mod/usr folder from the alias to the folder in the context cache containing the actual dependency.

Acting this way guarantees that the cue files within the workspace can be processed by CUE.

== Module registries

NOTE: We implement a file registry as well as an npm-based registry

== Deployments and testing

Testing a specification needs the ability of generating deployments.

TBD

== GIT We advise the usage of a git repository to handle the creation of kumori modules. When a new module is created, we include at its root a .gitignore file whose contents configure git to not include the cue.mod directory.