@pesy/esy-pesy v0.1.0-alpha.14

pesy

pesy is a command line tool to assist with your native Reason/OCaml workflow using the package.json itself.

1. Quickly bootstrap a new Reason project
2. Configure your builds

Installation

npm install -g pesy@next

Why pesy?

esy is driven by a package.json to bring the familiar NPM workflow to the native world of Reason/OCaml. pesy takes this further by adopting NPM conventions to configure the build.esy's package.json first approach in creating developer sandboxes brings interesting possibilities to the table. pesy leverages on those features to make native development both convenient and familiar.

pesy provides

1. A bootstrapper script to quickly create a project template
2. An alternative JSON syntax around Dune that is NPM like
3. Built in tasks that take full advantages of esy's capabilities

Getting started

Creating a new project

pesy can quickly bootstrap a basic native Reason/OCaml project.

cd my-new-project/
pesy

Adding project dependencies

Say, we need @opam/yojson and @reason-native/console in App.re.

Place App.re in a folder (say, bin/?) and add the following to you package.json.

"buildDirs": {
  "bin": {
    "imports": [
	  "Json = require('@opam/yojson')",
	  "Console = require('@reason-native/console/lib')"
	]
  }
}

Run esy pesy (once). Run esy after that to create builds.

pesy abstracts library identifiers (in this case yojson and console.lib) in file paths conceptually. Use native dependencies from OCaml like you did with NPM!

Where pesy shines

However, pesy's is truly useful for frameworks that need a lot dependencies and configuration. Ex: Morph and Revery

pesy --template pesy/template-revery --directory my-new-project

If you are authoring a framework, create a template like pesy/template-revery and run the project on the CI with a setup that pesy creates for you.

You users get cached CI artifacts that will let can hydrate their local esy cache avoid long wait times! See Templates to see how to create such templates.

Looking for a tutorial?

Checkout A simple native Reason project with pesy to get an idea of what developing with pesy feels like.

Relation to Dune

Pesy as accepts package.json as input and producing dune files as output.

                  +--------+
package.json ---> |  pesy  | +--->  dune files
                  +--------+

Note that not all of Dune's features are supported in pesy (PRs welcome). pesy doesn't intend to duplicate Dune's efforts - it's meant to compliment it. For simple use cases, pesy wants to provide a NPM friendly interface to Dune so that newcomers can quickly get started, without confusing themselves with the library vs packages nuances.

Namespacing

Every library, as we know, exposes a namespace/module under which it's APIs are available. However, as package authors, it can hard to make sure one is not using a namespace already taken by another package (Otherwise it could lead to collisions). Pesy works around this by assigning the library the upper camelcase of the root package name and directory the library/sub-package resides in.

Example: if a package.json looks like this

{
  "name": "@myscope/foo",
  "buildDirs": {
    "library": { ... }
  }
}

Then, subpackage library takes a namespace of MyScopeFooLibrary. As a user, however, you shouldn't have to worry much about yourself, since you can specify how you'd like to import subpackages (and packages). In the above example, another subpackage would consume it as follows

{
  "name": "@myscope/foo",
  "buildDirs": {
    "library": { ... },
    "anotherLibrary": {
      "imports": [
        "ThatOtherLibrary = require('@myscope/foo/library')"
      ]
    }
  }
}

And if you were consuming this package (after having published to npm), you can import it as follows:

{
  "name": "bar",
  "buildDirs": {
    "library": {
      "imports": [
        "ThatFooLibrary = require('@myscope/foo/library')"
      ]
    }
  }

With the new NPM like conventions, pesy automatically handles the namespace for you so that we don't have to worry about the nuances of a package and a library during development.

Dune files

pesy generates dune files on fly behind the scenes. And to be able to so, it needs a static dune file that looks like the following

(* -*- tuareg -*- *)

open Jbuild_plugin.V1

let () =
  run_and_read_lines ("pesy dune-file " ^ Sys.getcwd ())
  |> String.concat "\n"
  |> send

These have to created only once - after that they never change (unless you decide to eject). Every bootstrapped project has them already and you dont have to create them. In case, you need to re-generate them, run esy pesy (or esy @mysandbox pesy if you are using esy sandboxes). A common need for running esy pesy is when you add a new folder to your project.

Ejecting:

It is always possible to eject out of pesy config by running esy pesy eject ./subpackage-path

CI & local cache Warming up

Compilation artifacts built on the CI can be downloaded by pesy warmup. At the moment, we only support Azure Pipelines can be configured as follows.

{
  "pesy": {
    "azure-project":
      "<azure-project>/<azure-pipeline-name>",
    "github":
      "<github-org>/<github-repo>"
  }
}

This will fetch appropriate artifacts compiled for the current machine. Note however, the best way to get this feature to work is to use cache publish and restore mechanism provided bootstrapped files. pesy assumes that the artifact zip file names and/or paths haven't changed.

How it works

Compilation artifacts created by esy are relocatable - since esy sandboxes isolated, all the dependencies are accounted for and each dependency is loaded from a path with a fixed-length prefix. Such artifacts can be built on one machine and used on other provided the prefixes are rewritten to reflect the updated path on the new machine. And esy provides all the low level command to do this out of the box. pesy simply provides a convenient wrapper that drives them.

When the project is bootstrapped for the first time, it is identical to a copy that is run on the server. This is where the first set of cached builds come from, which is why the azure-project in the pesy config is set to esy-dev/esy and github to esy/pesy-reason-template.

Once the project sees changes, you most probably would add more dependencies (or remove some) which could change the build sandbox state. It is recommended that you run the provided CI setup to cache builds on your own Azure Pipelines instance and update github and azure-project accordingly.

Publishing and Consuming from NPM

Publishing libraries

Easiest way to get started with distributing you library is to publish the source to NPM.

Let's take a look at an example.

Consider a base package foo that you created and distributed on NPM. And let's assume, bar is the package that consumes foo.

$ pesy --directory foo-lib

This would have bootstrapped a project with the default template with Util.re in the library/ folder.

Let's publish it

$ npm publish

Consuming foo-lib

Let quickly create a new project, bar and add foo.

$ pesy --directory bar
$ esy add foo-lib

We can now require foo

  "buildDirs": {
    "library": {
      "imports": [
+        "FooLib = require('foo-lib/library')"
      ]
    }
  }

And then edit Utils.re

let foo = () => {
  FooLib.Util.foo();
  print_endline("This is from bar");
};

$ esy
$ esy start
Hello from foo!
This is from bar

Templates experimental

Usage

To use a custom template run pesy --template=github:your-name/your-pesy-template

Creating your own template

This is a experimental feature that could see a lot of churn. We request you to watch the issue tracker for updates.

It works by downloading a git repo and then replacing special strings in filenames and files inside the repo. The special strings are currently these:

In file names:

In file content:

Best way to get started creating a new template is to download https://github.com/esy/pesy-reason-template and work on it. Any changes can be tested with pesy test-template.

Supported Dune Config

This is reference guide explaining the config pesy supports.

Binaries

Configuration that applies to subpackages that create binary executables. Note that these executables can be ocaml bytecode or native binaries (ELF/Mach/PE)

bin : string

A subpackage produces binary when it contains a bin property.

{
  "buildDirs": {
    "src": {
      "bin": "Main.re"
    }
  }
}

Here is a complete, working example

modes : array(string)

An array of advanced linking modes. Each string should be of the form "compilation-mode binary-kind" where compilation-mode is one byte, native or best and binary-kind is one of c, exe, object, shared_object.

{
  "buildDirs": {
    "src": {
	  "bin": "Foo.re",
	  "modes": [ "native", "exe"] 
	}
  }
}

Here is a complete, working example

name (override) : string

A string that maps to Dune's public_name. Usually unnecessary (as bin property takes care of it) and must only be used to override.

main (override) : string

A string that maps to Dune's name. Usually unnecessary (as bin property takes care of it) and must only be used to override the entry point.

Libraries

modes : array(string)

modes can be used to configure the compilation target - native or bytecode An array of string, any of byte, native, best

"byte"

This mode generates byte code output

"native"

This mode generates native output

"best"

Sometimes it may not be obvious if native compilation is supported on a machine. In such circumstances, use "best" and "native" will be picked for you if it's available. Else, it'll be "byte"

cNames : array(string)

When writing C stubs to FFI into a library, simply mention the file name (without the .c extension) in the cNames field.

{
  "buildDirs": {
    "src": {
      "cNames": ["my-stub1", "my-stub-2"]
	}
  }
}

foreignStubs : list(ForeignStub)

From dune version 2.0 onwards the cNames field was removed and foreignStubs field was introduced to provide the FFI functionality, foreignStubs is a list of objects, where each foreignStub object should specify language, names & flags. Incase names & flags is not specified or empty, their default value will be considered Refer this for default values for names & flags

{
  "buildDirs": {
    "src": {
      "foreignStubs": [
        {
          "language": "c",
          "names": ["my-stub1", "my-stub-2"],
          "flags": ["-verbose"]
        } 
      ]
	}
  }
}

Here is a complete, working example

Config supported by both libraries and binaries

imports : array(string)

imports can be used to import a library (from a subpackage or an external npm/opam package) and utilise the namespace of the imported library.

{
  "buildDirs": {
    "src": {
	  "imports": [
	    "Console = require('console')"
	  ]
	}
  }
}

The above config makes a namespace Console available inside the subpackage src. Now any .re file inside src can use the console library.

// src/SomeFile.re
let foo = () => Console.log("Hello, world")

We can also import a package/subpackage under a different namespace

{
  "buildDirs": {
    "src": {
	  "imports": [
	    "NotConsole = require('console')"
	  ]
	}
  }
}

And we can import (oddly confusing) NotConsole from in src

// src/SomeFile.re
let foo = () => NotConsole.log("Hello, world");

We can also import other subpackages in the project.

{
  "buildDirs": {
    "src": {
      "bin": "Main.re",
      "imports": [
        "FooConsole = require('console/lib')",
        "MyOwnLibrary = require('../my-own-lib')"
      ]
    },
    "my-own-lib": {}
  }
}

Here is a complete, working example

Compiler flags

All of type list(string)

• flags - These flags will be passed to both the bytecode compiler and native compiler
• ocamlcFlags - These will be passed to ocamlc - the bytecode compiler
• ocamloptFlags - These will be passed to ocamlopt - the native compiler
• jsooFlags - These will be passed to jsoo compiler - the javascript compiler. Note: This is unrelated to Bucklescript

Preprocessor flags : list(string)

preprocess accepts options needed to pass the source files via a preprocessor first. When using custom syntax not natively supported in the compiler, we pass the sources in a subpackage via a preprocessor first.

For example, suppose we'd like to use let%lwt syntax for our Lwt promises

let%lwt foo = Lwt.return("foo");
print_endline(foo);

// instead of 
Lwt.return >>=
  (foo => print_endline(foo); Lwt.return())

We specify lwt_ppx in pps preprocess

{
  "buildDirs": {
    "src": {
	  "bin": "Main.re",
	  "preprocess": ["pps", "lwt_ppx"]
	}
  }
}

Here is a complete, working example

includeSubdirs : string = "no"|"unqualified"

Default is "no", and changing to "unqualified" will compile modules at deeper directories.

Escape hatches for unsupported Dune config

It's always possible that there are features Dune offers are needed and the options above are not enough. Use rawBuildConfig to add options in a given library or binary. Use rawBuildConfigFooter to add config to the footer.

rawBuildConfig : list(string)

Example

{
  "src": {
    "rawBuildConfig": [ "(libraries unix)" ],
    "bin": "Main.re"
  }
}

rawBuildConfigFooter : list(string)

Example

{
  "src": {
    "rawBuildConfigFooter": [
      "(install (section share_root) (files (plaintext.txt as asset.txt)))"
    ]
  }
}

Here is a complete, working example

Tutorials

Simple native example

CLI Apps

Web Development with Morph

Development

Clone the repo and run esy on it.

e2e tests

./_build/install/default/bin would contain (after running esy) Runner.exe. Runner.exe looks for PESY_CLONE_PATH variable in the environment to find pesy source. Set it to the path where the project was cloned.

To test if simple workflows work as expected. They assume both esy and pesy are installed globally (as on user's machines).

Changelog

version 0.4.0 (12/21/2018)

• Allow buildDirs to contain deeper directories such as "path/to/my-lib": {...}".
• Added support for wrapped property on libraries.
• Added support for virtualModules and implements - properties for Dune virtual libraries. (This will only be supported if you mark your project as Dune 1.7 - not yet released).
• Stopped using ignore_subdirs in new projects, instead using (dirs (:standard \ _esy)) which only works in Dune 1.6.0+, so made new projects have a lower bound of Dune 1.6.0.
• Support new properties rawBuildConfig which will be inserted at the bottom of the target being configured (library/executable).
  • It expects an array of strings, each string being a separate line in the generated config.
• Support new properties rawBuildConfigFooter which will be inserted at the bottom of the entire Dune file for the target being configured.
  • It expects an array of strings, each string being a separate line in the generated config.
• Support new properties modes for binaries and libraries list(string).