Harvest-docs-cli NPM

Documentation for Harvest users

The Harvest CLI documentation for users can be found in the docs folder.

Documentation for Harvest developers

Last updated by Tyler Dillon on 2/5/2020

Folder structure

Here I'll describe the folders & files that are relevant for jumping in quickly and getting your bearings.

`/src/harvest.js`

Entry point through which all commands typed in by a user will pass. This file passes commands from the user to the files in the /src/commands folder.

`/src/harvest-config.json`

Configuration file holding values that get referenced throughout the code. Most have to do with dev portal repo, like the URL, name, branch, etc.

`/src/commands`

This holds the files that represent each available command. One file per command, and the file name is the same as the command it represents (not gospel, just makes it easier).

That command will then call the appropriate functions from /src/lib/ based off what the command is supposed to do.

`/src/factories`

Separated by 'business object' (to the best of my ability), the subfolders in this folder all represent some sort of noun that we interact with. This is an attempt at getting into some clean architecture practices. You can think the factory folders as lego pieces used to compose larger functions.

Let's look at /src/factories/folder, for example.

Each folder within /src/factories/folder contains a single function. Those functions all get imported into /src/factories/folder/index.js.

/src/factories/folder/index.js exports an object that contains different keys representing the functions.

module.exports = {
  checkIfExists: v => require("./check-if-exists")(v),
  createFolder: v => require("./create-folder")(v),
  deleteFolder: v => require("./delete-folder")(v),
  getFolderContents: v => require("./get-folder-contents")(v),
  walkFolderSync: v => require("./walk-folder-sync")(v)
};

Going up a level, if you go to /src/factories/index.js, you'll see this:

const command = require("./command");
const file = require("./file");
const folder = require("./folder");
const git = require("./git");
const gitlab = require("./gitlab");
const harvest = require("./harvest");
const log = require("./log");
const mattermost = require("./mattermost");
const metadata = require("./metadata");

module.exports = {
  command,
  file,
  folder,
  git,
  gitlab,
  harvest,
  log,
  mattermost,
  metadata
};

This is just importing each factory object and exporting it all as one big object. So, this means that if you import factory from /src/factories/index.js, you have all the factory methods available to you. For example:

const factory = require("../../factories");

const newFolder = factory.folder.createFolder({ dir: "/path/to/folder" });

factory.log.logSuccess();

For further ease of use, you can destructure the individual factories when importing ...

const { folder, log } = require("../../factories");

const newFolder = folder.createFolder({ dir: "/path/to/folder" });

log.logSuccess();

To see an example of this concept in action, check out /src/lib/04_harvester.

The idea behind this structure is to keep code organized by business object, and to keep functions as simple, standalone, and testable as possible.

`/src/lib`

The contents of this folder are an attempt to organize the different actions taken by a given command.

Each folder within /src/lib contains a single function. You can think of each of these functions as a machine built from the lego pieces in the /factories folder.

/src/lib/04_harvester is the biggest monolith of them all, though until I have compelling reason to divvy its responsibilities further, a monolith it shall remain.

Flow of data

`/src/harvest.js`

This is the entry point to the code. A command comes through from the user, and is run through the processCommand function located in /src/lib/01_process-command.

processCommand outputs an object which contains a primaryCommand, which should either be reap, check, or help at the time of this writing (2/5/2020).

If the primary command is valid, it'll then be passed to either /src/commands/reap.js (reap, check) or /src/commands/help.js (help).

If it's invalid, the help info will be provided to the user via the terminal, and the process will abort.

`/src/commands/reap.js`

Harvest will use /src/lib/02_check-missing-options to check for missing options in the command.

If required options are missing, info will be provided to the user via the terminal, and the process will abort.

If required options are present, they'll be passed to /src/lib/03_create-harvest-options to create harvestOptions, an object containing all the necessary info to run the harvest.

harvestOptions will then be passed to /src/lib/04_harvester.

Example value for harvestOptions:

{
  "branch": "master",
  "commitMessage": "h-2020-01-08T12-04-05-0500-harvest-docs-cli",
  "devPortalCloneUrl": "https://oauth2:abc123@git.carekinesis.net/mass/dev-portal-server.git",
  "docRepoHarvestFolder": "docs",
  "docRepoName": "harvest-docs-cli",
  "gitConfig": ["user.name=bsmith", "user.email=bsmith@gmail.com"],
  "harvestHomeDir": "/Users/bsmith/.harvest",
  "harvestHomeDevPortalRepoDir": "/Users/bsmith/.harvest/dev-portal",
  "harvestHomeDocReposDir": "/Users/bsmith/.harvest/repos",
  "mmUsername": "bsmith",
  "processDateTime": "2020-01-08T12-04-05-0500",
  "repoCloneUrl": "https://oauth2:abc123@git.carekinesis.net/mass/harvest-docs-cli.git",
  "token": "abc123",
  "types": "MD"
}

`/src/lib/04_harvester`

And now we get to the true heart of harvest.

Step 1: Creating harvest home directories

Creates the folders that will be used for the harvesting process. Uses strings passed in from harvestOptions to create the paths:

"harvestHomeDir": "/Users/bsmith/.harvest",
"harvestHomeDevPortalRepoDir": "/Users/bsmith/.harvest/dev-portal",
"harvestHomeDocReposDir": "/Users/bsmith/.harvest/repos"

If successful, move on to step 2.
If unsuccessful, harvest logs the error and aborts

Step 2: Syncing repos from gitlab

Clones the desired dev portal repo/branch into /Users/bsmith/.harvest/dev-portal. Clones the desired documentation repo/branch into /Users/bsmith/.harvest/repos.

If successful, move on to step 3.
If unsuccessful, harvest logs the error and aborts the harvest.

Step 3: Ensuring "\${docRepoHarvestFolder}" folder exists

Checks to make sure the designated docs folder actually exists in the documentation repo.

If successful, move on to step 4.
If unsuccessful, harvest logs the error and aborts the harvest.

Step 4: Validating files are OK

Pulls all the filenames from the designated folder in the documentation repo.

Maps through each file name. During the map, it gleans info from doc's contents, and builds some values that we will need later.

Each file, once mapped, will return an object like this:

{
  "filePath": "/Users/bsmith/.harvest/repos/foo-repo/docs/foobar.md",
  "fileName": "foobar.md",
  "validExtension": true,
  "extension": "md",
  "frontMatter": {
    "id": "foobar",
    "api": "foo-api",
    "title": "Foo Bar Doc"
  },
  "accessLevel": "public",
  "okToHarvest": true,
  "newFileName": "foo-repo_foobar.md"
}

Then, we look through that newly created array to check for files that are harvestable vs files that aren't.

If there are harvestable files, harvest logs out the names of files to be harvested, and moves on to step 5.
If there are unharvestable files, harvest logs out the names of unharvestable files.
If there are NO harvestable files, harvest logs the error and aborts the harvest.

Step 5. Copying OK files to dev portal

Harvest then copies all the harvestable files from the doc folder to the dev portal folder.

If successful, move on to step 6.
If unsuccessful, harvest logs the error and aborts the harvest.

Step 6. Generating new metadata to represent the new docs

First, harvest looks through the harvested files for non-image files.

Then, harvest loops those files through metadata.createMetadata, which generates a metadata object that represents that document and includes the processDateTime.

This results in an array of metadata objects representing all the freshly harvested docs.

Then, harvest grabs the existing metadata array from the dev portal.

Then, harvest reconciles the new metadata with the existing. It does this by looking through the existing metadata, removing any objects that represent older versions of the incoming documents, and then adding in the new metadata objects.

This results in an array of metadata objects representing all the documentation in the dev portal, including the new metadata.