data-kraken v2.1.0

data-kraken

A command line tool that fetches info about users, commits, repositories, Docker images and npm dependencies from GitHub.

How to install

• Check out the source code
• Use correct Node.js version:

nvm use

• Install dependencies:

yarn install

• Create .env file:

cp .env.example .env

• Put your personal GitHub access token in the .env file (instructions)

How to run

yarn data-kraken

(this will display a help message to get you on your way)

Usage examples

Tech Debt

Shows a technical debt score for one or more GitHub repositories.

yarn data-kraken tech-debt --org mobile-de --repo consumer-fe

The repository parameter is optional:

• if repository is provided, the output will show the score along with some improvement hints
• if repository is omitted, the output will show a list of all the repos in the org, ranked according to their tech debt score

Configuration

To determine the tech debt score, the program analyses the Dockerfiles and package.json files of the repositories and assigns tech debt scores for dependencies that are outdated or banned. The algorithm uses a YAML config file to do this:

• config/tech-debt-evaluator.yaml

More info

Run --help for more info on the tech-debt command:

yarn data-kraken help tech-debt

Inactive

Shows the level of inactivity of GitHub repository. The inactivity score is a value from 0 to 100, 0 being a repo that currently gets updated every day and 100 being a repo that has not been updated in a very long time.

yarn data-kraken inactive --org mobile-de --repo consumer-fe

The repository parameter is optional:

• if repository is provided, the output will show the repo's inactivity score along with additional info on how the score is componsed
• if repository is omitted, the output will show a list of all the repos in the org, ranked according to their level of inactivity

More info

Run --help for more info on the inactive command:

yarn data-kraken help inactive

Docker images

Shows the Docker images used in the specified GitHub org and repository, found in a search of all the Dockerfiles in each repo.

yarn data-kraken docker-images --org mobile-de --repo consumer-fe

Repository is optional, if omitted, the whole org is searched.

Search expressions

You can pass a regular expression to match the images against. In the simplest usage example, the expression can just be a search term:

yarn data-kraken docker-images --org mobile-de node

This will give you a list of repositories that use a Node.js image.

More advanced example:

yarn data-kraken docker-images --org mobile-de "^.+/shared/node1[46].+$"

This will list all the repos that use dock.es.ecg.tools/shared/node14 or dock.es.ecg.tools/shared/node16, but not dock.es.ecg.tools/shared/node12.

• See also: Regular expressions

More info

Run --help for more info on the docker-images command:

yarn data-kraken help docker-images

Npm packages

Shows the npm packages that repositories are dependent on according to their package.json files.

yarn data-kraken npm-packages --org mobile-de --repo consumer-fe

Repository is optional, if omitted, the whole org is searched.

Search expressions

You can pass one or two regular expressions to match the package names or versions against. In the simplest usage example, the expression can just be a search term:

yarn data-kraken npm-packages --org mobile-de react

…gives you results for packages that have “react” in them (e.g. react, react-dom, react-router, etc.).

More advanced example:

yarn data-kraken npm-packages --org mobile-de ^react$ "^[~^]*1[68]{1}"

…gives you results for precise package “react” with major versions 16 or 18.

• See also: Regular expressions

More info

Run --help for more info on the npm-packages command:

yarn data-kraken help npm-packages

Repos

Shows info about the repositories a user contributed to in “pretty print” on the console:

yarn data-kraken repos patrick-hund

You can use the --org option to constrain output to a specific GitHub org:

yarn data-kraken repos --org mobile-de patrick-hund

You can specify multiple users:

yarn data-kraken repos patrick-hund daniel-korger uwe-loydl

• Caveat: Data time range

More info

Run --help for more info on the repos command:

yarn data-kraken help repos

Files

Shows info about what kinds of files the user modified (frontend or backend):

yarn data-kraken files patrick-hund

As with the repos command, you can specify multiple users and a GitHub org. In addition, you can also constrain output to a specific repository:

yarn data-kraken files --org mobile-de --repo consumer-fe nina-maass

• Caveat: Data time range

More info

Run --help for more info on the files command:

yarn data-kraken help files

Options

CSV output

To facilitate importing the output into a Google Sheet, you can specify CSV format:

yarn data-kraken repos --format csv patrick-hund

…or…

yarn data-kraken files --format csv patrick-hund

This is particularly useful when using multiple users. You can pipe a list of usernames into data-kraken using xargs and store the output in a CSV file, like this:

cat users.txt | xargs yarn data-kraken files --format csv > files.csv

You can then upload and import the CSV file into Google Sheets.

• See also: CSV date format

JSON output

You can also have data-kraken deliver its output in JSON format, for example:

yarn data-kraken npm-packages --org mobile-de --format json

Verbose output

All commands support a flag for getting more verbose output:

-v or --verbose

The effect of using verbose mode is different depending on the command and the format type.

Caching

When executing a command, data-kraken does a lot of requests to the GitHub API, which can take a long time. Be patient when executing a command that you haven't used before!

For subsequent command executions, data-kraken uses cached data from previous API calls to speed things up.

The time to live of the caching can be configured through the environment variable FETCH_CACHE_TTL. In .env.example, this is set to 86400000 milliseconds, which is one day.

Additional notes and caveats

Data time range

For commands related to users (e.g. repos, files), data-kraken fetches commit data of the users.

We fetch data from GitHub as far back as it is allowed to by constraints of the GitHub API. This is usually data for around two weeks, depending on how active the user was (less activity – data ranges further back in time).

Regular expressions

Some hints on how to use regular expression with commands that support them (e.g. docker-images , npm-packages):

• Specify regular expressions without enclosing forward slashes
• Providing regular expression flags (g, i, u, etc.) is not supported
• The search is always case-insensitive
• Complex regular expressions need to be quoted, otherwise your shell will complain because it tries to evaluate the expression

CSV date format

For commands that create CSV data with times in them (e.g. repos, files), importing the CSV file in Google Sheets works best if you set the LOCALE and TIME_ZONE options in your .env file to the locale and time zone your Google Sheets is set to. Then dates and times will be imported properly as dates you can calculate with rather than mere strings.

If your Google Workspace is in German, for example, you want to specify LOCALE=de-DE. If you are located in Toronto, you want to specify TIME_ZONE=EST.

Default locale is English / Great Britain (en-GB) and Barcelona / Berlin / Amsterdam time (CET).

Contributing

Tests

This package uses Jest for automated testing.

Running tests

To run unit test:

yarn test

Style considerations

Write unit tests mostly for low-level functions that have lots of different input to make sure that they return the expected result. Use test and test.each instead of describe and it.

• Code example: getTechDebtScore.test.js

Terminating with error

Whenever the program encounters a situation where it can't continue, e.g. network errors from API request attempts, it should terminate with an error code. Use the function die in these cases, supplying an error message:

import die from "./utils/die.js";

die("Failed to execute command");

Using the GitHub API

The codebase provides a package with utility functions for fetching data from the GitHub API.

Main API functions

The main function for fetching data are:

• fetchResult – given a REST API path and an optional result page, fetches the result from that path
• fetchSearchResult – given a search query and an optional result page, fetches search results

Additional API utilities

This program includes numerous ways to reduce the number of requests to the GitHub API while making it resilient against connection problems and improving performance.

If you implement additional commands that fetch data from GitHub, you need to use these the same way the existing commands do:

• inBatches – executes fetch commands in batches rather than executing them all at once
• withPagination – fetches paged results one page after another
• withRetry – retries API requests if they fail
• fetchWithCache – caches fetch results using the local file system; note: this is already built-in into fetchData, so you'll only need this when implementing your own fetch function ( see Caching)

Debugging

You can turn on a debug logger through the environment variable DEBUG, example:

DEBUG=* yarn data-kraken docker-images --org mobile-de

This will print log statements to the console that are created through the log function.

The asterisk argument in the above example means show all log statements; you can only show specific log statements by specifying a logger name.

The logger name is the relative path to the logging JavaScript module, prefixed with data-kraken:, with forward slashes replaced by colons and without the file extension.

For example, the logger name for module src/commands/dockerImages/run.js is data-kraken:commands:dockerImages:run, and you can show only log statements from this module with this command:

DEBUG=data-kraken:commands:dockerImages:run yarn data-kraken docker-images --org mobile-de

• See also: debug lib documentation on GitHub

Object logging depth

Objects are logged only up to a certain depth. You can increase this depth with the environment variable DEBUG_DEPTH.

Adding log statements in the code

You can add log statements to any module through the utility function createLogFunction, like this:

import createLogFunction from "./utils/createLogFunction.js";

const log = createLogFunction();

log("I'm a happy camper");

The logger name will be set automatically. You can override this behaviour by providing a name as a string argument to createLogFunction:

const log = createLogFunction("my awesome logger");