Vuln-func-validator NPM

vuln-func-validator

There are multiple components in this service, plus the ability to run it from the command line.

validate a snapshot: npm start java java-snapshot.json
run the server: npm start serve
run the node report (including hodor):

(cd dump-class && ./gradlew build) && \
    npm run build && \
    curl https://storage.googleapis.com/snyk-function-snapshots/PROD-functions-node-6[..]T.json > node.json && \
    (npm start node node.json || true) && \
    python3 hodor/hodor.py PROD-functions-node.json && \
    python3 report.py

You need to build/install the dependencies yourself, see below.

Logging is off by default, consider starting everything with DEBUG=snyk:\*.

Architecture

arch overview

The main parts are:

Validation, which takes JSON from the VMS, and tells us if it's probably right
- Details of the ideas checked are below.
- This uses version info (from maven and npm), and
- "compiled" code (parsed by dump-class (using asm) or nodejs-runtime-agent).
Lookup, which exposes information about some source
- Primarily, it is exposing a mapping between "the source" and "what the agent wants".
- The source is fetched from maven/npm (i.e. never from github), and
- interpreted by nodejs-runtime-agent, or by another part of dump-class (using ecj).
Search, which walks through the version list, and runs a grep on each version

The code layout is pretty linear:

index.ts decides if we are a server.ts or a check-file.ts.
check-file.ts loads an array, and calls real-world.ts on every item.
server.ts runs express(), exposing real-world.ts, plus some direct calls to npmjs-com.ts and maven-central.ts.
real-world.ts takes a raw JSON fragment, and checks it matches the spec, and provides the node.ts or java.ts validator.
npmjs/node.ts implements the node validation rules, relying heavily on npmjs-com.ts.
maven/java.ts implements the Java validation rules, relying heavily on maven-central.ts.
npmjs/npmjs-com.ts is our client for npmjs.com, and npmjs-aware helper APIs.
maven/maven-central.ts is our client for Maven Central, and Central-aware helper APIs.

Endpoints

The server is deployed on dev and prod. It has many endpoints:

/lookup/ - the code exploration tool
/search/ - the code search tool

This uses these helpers, all of which take a lang/ecosystem:

/api/explain/version-list - list versions in an artifact
/api/explain/file-list - list files in an artifact
/api/explain/get-file - fetch a named file from an artifact
POST /api/explain/:lang - compile the file, in context

And, it exposes validation:

curl -HContent-Type:' application/json' -d'{
    "functionId": {
      "className": "UTF8StreamJsonParser",
      "filePath": "com/fasterxml/jackson/core/json/UTF8StreamJsonParser.java",
      "functionName": "_portInvalidToken"
    },
    "packageName": "com.fasterxml.jackson.core:jackson-core",
    "version": [
      "[,2.8.1)"
    ]
  }' -u$PROXY_CREDS https://func-validator.dev.snyk.io/api/validate/java

{
  "ok": false,
  "model": {
    "result": {
      "testedVersions": [
        "2.8.0"
      ],
      "errors": [
        {
          "version": "2.8.0",
          "msg": "Error: no such method _portInvalidToken in com/fasterxml/jackson/core/json/UTF8StreamJsonParser"
        }
      ]
    }
  }
}

Validation rules

java checks

@snyk/maven-semver can parse the version ranges, individually. (This is not what the agent wants, but a good start).
An artifact exists, and a version matches the version range specified.
All selected versions contain a class file named: functionId.filePath.replace(/.java$/, '.class').
That class file contains at least one method named functionId.functionName.

node checks

(node) semver can parse the version ranges.
npm-api can find the artifact, and a matching version.
All selected versions (as downloaded by pacote) contains the file.
The agent parses the file correctly, and returns the matching method.

insane Hodor validator

Note: This tool intentionally downloads and runs vulnerable code.

npm can install each listed version expression individually, in a clean project.
node can require the package name, with the agent loaded.
The module exports a function with the "right" name, or is a function itself. (This is the ~~insane~~ heuristic part.)
The ~~guessed~~ picked function is called with no arguments, and its output discarded.
The agent generated an event stating that the function was called.

Errors

The errors are designed to be human readable. Here are some hints, if you're stuck.

`no versions at all`

The package name doesn't appear to have any releases at all.

Are we looking in the right repositories? Is there a typo in the name?

Does it have library releases? Metadata-only releases are no good. We try to report this differently, but might be (very) wrong.

Is the version range syntax correct? We're using native version range specifiers, so <5.3 will match 0 Java versions (as that's a npmjs-style specifier).

`version range matched nothing`

We could find a list of versions for the package, but one of the ranges specified doesn't match anything. The tool found 1.0, 2.0, and 2.1; and you asked for [5,7).

Has the artifact been renamed between versions? e.g. foo up until version 2, foo3 from version 3 onwards? Is your version expression too complicated for the tool?

Note that other version ranges may have matched / processed successfully.

`class missing` / `path missing`

We picked a version of a package to test, we grabbed it, and we couldn't find the binary file in it.

Is there a typo in the file name? Is the file in a different artifact? Look at the filename to have a guess. com/foo/Bar is probably not in org.yellow:pink. It might be, though! Does the file exist in new versions, but not in older versions? Has someone included a source path, and the build system has moved the file around?

/lookup is very useful for these cases.

`method missing`

We found the file you asked for, but not the function inside it.

Is there a typo in the function name? Is the function in an inner class? Can the agent not see the method (or any methods) in this class?

/lookup is very useful for these cases. If it shows an empty function list, then there's probably an agent bug.

`invalid package name`

We don't think the package name is even syntactically valid.

Maven packages need a group and an artifact, connected by a :?

`failed fetch`

We thought your input was valid, but the package manager disagreed!

This is probably a bug in the validator, and quite possibly a bug our assumptions about the ecosystem.

`version had no acceptable classifiers`

The package name doesn't appear to have any library releases.

Are we looking in the right repositories? Is there a typo in the name? Is this some kind of "grouping" artifact, and there are specific "implementation" artifact? Is this an application, not a library?

This is very poorly presented on Maven, and confusing. sad-trombone

`resolution`: (anything but `known`)

hodor tries to guess the method to call. It's really not very good. If the resolution isn't listed as known, it's probably not worth you wasting your time on the issue.

Expectations

The validator has served us well. We have not had a reported false negative. We have seen false positives: the validator believes the data is okay, but the agent fails to match it at runtime. This has always been an agent bug, which are slowly fixed.

Hodor has not served us well. Initially, it had a very high false negative rate, and didn't explain why.

Now, it does a better job of explaining why it is reporting a negative (so they can be treated as more of a warning).

Many of these warnings need to be fixed in Hodor. Work on that has started, see victim.js#resolutionKnown.

Hodor is very unlikely to report false positives, as it is a full e2e test of the application and agent, mocking only http and some timers.

Project info

dependencies

node build environment
https://github.com/snyk/maven-info, locally will use $SNYK_INTERNAL_PROXY_CREDENTIALS to access dev's deployment.
npm in the path and working.
java (8+) and mvn in the path and working.
python 3.6+ and modern pip/asttokens/setuptools/packaging in the path and working.
The dump-class sub-project built and working: (cd dump-class && ./gradlew build)

It will fill your disc with downloads, and never deletes anything. om nom nom

% du -shc ~/.m2 ~/.cache/npm ~/.cache/func-validator/archives
7.1G   ~/.m2
779M   ~/.cache/npm
 15G   ~/.cache/func-validator/archives