data-loss-signatures v1.0.5

data-loss-signatures

Identify confidential and sensitive info in source code repositories by data-loss "signatures".

data-loss-signatures is a Node.js module for storing and accessing to data-leakage detection definitions. We call the data structure that represents a data-leakage detection defintion a "signature." We store a community-tested list of signatures in a file called signatures.json.

Table of Contents

• 1. Security
• 2. Install
• 3. Usage
• 4. API
  • 4.1. data-loss-signatures.Signature
  • 4.2. data-loss-signatures.Signature.prototype.match
• 5. Accessing signatures with other tools and programming languages
• 6. Maintainers
• 7. Contributions
  • 7.1. Adding a Signature
  • 7.2. Editing a Signature
  • 7.3. Removing a Signature
• 8. License
• 9. References and Attributions

1. Security

Data leakage is the unauthorized transmission of data from within an organization to an external destination or recipient.^1

One of the most common forms of data-loss (aka, "data leakage") happens when developers (inadvertently) commit and push passwords, access-tokens, and sensitive data to a source-control management system (like Git). Consequently, confidential information "leaks" into search results and commit history.

The signatures.json contains a growing list of definitions to help you detect secrets in your source code repositories.

2. Install

Before you begin, you'll need to have these

• Programming languages:

  • Node.js >= 10.x

  • NPM >= 4.0.0

• Skills:

  You'll need to know how to access the command line (aka, "Terminal") on your machine.

Open a Terminal and enter the following command:

# As a dependency in your Node.js app
npm i data-loss-signatures --save-prod

3. Usage

Use data-loss-signatures.signatures to find file extensions, names, and paths that commonly leak secrets.

const { signatures } = require('data-loss-signatures')
// ⚠️ Note: the 'recursive-readdir' module is not bundled with
//    data-loss-signatures. 'recursive-readdir' is referenced
//    only as an example.
const recursiveReaddir = require('recursive-readdir')

const potentialLeaks = recursiveReaddir('/path/to/local/repo')
  .then(files => files
    .map(file => signatures
    .map(signature => signature.match(file)))
  )
  .catch(err => err)

4. API

The data-loss-signatures module provides a Signatures class, which validates data-loss-signatures and converts regular expression strings to RE2 (whenever possible).

The data-loss-signatures module's public API provides:

1. factory method: a convenience function that creates a signature object.
2. nullSignature: implements a default object literal with all signatures properties set to null.
3. Signature: a class that constructs a signature object.
4. signatures: an array of Signature instances.
5. toArray(data: {String|Array.<Object>}): generates an Array.<Signature> from a JSON string or object literal array.
6. validParts: a constants enum of valid Signature.prototype.part values.
7. validTypes: a constants enum of valid Signature.prototype.type values.

4.1. data-loss-signatures.Signature

A class that constructs Signature objects.

const { Signature, validParts, validTypes } = require('data-loss-signatures')

const signature = new Signature({
  caption: 'Potential cryptographic private key',
  description: '',
  part: validParts.EXTENSION,
  pattern: '.pem',
  type: validTypes.MATCH
})

4.2. data-loss-signatures.Signature.prototype.match

Discover possible data leaks by matching a Signature pattern against file extensions, names, and paths.

const rsaTokenSignature = new Signature({
  'caption': 'Private SSH key',
  'description': '',
  'part': 'filename',
  'pattern': '^.*_rsa$',
  'type': 'regex'
})

const suspiciousFilePath = '/hmm/what/might/this/be/id_rsa'
rsaTokenSignature.match(suspiciousFilePath)
// => ['/hmm/what/might/this/be/id_rsa']

const fileThatIsJustBeingCoolBruh = 'file/that/is/just/being/cool/bruh'
rsaTokenSignature.match(suspiciousFilePath)
// => null

Review the source code for signature.

5. Accessing signatures with other tools and programming languages

You can access signatures.json without the data-loss-signatures Node module. Select a tool or programming language below to view examples.

You can access data-loss rules using HTTPS. You can GET all signatures directly from Gitlab with cURL.

curl -X GET \
  'https://gitlab.com/gregswindle/data-loss-signatures/raw/master/signatures.json'

package main

import (
	"fmt"
	"net/http"
	"io/ioutil"
)

func main() {

	url := "https://gitlab.com/gregswindle/data-loss-signatures/raw/master/signatures.json"

	req, _ := http.NewRequest("GET", url, nil)

	req.Header.Add("Private-Token", "<your-personal-token>")
	req.Header.Add("cache-control", "no-cache")

	res, _ := http.DefaultClient.Do(req)

	defer res.Body.Close()
	body, _ := ioutil.ReadAll(res.Body)

	fmt.Println(res)
	fmt.Println(string(body))

}

const http = require('https')

const options = {
  method: 'GET',
  hostname: ['gitlab', 'com'],
  path: ['api', 'v4', 'projects'],
  headers: {
    'Private-Token': '<your-access-token>',
    'cache-control': 'no-cache'
  }
}

const req = http.request(options, res => {
  const chunks = []

  res.on('data', chunk => {
    chunks.push(chunk)
  })

  res.on('end', () => {
    var body = Buffer.concat(chunks)
    console.log(body.toString())
  })
})

req.end()

Python3

import http.client

conn = http.client.HTTPConnection("gitlab,com")

payload = ""

headers = {
  'Accept': "application/json",
  'cache-control': "no-cache"
}

conn.request("GET", "gregswindle,data-loss-signatures,raw,master,signatures.json", payload, headers)

res = conn.getresponse()
data = res.read()

print(data.decode("utf-8"))

Python2

import requests

url = "https://gitlab.com/gregswindle/data-loss-signatures/raw/master/signatures.json"

payload = ""
headers = {
  'Accept': "application/json",
  'cache-control': "no-cache"
}

response = requests.request("GET", url, data=payload, headers=headers)

print(response.text)

require 'uri'
require 'net/http'

url = URI("'https://gitlab.com/gregswindle/data-loss-signatures/raw/master/signatures.json")

http = Net::HTTP.new(url.host, url.port)

request = Net::HTTP::Get.new(url)
request["Private-Token"] = '<your-personal-token>'
request["cache-control"] = 'no-cache'

response = http.request(request)
puts response.read_body

6. Maintainers

@gregswindle

The Maintainer Guide has useful information for Maintainers and Trusted Committers.

7. Contributions

We gratefully accept Merge Requests! Here's what you need to know to get started.

Before submitting a Merge Request, please read our:

• Code of Conduct
• Contributing Aggreement
• Developer Guide
• Maintainer/Trusted Committer Guide

Thanks goes to our awesome contributors (emoji key):

This project follows the all-contributors specification. Contributions of any kind welcome!

7.1. Adding a Signature

Before adding a new Signature, please review all current definitions: the Signature might already exist.

If the Signature does not exist, please be sure to add your Signature with the following properties:

1. caption: A succinct summary for the Signature. Think of caption as a well-written email subject.

2. description: Provide more details about the Signature if necessary. description is especially useful for differentiating similar Signatures.

3. part: An enumeration that defines what the Signature is evaluating. Valid values are:

   • contents: The string(s) within a file.
   • extension: A file extension (which defines the Content-Type or mime-type).
   • filename: The unique name of the file.
   • path: The directory path relative to the repo and without the filename.

4. pattern: The string or regular expression to look for.

5. type: An enumeration that defines how to evaluate for secrets. Valid values are:

   • match: A strict string equivalency evaluation.
   • regex: A regular expression "search" or "test".

7.2. Editing a Signature

Edits are welcome! Just be sure to unit test.

7.3. Removing a Signature

Please provide a testable justification for any Signature removal.

8. License

Apache-2.0 © 2019 Greg Swindle

9. References and Attributions

^1: What is Data Leakage? Defined, Explained, and Explored | Forcepoint. (2019) Retrieved January 27, 2019, from https://www.forcepoint.com/cyber-edu/data-leakage