3.5.10 • Published 6 months ago

jscpd v3.5.10

Weekly downloads
43,644
License
MIT
Repository
github
Last release
6 months ago

jscpd

npm jscpd license npm

codecov FOSSA Status Backers on Open Collective Sponsors on Open Collective

NPM

Copy/paste detector for programming source code, supports 150+ formats.

Copy/paste is a common technical debt on a lot of projects. The jscpd gives the ability to find duplicated blocks implemented on more than 150 programming languages and digital formats of documents. The jscpd tool implements Rabin-Karp algorithm for searching duplications.

Table of content

Features

  • Detect duplications in programming source code, use semantic of programing languages, can skip comments, empty lines etc.
  • Detect duplications in embedded blocks of code, like <script> or <style> sections in html
  • Blame authors of duplications
  • Generate XML report in pmd-cpd format, JSON report, HTML report
  • Integrate with CI systems, use thresholds for level of duplications

Getting started

Installation

$ npm install -g jscpd

Usage

$ npx jscpd /path/to/source

or

$ jscpd /path/to/code

or

$ jscpd --pattern "src/**/*.js"

Options

Pattern

Glob pattern for find files to detect

  • Cli options: --pattern, -p
  • Type: string
  • Default: "*/"

Example:

$ jscpd --pattern "**/*.js"

Min Tokens

Minimal block size of code in tokens. The block of code less than min-tokens will be skipped.

  • Cli options: --min-tokens, -k
  • Type: number
  • Default: 50

    This option is called minTokens in the config file.

Min Lines

Minimal block size of code in lines. The block of code less than min-lines will be skipped.

  • Cli options: --min-lines, -l
  • Type: number
  • Default: 5

Max Lines

Maximum file size in lines. The file bigger than max-lines will be skipped.

  • Cli options: --max-lines, -x
  • Type: number
  • Default: 1000

Max Size

Maximum file size in bytes. The file bigger than max-size will be skipped.

  • Cli options: --max-size, -z
  • Type: string
  • Default: 100kb

Threshold

The threshold for duplication level, check if current level of duplications bigger than threshold jscpd exit with error.

  • Cli options: --threshold, -t
  • Type: number
  • Default: null

Config

The path to configuration file. The config should be in json format. Supported options in config file can be the same with cli options.

  • Cli options: --config, -c
  • Type: path
  • Default: null

Ignore

The option with glob patterns to ignore from analyze. For multiple globs you can use comma as separator. Example:

$ jscpd --ignore "**/*.min.js,**/*.map" /path/to/files
  • Cli options: --ignore, -i
  • Type: string
  • Default: null

Reporters

The list of reporters. Reporters use for output information of clones and duplication process.

Available reporters:

  • console - report about clones to console;
  • consoleFull - report about clones to console with blocks of code;
  • json - output jscpd-report.json file with clones report in json format;
  • xml - output jscpd-report.xml file with clones report in xml format;
  • csv - output jscpd-report.csv file with clones report in csv format;
  • markdown - output jscpd-report.md file with clones report in markdown format;
  • html - generate html report to html/ folder;
  • verbose - output a lot of debug information to console;

Note: A reporter can be developed manually, see @jscpd/finder package.

  • Cli options: --reporters, -r
  • Type: string
  • Default: console

Output

The path to directory for reports. JSON and XML reports will be saved there.

  • Cli options: --output, -o
  • Type: path
  • Default: ./report/

Mode

The mode of detection quality.

  • strict - use all types of symbols as token, skip only blocks marked as ignored.
  • mild - skip blocks marked as ignored and new lines and empty symbols.
  • weak - skip blocks marked as ignored and new lines and empty symbols and comments.

Note: A mode can be developed manually, see API section.

  • Cli options: --mode, -m
  • Type: string
  • Default: mild

Format

The list of formats to detect for duplications. Available over 150 formats.

Example:

$ jscpd --format "php,javascript,markup,css" /path/to/files
  • Cli options: --format, -f
  • Type: string
  • Default: {all formats}

Blame

Get information about authors and dates of duplicated blocks from git.

  • Cli options: --blame, -b
  • Type: boolean
  • Default: false

Silent

Don't write a lot of information to a console.

Example:

$ jscpd /path/to/source --silent
Duplications detection: Found 60 exact clones with 3414(46.81%) duplicated lines in 100 (31 formats) files.
Execution Time: 1381.759ms
  • Cli options: --silent, -s
  • Type: boolean
  • Default: false

Absolute

Use the absolute path in reports.

  • Cli options: --absolute, -a
  • Type: boolean
  • Default: false

Ignore Case

Ignore case of symbols in code (experimental).

  • Cli options: --ignoreCase
  • Type: boolean
  • Default: false

No Symlinks

Do not follow symlinks.

  • Cli options: --noSymlinks, -n
  • Type: boolean
  • Default: false

Skip Local

Use for detect duplications in different folders only. For correct usage of --skipLocal option you should provide list of path's with more than one item.

Example:

jscpd --skipLocal /path/to/folder1/ /path/to/folder2/

will detect clones in separate folders only, clones from same folder will be skipped.

  • Cli options: --skipLocal
  • Type: boolean
  • Default: false

Formats Extensions

Define the list of formats with file extensions. Available over 150 formats.

In following example jscpd will analyze files *.es and *.es6 as javascript and *.dt files as dart:

$ jscpd --formats-exts javascript:es,es6;dart:dt /path/to/code

Note: formats defined in the option redefine default configuration, you should define all need formats manually or create two configuration for run jscpd

  • Cli options: --formats-exts
  • Type: string
  • Default: null

Store

Stores used for collect information about code, by default all information collect in memory.

Available stores:

  • leveldb - leveldb store all data to files. The store recommended as store for big repositories. Should install @jscpd/leveldb-store before;

Note: A store can be developed manually, see @jscpd/finder package and @jscpd/leveldb-store as example.

  • Cli options: --store
  • Type: string
  • Default: null

Ignore Pattern

Ignore code blocks matching the regexp patterns.

  • Cli options: --ignore-pattern
  • Type: string
  • Default: null

Example:

$ jscpd /path/to/source --ignore-pattern "import.*from\s*'.*'"

Excludes import statements from the calculation.

Config File

Put .jscpd.json file in the root of the projects:

{
  "threshold": 0,
  "reporters": ["html", "console", "badge"],
  "ignore": ["**/__snapshots__/**"],
  "absolute": true
}

Also you can use section in package.json:

{
  ...
  "jscpd": {
    "threshold": 0.1,
    "reporters": ["html", "console", "badge"],
    "ignore": ["**/__snapshots__/**"],
    "absolute": true,
    "gitignore": true
  }
  ...
}

Exit code

By default, the tool exits with code 0 even code duplications were detected. This behaviour can be changed by specifying a custom exit code for error states.

Example:

jscpd --exitCode 1 .
  • Cli options: --exitCode
  • Type: number
  • Default: 0

Ignored Blocks

Mark blocks in code as ignored:

/* jscpd:ignore-start */
import lodash from 'lodash';
import React from 'react';
import {User} from './models';
import {UserService} from './services';
/* jscpd:ignore-end */
<!--
// jscpd:ignore-start
-->
<meta data-react-helmet="true" name="theme-color" content="#cb3837"/>
<link data-react-helmet="true" rel="stylesheet" href="https://static.npmjs.com/103af5b8a2b3c971cba419755f3a67bc.css"/>
<link data-react-helmet="true" rel="stylesheet" href="https://static.npmjs.com/cms/flatpages.css"/>
<link data-react-helmet="true" rel="apple-touch-icon" sizes="120x120" href="https://static.npmjs.com/58a19602036db1daee0d7863c94673a4.png"/>
<link data-react-helmet="true" rel="apple-touch-icon" sizes="144x144" href="https://static.npmjs.com/7a7ffabbd910fc60161bc04f2cee4160.png"/>
<link data-react-helmet="true" rel="apple-touch-icon" sizes="152x152" href="https://static.npmjs.com/34110fd7686e2c90a487ca98e7336e99.png"/>
<link data-react-helmet="true" rel="apple-touch-icon" sizes="180x180" href="https://static.npmjs.com/3dc95981de4241b35cd55fe126ab6b2c.png"/>
<link data-react-helmet="true" rel="icon" type="image/png" href="https://static.npmjs.com/b0f1a8318363185cc2ea6a40ac23eeb2.png" sizes="32x32"/>
<!--
// jscpd:ignore-end
-->

Reporters

HTML

Demo report

Badge

jscpd

More info jscpd-badge-reporter

PMD CPD XML

<?xml version="1.0" encoding="utf-8"?>
<pmd-cpd>
  <duplication lines="10">
      <file path="/path/to/file" line="1">
        <codefragment><![CDATA[ ...first code fragment... ]]></codefragment>
      </file>
      <file path="/path/to/file" line="5">
        <codefragment><![CDATA[ ...second code fragment...}]]></codefragment>
      </file>
      <codefragment><![CDATA[ ...duplicated fragment... ]]></codefragment>
  </duplication>
</pmd-cpd>

JSON reporters

{
  "duplicates": [{
      "format": "javascript",
      "lines": 27,
      "fragment": "...code fragment... ",
      "tokens": 0,
      "firstFile": {
        "name": "tests/fixtures/javascript/file2.js",
        "start": 1,
        "end": 27,
        "startLoc": {
          "line": 1,
          "column": 1
        },
        "endLoc": {
          "line": 27,
          "column": 2
        }
      },
      "secondFile": {
        "name": "tests/fixtures/javascript/file1.js",
        "start": 1,
        "end": 24,
        "startLoc": {
          "line": 1,
          "column": 1
        },
        "endLoc": {
          "line": 24,
          "column": 2
        }
      }
  }],
  "statistic": {
    "detectionDate": "2018-11-09T15:32:02.397Z",
    "formats": {
      "javascript": {
        "sources": {
          "/path/to/file": {
            "lines": 24,
            "sources": 1,
            "clones": 1,
            "duplicatedLines": 26,
            "percentage": 45.33,
            "newDuplicatedLines": 0,
            "newClones": 0
          }
        },
        "total": {
          "lines": 297,
          "sources": 1,
          "clones": 1,
          "duplicatedLines": 26,
          "percentage": 45.33,
          "newDuplicatedLines": 0,
          "newClones": 0
        }
      }
    },
    "total": {
      "lines": 297,
      "sources": 6,
      "clones": 5,
      "duplicatedLines": 26,
      "percentage": 45.33,
      "newDuplicatedLines": 0,
      "newClones": 0
    }
  }
}

API

For integration copy/paste detection to your application you can use programming API:

jscpd Promise API

import {IClone} from '@jscpd/core';
import {jscpd} from 'jscpd';

const clones: Promise<IClone[]> = jscpd(process.argv);

jscpd async/await API

import {IClone} from '@jscpd/core';
import {jscpd} from 'jscpd';
(async () => {
  const clones: IClone[] = await jscpd(['', '', __dirname + '/../fixtures', '-m', 'weak', '--silent']);
  console.log(clones);
})();

detectClones API

import {detectClones} from "jscpd";

(async () => {
  const clones = await detectClones({
    path: [
      __dirname + '/../fixtures'
    ],
    silent: true
  });
  console.log(clones);
})()

detectClones with persist store

import {detectClones} from "jscpd";
import {IMapFrame, MemoryStore} from "@jscpd/core";

(async () => {
  const store = new MemoryStore<IMapFrame>();

  await detectClones({
    path: [
      __dirname + '/../fixtures'
    ],
  }, store);

  await detectClones({
    path: [
      __dirname + '/../fixtures'
    ],
    silent: true
  }, store);
})()

In case of deep customisation of detection process you can build your own tool: If you are going to detect clones in file system you can use @jscpd/finder for make a powerful detector. In case of detect clones in browser or not node.js environment you can build your own solution base on @jscpd/code

Changelog

Changelog

Who uses jscpd

  • Code-Inspector is a code analysis and technical debt management service.
  • Mega-Linter is a 100% open-source linters aggregator for CI (GitHub Action & other CI tools) or to run locally
  • vscode-jscpd VSCode Copy/Paste detector plugin.

Contributors

This project exists thanks to all the people who contribute.

Backers

Thank you to all our backers! 🙏 [Become a backer]

Sponsors

Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [Become a sponsor]

ga tracker

License

MIT © Andrey Kucherenko

3.5.10

6 months ago

3.5.6

11 months ago

3.5.5

11 months ago

3.5.9

11 months ago

3.5.8

11 months ago

3.5.4

1 year ago

3.5.3

1 year ago

3.5.1

1 year ago

3.5.0

1 year ago

3.4.2

2 years ago

3.4.1

2 years ago

3.4.5

2 years ago

3.3.26

3 years ago

3.3.25

3 years ago

3.3.24

3 years ago

3.3.23

3 years ago

3.3.22

3 years ago

3.3.20

3 years ago

3.3.21

3 years ago

3.3.19

4 years ago

3.3.17

4 years ago

3.3.14

4 years ago

3.3.16

4 years ago

3.3.13

4 years ago

3.3.9

4 years ago

3.3.11

4 years ago

3.3.8

4 years ago

3.3.3

4 years ago

3.3.2

4 years ago

3.3.1

4 years ago

3.3.0-rc.10

4 years ago

3.3.0-rc.9

4 years ago

3.3.0-rc.8

4 years ago

3.3.0-rc.7

4 years ago

3.3.0-rc.6

4 years ago

3.3.0-rc.5

4 years ago

3.3.0-rc.4

4 years ago

3.3.0-rc.3

4 years ago

3.3.0-alpha.8

4 years ago

3.3.0-alpha.3

4 years ago

3.3.0-alpha.2

4 years ago

3.2.1

4 years ago

3.2.0

4 years ago

3.1.0

4 years ago

3.0.1

4 years ago

3.0.0

4 years ago

2.0.16

5 years ago

2.0.15

5 years ago

2.0.14

5 years ago

2.0.13

5 years ago

2.0.12

5 years ago

2.0.11

5 years ago

2.0.10

5 years ago

2.0.9

5 years ago

2.0.8

5 years ago

2.0.7

5 years ago

2.0.6

5 years ago

2.0.5

5 years ago

2.0.4

5 years ago

2.0.3

5 years ago

2.0.2

5 years ago

2.0.1

5 years ago

2.0.0

5 years ago

1.2.3

5 years ago

1.2.2

5 years ago

1.2.1

5 years ago

1.2.0

5 years ago

1.1.0

5 years ago

1.0.3

5 years ago

1.0.2

5 years ago

1.0.1

5 years ago

1.0.0

5 years ago

1.0.0-rc.6

5 years ago

1.0.0-rc.5

5 years ago

1.0.0-rc.4

5 years ago

0.6.25

5 years ago

1.0.0-rc.3

5 years ago

1.0.0-rc.2

5 years ago

1.0.0-rc.1

5 years ago

1.0.0-rc.0

5 years ago

1.1.0-rc.0

5 years ago

0.6.24

6 years ago

0.6.23

6 years ago

1.0.0-alpha.2

6 years ago

1.0.0-alpha.1

6 years ago

1.0.0-alpha.0

6 years ago

0.6.22

6 years ago

0.6.21

6 years ago

0.6.20

6 years ago

0.6.19

6 years ago

0.6.18

6 years ago

0.6.17

6 years ago

0.6.15

7 years ago

0.6.14

7 years ago

0.6.13

7 years ago

0.6.12

7 years ago

0.6.11

7 years ago

0.6.10

7 years ago

0.6.9

7 years ago

0.6.8

7 years ago

0.6.7

7 years ago

0.6.6

7 years ago

0.6.5

7 years ago

0.6.4

8 years ago

0.6.3

8 years ago

0.6.2

8 years ago

0.6.1

8 years ago

0.6.0

8 years ago

0.5.4

8 years ago

0.5.3

8 years ago

0.5.2

8 years ago

0.5.1

8 years ago

0.5.0

8 years ago

0.4.3

8 years ago

0.4.2

9 years ago

0.4.1

9 years ago

0.4.0

9 years ago

0.3.6

9 years ago

0.3.5

9 years ago

0.3.4

9 years ago

0.3.3

10 years ago

0.3.2

10 years ago

0.3.1

10 years ago

0.3.0

10 years ago

0.2.19

10 years ago

0.2.18

10 years ago

0.2.17

10 years ago

0.2.16

10 years ago

0.2.15

10 years ago

0.2.14

10 years ago

0.2.13

10 years ago

0.2.12

10 years ago

0.2.11

10 years ago

0.2.10

10 years ago

0.2.9

10 years ago

0.2.8

11 years ago

0.2.7

11 years ago

0.2.6

11 years ago

0.2.4

11 years ago

0.2.3

11 years ago

0.2.1

11 years ago

0.2.0

11 years ago

0.1.7

11 years ago

0.1.6

11 years ago

0.1.5

11 years ago

0.1.4

11 years ago

0.1.3

11 years ago

0.1.2

11 years ago

0.1.1

11 years ago